Blueprint: Standalone Unit Tests
|Proposal||Run unit tests without TYPO3 CMS bootstrap|
|Status||Draft, Discussion, Voting Phase, Accepted, Declined, Withdrawn|
|Current Progress||Unknown, Started, Good Progress, Bad Progress, Stalled, Review Needed, DONE|
|Topic for Gerrit||standalone-tests|
- Started during TYPO3 CMS 6.2 development
Goals / Motivation
- Clean and side-effect free tests
- Spend less time on hunting hard to track cross-dependencies when changing non related code sections
- Run tests within Phpstorm and travis more easily
- Use native phpunit
- Don't rely on TYPO3 CMS instance setup, run (core) tests without dependencies to setup
- Obsolete ext:phpunit
- Improve overall code quality
- Clear distinction between unit tests and functional tests
- Improve code quality and encapsulation
- Simplify unit test system for extension authors, easier integration of extension tests in travis
At the time of this writing there are roughly 6.200 unit tests executed on each merge by travis-ci.org. Developers contributing to the core are encouraged to cover functionality with tests to specify and ensure working solutions of patches. Unit tests are currently often not free of side-affects and rely on a working TYPO3 CMS instance.
Executing TYPO3 CMS unit tests currently requires the phpunit extension to be loaded that bundles native phpunit itself together with a bunch of wrapper scripts to run tests within the CMS framework. This has a lot of negative side effects to the nature of unit tests:
- On command line unit tests are executed through the cli_dispatch TYPO3 CMS wrapper script. The tests must run within a full blown TYPO3 CMS instance and its specific environment that in not under control of the test system. For example, to run the unit tests of indexed_search extension, it is required to have a local setup with this extension loaded in PackageStates.php, otherwise the tests will fatal since the autoloader won't find the classes.
- Lots of tests depend on the configuration and bootstrap of the instance and may fail depending on system setup that is not under control of the unit test system.
- Lots of "unit" tests are in fact "functional" tests and do not mock dependencies. Those are hard to maintain and may fail if the underlying system or configuration is changed.
To solve this, unit test execution should be separated from instance specific bootstrap. In the end, unit tests should be run on git master directly without a document root or other specific dependencies like LocalConfiguration.php settings.
- Release unit tests runner from cli_dispatch by calling the UnitTests.xml with phpunit directly that calls a bootstrap.php script which emulates the current cli_dispatch.php code. This is similar to what was done for functional tests already. A first patch for this has been merged with https://review.typo3.org/#/c/23396/
- Reduce the bootstrap calls to TYPO3 CMS bootstrap one-by-one and improve mocking of then failing unit tests. Create some real functional tests from current unit tests and improve unit tests in general, especially for sections like FAL for a clear separation of concerns. Lots of single patches will pop up in this area.
- Introduce an own unit test autoloader to find classes and do not depend on instance specific settings like PackageStates.php for test execution.
- Configure travis-ci to use existing phpunit directly without relying on ext:phpunit
- Document the whole system and options to run and write tests.
Issues and reviews
Please use topic "standalone-tests"