Translations
Info
All page names need to be in English.
en da  de  fr  it  ja  km  nl  ru  zh

Functional testing

From TYPO3Wiki
Jump to: navigation, search

<< Back to Document matrix

see also: Category:PHPUnit

Difference between unit and functional tests

Since TYPO3 6.2, additionally to unit tests, you can also write functional tests for TYPO3.

Unit tests should test only one small piece of code, and should not modify the environment (files, database). However with functional tests you can test the complete functionality.

With TYPO3 CMS version 6.2 the functional test execution and its required setup was streamlined. See Blueprints/StandaloneUnitTests for more details.


How to run functional tests?

Setup

https://getcomposer.org/ should be available on the system already, see its documentation for installation details. For functional tests, a database connection and credentials should be at hand.

shell script:
  git clone --single-branch --branch master git://git.typo3.org/Packages/TYPO3.CMS.git
  cd TYPO3.CMS
  composer install

Execute all functional tests

TYPO3 >=8.7

shell script:
  typo3DatabaseName="yourDatabase" typo3DatabaseUsername="yourUser" typo3DatabasePassword="yourPassword" typo3DatabaseHost="localhost" \
  bin/phpunit -c vendor/typo3/testing-framework/Resources/Core/Build/FunctionalTests.xml

TYPO3 < 8.7

shell script:
  typo3DatabaseName="yourDatabase" typo3DatabaseUsername="yourUser" typo3DatabasePassword="yourPassword" typo3DatabaseHost="localhost" \
  bin/phpunit -c typo3/sysext/core/Build/FunctionalTests.xml


Note: You can skip the first line, if you run the tests within an existing TYPO3 CMS installation. Be aware that the DB user needs extended permissions in order to run the functional tests. The user needs permission to create databases, because this is what happens when bootstrapping a clean typo3 env.

Troubleshooting: When using OSX with MySQL 5.6, it might be that during the execution it loses the connection to the database. This seems to be a bug, mentioned at bugs.mysql.com. Adding or changing the setting table_open_cache to 500 in your MySQL configuration seems to solve the problem. (table_open_cache = 500)


Execute all functional tests on PostgreSQL

TYPO3 >=8.7

shell script:
  typo3DatabaseName="yourDatabase" typo3DatabaseUsername="yourUser" typo3DatabasePassword="yourPassword" typo3DatabaseHost="localhost" \
  bin/phpunit --exclude-group mysql -c vendor/typo3/testing-framework/Resources/Core/Build/FunctionalTests.xml

Execute only a single test suite

TYPO3 >=8.7

shell script:
  typo3DatabaseName="yourDatabase" typo3DatabaseUsername="yourUser" typo3DatabasePassword="yourPassword" typo3DatabaseHost="localhost" \
  bin/phpunit -c vendor/typo3/testing-framework/Resources/Core/Build/FunctionalTests.xml \
  typo3/sysext/extbase/Tests/Functional/Persistence/CountTest.php

TYPO3 < 8.7

shell script:
  typo3DatabaseName="yourDatabase" typo3DatabaseUsername="yourUser" typo3DatabasePassword="yourPassword" typo3DatabaseHost="localhost" \
  bin/phpunit -c typo3/sysext/core/Build/FunctionalTests.xml \
  typo3/sysext/extbase/Tests/Functional/Persistence/CountTest.php

Execute only a single test method

TYPO3 >=8.7

shell script:
  typo3DatabaseName="yourDatabase" typo3DatabaseUsername="yourUser" typo3DatabasePassword="yourPassword" typo3DatabaseHost="localhost" \
  bin/phpunit -c vendor/typo3/testing-framework/Resources/Core/Build/FunctionalTests.xml \
  typo3/sysext/extbase/Tests/Functional/Persistence/CountTest.php --filter ::offsetCountTest$

TYPO3 < 8.7

shell script:
  typo3DatabaseName="yourDatabase" typo3DatabaseUsername="yourUser" typo3DatabasePassword="yourPassword" typo3DatabaseHost="localhost" \
  bin/phpunit -c typo3/sysext/core/Build/FunctionalTests.xml \
  typo3/sysext/extbase/Tests/Functional/Persistence/CountTest.php --filter ::offsetCountTest$

A note on debugging: You will not be able to remote-debug the functional tests on CLI, because the test require the "--process-isolation" flag which will try to connect to your IDE with each process separately. Most IDEs are not able to cope with this (including PHPStorm). You can work around it by running only one test method and disabling process-isolation:

TYPO3 >=8.7

shell script:
  typo3DatabaseName="yourDatabase" typo3DatabaseUsername="yourUser" typo3DatabasePassword="yourPassword" typo3DatabaseHost="localhost" \
  bin/phpunit --bootstrap vendor/typo3/testing-framework/Resources/Core/Build/FunctionalTestsBootstrap.php \
  typo3/sysext/extbase/Tests/Functional/Persistence/CountTest.php --filter ::offsetCountTest$

TYPO3 < 8.7

shell script:
  typo3DatabaseName="yourDatabase" typo3DatabaseUsername="yourUser" typo3DatabasePassword="yourPassword" typo3DatabaseHost="localhost" \
  bin/phpunit --bootstrap typo3/sysext/core/Build/FunctionalTestsBootstrap.php \
  typo3/sysext/extbase/Tests/Functional/Persistence/CountTest.php --filter ::offsetCountTest$

Troubleshooting:

In case something goes wrong with the tests, try to clear the typo3temp data with

shell script:
# TYPO3 CMS 6.2 and 7
rm -rf typo3temp/functional-*
# TYPO3 CMS 8 and above
rm -rf typo3temp/var/tests/functional-*

Performance boost for test running

The functional tests take quite some time (~20 minutes) therefor you might want to consider some things boosting the performance of the tests. Two possibilities would be to use a RAM disk and gnu parallels for running the tests. Both descriptions both consider only Linux at the moment, feel free to add descriptions for mac and Windows if you can.

Using a ram disk for the database

Check if your /tmp/ directory is already mounted with tmpfs:

shell script:
df
 ...
 tmpfs  1024000    234460    789540   23% /tmp

Use the following script to create a ram disk and run a mysql daemon using it. If your /tmp/ directory is mounted with tmpfs, you can skip the "mount" step below:

shell script:
if [[ ! -e /tmp/typo3ramdisk ]]; then
	sudo mkdir /tmp/typo3ramdisk
	sudo mount -t tmpfs -o size=512m tmpfs /tmp/typo3ramdisk
	sudo chmod 700 /tmp/typo3ramdisk
	sudo chown mysql:mysql /tmp/typo3ramdisk
	sudo mysqld_safe --port=3307 --socket="/var/run/mysqld/mysqld-testing.sock" --user mysql --pid-file="/var/run/mysqld/mysqld-testing.pid" --datadir="/tmp/typo3ramdisk" --skip-grant-tables &
	while [ ! -S /var/run/mysqld/mysqld-testing.sock ]; do sleep 1; done
fi

Make sure you have enough free RAM. Otherwise tmpfs might swap to the harddisk, slowing down the tests.

Afterwards adjust the database parameters to match your new mysql demon (typo3DatabaseName="yourDatabase" typo3DatabaseUsername="root", typo3DatabasePassword="", typo3DatabaseHost="127.0.0.1", typo3DatabasePort="3307").

If you are using Ubuntu, you also have to take care of apparmor, it will log stuff similar to this to /var/log/syslog

shell script:
Feb 23 00:25:39 klaus kernel: [   56.248335] type=1400 audit(1456183539.807:21): apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/home/vagrant/.my.cnf" pid=4199 comm="mysqld" requested_mask="r" denied_mask="r" fsuid=0 ouid=1000

Allow creating these sockets by adding the following two lines to /etc/apparmor.d/usr.sbin.mysqld, it is most easy to reboot the system afterwards.

shell script:
  /run/mysqld/mysqld-testing.pid rw,
  /run/mysqld/mysqld-testing.sock w,

Alternative to using a RAM disk

Instead of using a dedicated RAM disk which could eventually cause swapping you could alter the mysql configuration to avoid flushing/syncing to disk. The functional units tests are quite demanding because some of the SQL statements being executed cause mysql to sync/flush data to the filesystem which causes storage IO. To avoid this you can tune your mysql by adding the following options in your "my.cnf" and restarting the mysql daemon. Place the configuration options at the appropriate places in your my.cnf and take care they do not get overwritten by later statments of the same variable.

   [mysqld]
   
   # Turn off logs
   general_log = 0
   slow_query_log = off
   
   # Turn off flushing, double-buffering, checksums, etc.
   # Do not use those for production type systems
   innodb_flush_method = nosync
   innodb_flush_log_at_trx_commit = 0
   innodb_doublewrite = 0
   innodb_checksums = 0
   
   # mysql > 5.6.3: Can give performance on SSDs
   # innodb_flush_neighbors = 0

This setup tries to avoid flushing of data to disk and thus to prevent IO operations. It relies on the fact that the OS (Linux) is doing a proper FS caching in RAM already.

Running tests in parallel

With "parallel" it is possible to execute command line processes in parallel. Add some find magic and you can run your tests in parallel. Be aware that parallel does not halt on errors from its jobs by default. So there is a high risk to overlook single test failures. Using `--halt-on-error 2` will immediately stop all jobs if one job exited with an error.

shell script:
time find -L . -name \*Test.php -path \*typo3_src/typo3/sysext/*/Tests/Functional* | parallel --halt-on-error 2 --gnu 'echo; echo "Running functional {} test case";  ./bin/phpunit --colors -c typo3/sysext/core/Build/FunctionalTests.xml" {}'

Optimize parallel test result printing

If you want you can additionally use the psychomieze/parallel-phpunit-resultPrinter - see more information on github: https://github.com/psychomieze/parallel-phpunit-resultPrinter

In the end, your script should look something like this: https://gist.github.com/psychomieze/520e7180c85a34007a54#file-fastfunctionaltests

Windows users

similar like the above, just instead of the last line:

shell script:
set typo3DatabaseName=yourDatabase  
set typo3DatabaseHost=yourHost
set typo3DatabaseUsername=yourUser
set typo3DatabasePassword=yourPassword
bin\phpunit -c typo3/sysext/core/Build/FunctionalTests.xml

House keeping

The functional test do leave some "dirt" behind. That is:

  • some folders in typo3temp/; naming pattern funtional-*
  • some databases

These leftovers are intended to speed up reuse of the functional tests. One may delete them anytime. If you want to cleanup the databases, you can use a shell script which drop the databases. see this gist

How to write functional test?

Place you test class under <extensionFolder>/Tests/Functional folder. Your class have to extend \TYPO3\CMS\Core\Tests\FunctionalTestCase class

Important: If you're overriding setUp or tearDown methods in your test, do not forget to call parent::setUp and parent::tearDown.

For example implementation see

  • \TYPO3\CMS\Core\Tests\Functional\DataHandling\DataHandlerTest
  • \TYPO3\CMS\Workspaces\Tests\Functional\DataHandling\DataHandlerTest

If your test rely on some core extension being installed, you need to override the property $this->coreExtensionsToLoad. For example:

PHP script:
/**
 * @var array
 */
protected $coreExtensionsToLoad = array('workspaces', 'info');

Paths:

  • Absolute path to the original TYPO3 environment is stored in the ORIGINAL_ROOT constant
  • Absolute path to the current test environment is stored in $this->instancePath field.

Test/fixture extensions

You can provide a test/fixture extension with your test. This way you can provide custom models, controllers, TCA configuration for more complex testing scenarios.

  • You need to declare test extensions in the $this->testExtensionsToLoad property. It should contain paths to extensions, relative to the document root. For example:
PHP script:
protected $testExtensionsToLoad = array(
    'typo3conf/ext/some_extension/Tests/Functional/Fixtures/Extensions/test_extension',
    'typo3conf/ext/base_extension',
);
  • Extensions in this array are linked to the test instance and loaded
  • and their ext_tables.sql will be applied.

How does the functional testing framework work?

For each test case (that is a file with multiple tests in it) the framework creates a separate TYPO3 CMS instance within typo3temp/ together with a separate database and LocalConfiguration. A full TYPO3 CMS bootstrap of this instance is done.

A new environment with a fresh PHP process is created for each and every single test. Functional test can use the API by calling parent::setUp() and parent::tearDown().

Fixtures

It is possible to load test data from fixture files directly into the database. Please put fixture files in the <extensionFolder>/Tests/Functional/Fixtures/ folder.

Please also note that Core comes with ready to use common fixtures placed in typo3/sysext/core/Tests/Functional/Fixtures folder. It contains fixtures like

  • pages
  • be_users
  • tt_content
  • ...

You can load fixture data to the database by calling $this->importDataSet('path/to/fixture/file.xml');

Fixture file structure:

XML / HTML:
<?xml version="1.0" encoding="utf-8"?>
<dataset>
	<table_name>
		<field_name>1</field_name>
		<field2_name>0</field2_name>
	</table_name>
        <table_name2>
		<field3_name>1</field3_name>
		<field4_name>0</field4_name>
	</table_name2>
</dataset>

Example

XML / HTML:
<?xml version="1.0" encoding="utf-8"?>
<dataset>
    <pages>
        <uid>1</uid>
        <pid>0</pid>
        <title>Root</title>
        <deleted>0</deleted>
        <perms_everybody>15</perms_everybody>
    </pages>
    <pages>
        <uid>2</uid>
        <pid>1</pid>
        <title>Dummy 1-2</title>
        <deleted>0</deleted>
        <perms_everybody>15</perms_everybody>
    </pages>
    <pages_language_overlay>
        <uid>1</uid>
        <pid>1</pid>
        <title>Root [Dansk]</title>
        <deleted>0</deleted>
        <hidden>0</hidden>
        <sys_language_uid>1</sys_language_uid>
    </pages_language_overlay>
    <tt_content>
        <uid>1</uid>
        <pid>1</pid>
        <header>Test content</header>
        <deleted>0</deleted>
        <t3ver_oid>0</t3ver_oid>
        <t3ver_wsid>0</t3ver_wsid>
    </tt_content>

Functional tests running slow?

If running one functional test takes too much time, please check your MySQL settings. Especially setting innodb_file_per_table = 0 result in much faster test execution. Travis needs about 1s to execute one test.

Additional Resources

Functional Testing of TYPO3 extensions