a13b1a93bc
When we run cypress tests in parallel (e.g. make check -j4), the tests randomly fail with an uncaught error. The issue might be because the same support file is shared between the different threads, and they might access it at the same time. We try to create different instances of the support file via preprocessing. Let's see whether it makes the issue go away. Issue's error message was: "An uncaught error was detected outside of a test: SyntaxError: The following error originated from your test code, not from Cypress. Unexpected end of input" Signed-off-by: Tamás Zolnai <tamas.zolnai@collabora.com> Change-Id: I7b256cd686fb2b0e79ab602e95d197388a5aadca |
||
---|---|---|
.. | ||
data | ||
eslint_plugin | ||
integration_tests | ||
plugins | ||
support | ||
.eslintignore | ||
.eslintrc | ||
.gitignore | ||
cypress.json | ||
Makefile.am | ||
package.json | ||
README | ||
run_iteratively.sh | ||
run_parallel.sh |
Cypress based test framework for Collabora Online =================================================== Installation ------------ You need to have run configure with the --enable-cypress option. In a normal desktop environment you only need to install npm packages for running cypress tests. This is done by the build system, so running 'make check' will do the basic installation. https://docs.cypress.io/guides/getting-started/installing-cypress.html#npm-install For CI you might need to install some additional dependencies: https://docs.cypress.io/guides/guides/continuous-integration.html#Dependencies Running tests ------------- All tests are part of the make check build. So you can just execute it from the root folder or under the cypress_test folder to run cypress tests only. make check IMPORTANT: Before stepping under cypress_test folder and running any command listed here, make sure you've done a top-level make, so everything is up-to-date. Running commands from under cypress_test folder won't trigger a rebuild. To run cypress test cases selectively, you need to go in to the cypress_test folder first and run one of the following commands. To run all desktop tests: make check-desktop To run all mobile tests: make check-mobile To run one specific test suite of desktop tests, use spec argument with a relative path to cypress_test/integration_tests/desktop/: make check-desktop spec=writer/form_field_spec.js To run one specific test suite of mobile tests, use spec argument with a relative path to cypress_test/integration_tests/mobile/: make check-mobile spec=writer/toolbar_spec.js Running one specific test ------------------------- To run one test case of a test suite you can use Mocha's 'only' feature. Just replace the it(...) function with it.only(...) in the spec file for the selected test case and run the test suite using the spec parameter. For example, to run the test with title 'Apply font name.' inside apply_font_spec.js file, you need to add it.only(): - it('Apply font name.', function() { + it.only('Apply font name.', function() { Then run the test suite with: make check-mobile spec=writer/apply_font_spec.js Or open the test suite in the interactive test runner: make run-mobile spec=writer/apply_font_spec.js Openning interactive test runner -------------------------------- Cypress has an interactive test runner application which runs the test in the browser. So you can see the result of the different steps your test makes in the browser. It's useful during writing new tests or checking why an existing test fails. https://docs.cypress.io/guides/core-concepts/test-runner.html To open desktop tests in the test runner: make run-desktop To open mobile tests in the test runner: make run-mobile To open one specific test suite of desktop tests, use spec argument with a relative path to cypress_test/integration_tests/desktop/: make run-desktop spec=writer/form_field_spec.js To open one specific test suite of mobile tests, use spec argument with a relative path to cypress_test/integration_tests/mobile/: make run-mobile spec=writer/toolbar_spec.js During the build we run the tests with Chrome browser, so make sure you select Chrome browser on the GUI while checking tests. We are using different configuration and environment variables for mobile and desktop tests, that's why there are two separate commands for them and there is no option to open all the tests in the test runner. Running tests in different browsers ----------------------------------- By default, the tests are run with chrome / chromium. If you need to run the tests with a different browser, you can use the CYPRESS_BROWSER variable: CYPRESS_BROWSER="firefox" make check This variable can be set to any value which is accepted by cypress --browser command line argument: https://docs.cypress.io/guides/guides/command-line.html#cypress-run-browser-lt-browser-name-or-path-gt Running tests with nextcloud integration ---------------------------------------- You can run any test runner command in an NC environment with setting CYPRESS_INTEGRATION environment variable. For example: CYPRESS_INTEGRATION="nextcloud" make check Prerequisites: * Need a local NC 20 installation connected with Collabora Online * NC should be available at http://localhost/nextcloud/ (no ssl) * Need an NC user with 'cypress_test' as user name and password. Limitations: * NC integration uses iframes which makes harder to test with cypress. * cy.document() and cy.window() not properly works by now. Running tests with php-proxy ---------------------------- You can run any test runner command with php-proxy with setting CYPRESS_INTEGRATION environment variable. For example: CYPRESS_INTEGRATION="php-proxy" make check Prerequisites: * Download php-proxy script: https://github.com/CollaboraOnline/richdocumentscode/blob/master/proxy.php * Put it under /srv/www/htdocs/richproxy/ folder. Known issues ------------ 1. Different builddir and sourcedir with symlinks. Cypress has an issue with symlinks: https://github.com/cypress-io/cypress/issues/3482 We use the related feature only when the build directory is different from the source directory. So to avoid this issue with the supportFile, you should build in the source directory or you should avoid symlinks in the path of the build directory. Code coverage ------------- We use nyc to instrument the code and then cypress code coverage plugin is used to generate coverage numbers. This workflow is called by the following command: make run-cov The output is put under cypress_test/covarage folder. Open the cypress_test/covarage/lcov-report/index.html file to get the summary report. The make command above touches the source files under loleaflet/dist/src folder, so after testing the coverage doing a clean build is a good idea (e.g. make clean). See also this link: https://docs.cypress.io/guides/tooling/code-coverage.html Used Packages ------------- - cypress: Cypress integration test framework. - cypress-failed-log: This package makes cypress to dump test logs to the command line when a test fails. You can write things to this log by calling cy.log() in the test code. - cypress-log-to-output: This one can be used to dump console.error() messages to the command line. To enable this functionality you need to set ENABLE_CONSOLE_LOG environment variable. e.g. `ENABLE_CONSOLE_LOG="1" make check-mobile`. - cypress-select-tests: We can filter out tests or test suites before execution using this package. Now, it is used to filter out tests based on the core's version. See plugins/blacklists.js. - cypress-wait-until: Introduces cy.waitUntil() command which can be used as a while loop. We can't write loops in a cypress test otherwise. It's useful when the cypress tools can't be used to wait on something. - eslint: A JS linter tool for identifying patterns in JavaScript code. We use this to make sure code conventions are met. Run by make check and make run-*. - eslint-plugin-cypress-rules: This is our own eslint plugin to catch cypress specific patterns in the test code. - get-port-cli: Used by the build system to find an available port to use as loolwsd's communication port. - wait-on: Used by the build system to wait on loolwsd server to start before tests are started.