235 lines
7.3 KiB
Text
235 lines
7.3 KiB
Text
Cypress based test framework for Collabora Online
|
|
===================================================
|
|
|
|
|
|
Installation
|
|
------------
|
|
|
|
You need to have run configure with the --enable-cypress option.
|
|
|
|
In a normal desktop environment you only need to install npm
|
|
packages for running cypress tests. This is done by the build
|
|
system, so running 'make check' will do the basic installation.
|
|
https://docs.cypress.io/guides/getting-started/installing-cypress.html#npm-install
|
|
|
|
For CI you might need to install some additional dependencies:
|
|
https://docs.cypress.io/guides/guides/continuous-integration.html#Dependencies
|
|
|
|
|
|
Running tests
|
|
-------------
|
|
|
|
All tests are part of the make check build. So you can
|
|
just execute it from the root folder or under the
|
|
cypress_test folder to run cypress tests only.
|
|
|
|
make check
|
|
|
|
IMPORTANT: Before stepping under cypress_test folder
|
|
and running any command listed here, make sure you've
|
|
done a top-level make, so everything is up-to-date.
|
|
Running commands from under cypress_test folder won't
|
|
trigger a rebuild.
|
|
|
|
To run cypress test cases selectively, you need to
|
|
go in to the cypress_test folder first and run one of
|
|
the following commands.
|
|
|
|
To run all desktop tests:
|
|
|
|
make check-desktop
|
|
|
|
To run all mobile tests:
|
|
|
|
make check-mobile
|
|
|
|
To run one specific test suite of desktop tests,
|
|
use spec argument with a relative path to
|
|
cypress_test/integration_tests/desktop/:
|
|
|
|
make check-desktop spec=writer/form_field_spec.js
|
|
|
|
To run one specific test suite of mobile tests,
|
|
use spec argument with a relative path to
|
|
cypress_test/integration_tests/mobile/:
|
|
|
|
make check-mobile spec=writer/toolbar_spec.js
|
|
|
|
|
|
Running one specific test
|
|
-------------------------
|
|
|
|
To run one test case of a test suite you can use Mocha's
|
|
'only' feature. Just replace the it(...) function with
|
|
it.only(...) in the spec file for the selected test case
|
|
and run the test suite using the spec parameter.
|
|
|
|
For example, to run the test with title 'Apply font name.'
|
|
inside apply_font_spec.js file, you need to add it.only():
|
|
|
|
- it('Apply font name.', function() {
|
|
+ it.only('Apply font name.', function() {
|
|
|
|
Then run the test suite with:
|
|
|
|
make check-mobile spec=writer/apply_font_spec.js
|
|
|
|
Or open the test suite in the interactive test runner:
|
|
|
|
make run-mobile spec=writer/apply_font_spec.js
|
|
|
|
Openning interactive test runner
|
|
--------------------------------
|
|
|
|
Cypress has an interactive test runner application which
|
|
runs the test in the browser. So you can see the result of
|
|
the different steps your test makes in the browser. It's useful
|
|
during writing new tests or checking why an existing
|
|
test fails.
|
|
https://docs.cypress.io/guides/core-concepts/test-runner.html
|
|
|
|
To open desktop tests in the test runner:
|
|
|
|
make run-desktop
|
|
|
|
To open mobile tests in the test runner:
|
|
|
|
make run-mobile
|
|
|
|
To open one specific test suite of desktop tests,
|
|
use spec argument with a relative path to
|
|
cypress_test/integration_tests/desktop/:
|
|
|
|
make run-desktop spec=writer/form_field_spec.js
|
|
|
|
To open one specific test suite of mobile tests,
|
|
use spec argument with a relative path to
|
|
cypress_test/integration_tests/mobile/:
|
|
|
|
make run-mobile spec=writer/toolbar_spec.js
|
|
|
|
During the build we run the tests with Chrome browser, so make sure
|
|
you select Chrome browser on the GUI while checking tests.
|
|
We are using different configuration and environment variables for
|
|
mobile and desktop tests, that's why there are two separate commands
|
|
for them and there is no option to open all the tests in the
|
|
test runner.
|
|
|
|
Running tests in different browsers
|
|
-----------------------------------
|
|
|
|
By default, the tests are run with chrome / chromium. If you need to
|
|
run the tests with a different browser, you can use the CYPRESS_BROWSER
|
|
variable:
|
|
|
|
CYPRESS_BROWSER="firefox" make check
|
|
|
|
This variable can be set to any value which is accepted by cypress
|
|
--browser command line argument:
|
|
https://docs.cypress.io/guides/guides/command-line.html#cypress-run-browser-lt-browser-name-or-path-gt
|
|
|
|
Running tests with nextcloud integration
|
|
----------------------------------------
|
|
|
|
You can run any test runner command in an NC environment with setting
|
|
CYPRESS_INTEGRATION environment variable. For example:
|
|
|
|
CYPRESS_INTEGRATION="nextcloud" make check
|
|
|
|
Prerequisites:
|
|
* Need a local NC installation connected with Collabora Online
|
|
* NC should be available at http://localhost/nextcloud/ (no ssl)
|
|
* Need an NC user with 'cypress_test' as user name and password.
|
|
|
|
Limitations:
|
|
* NC integration uses iframes which makes harder to test with cypress.
|
|
* cy.document() and cy.window() not properly works by now.
|
|
|
|
Running tests with php-proxy
|
|
----------------------------
|
|
|
|
You can run any test runner command with php-proxy with setting
|
|
CYPRESS_INTEGRATION environment variable. For example:
|
|
|
|
CYPRESS_INTEGRATION="php-proxy" make check
|
|
|
|
Prerequisites:
|
|
* Download php-proxy script:
|
|
https://github.com/CollaboraOnline/richdocumentscode/blob/master/proxy.php
|
|
* Put it under /srv/www/htdocs/richproxy/ folder.
|
|
|
|
Known issues
|
|
------------
|
|
|
|
1. Different builddir and sourcedir with symlinks.
|
|
|
|
Cypress has an issue with symlinks:
|
|
https://github.com/cypress-io/cypress/issues/3482
|
|
|
|
We use the related feature only when the build directory
|
|
is different from the source directory. So to avoid this
|
|
issue with the supportFile, you should build in the source
|
|
directory or you should avoid symlinks in the path of the
|
|
build directory.
|
|
|
|
Code coverage
|
|
-------------
|
|
|
|
We use nyc to instrument the code and then cypress code coverage
|
|
plugin is used to generate coverage numbers. This workflow
|
|
is called by the following command:
|
|
|
|
make run-cov
|
|
|
|
The output is put under cypress_test/covarage folder.
|
|
Open the cypress_test/covarage/lcov-report/index.html
|
|
file to get the summary report.
|
|
|
|
The make command above touches the source files under
|
|
loleaflet/dist/src folder, so after testing the coverage
|
|
doing a clean build is a good idea (e.g. make clean).
|
|
|
|
See also this link:
|
|
https://docs.cypress.io/guides/tooling/code-coverage.html
|
|
|
|
Used Packages
|
|
-------------
|
|
|
|
- cypress:
|
|
Cypress integration test framework.
|
|
|
|
- cypress-failed-log:
|
|
This package makes cypress to dump test logs to the command line
|
|
when a test fails. You can write things to this log by calling
|
|
cy.log() in the test code.
|
|
|
|
- cypress-log-to-output:
|
|
This one can be used to dump console.error() messages to the command
|
|
line. To enable this functionality you need to set ENABLE_CONSOLE_LOG
|
|
environment variable. e.g. `ENABLE_CONSOLE_LOG="1" make check-mobile`.
|
|
|
|
- cypress-select-tests:
|
|
We can filter out tests or test suites before execution using this
|
|
package. Now, it is used to filter out tests based on the core's version.
|
|
See plugins/blacklists.js.
|
|
|
|
- cypress-wait-until:
|
|
Introduces cy.waitUntil() command which can be used as a while loop.
|
|
We can't write loops in a cypress test otherwise. It's useful when
|
|
the cypress tools can't be used to wait on something.
|
|
|
|
- eslint:
|
|
A JS linter tool for identifying patterns in JavaScript code. We use this
|
|
to make sure code conventions are met. Run by make check and make run-*.
|
|
|
|
- eslint-plugin-cypress-rules:
|
|
This is our own eslint plugin to catch cypress specific patterns in the
|
|
test code.
|
|
|
|
- get-port-cli:
|
|
Used by the build system to find an available port to use as loolwsd's
|
|
communication port.
|
|
|
|
- wait-on:
|
|
Used by the build system to wait on loolwsd server to start before
|
|
tests are started.
|