529f48e423
Signed-off-by: Gökay Şatır <gokaysatir@gmail.com> Change-Id: If07b5f54b60cd0d7a934f97973a487f103702581
312 lines
10 KiB
Text
312 lines
10 KiB
Text
Cypress based test framework for Collabora Online
|
|
===================================================
|
|
|
|
|
|
Installation
|
|
------------
|
|
|
|
You need to have run configure with the --enable-cypress option.
|
|
|
|
In a normal desktop environment you only need to install npm
|
|
packages for running cypress tests. This is done by the build
|
|
system, so running 'make check' will do the basic installation.
|
|
https://docs.cypress.io/guides/getting-started/installing-cypress.html#npm-install
|
|
|
|
For CI you might need to install some additional dependencies:
|
|
https://docs.cypress.io/guides/guides/continuous-integration.html#Dependencies
|
|
|
|
|
|
Running tests
|
|
-------------
|
|
|
|
All tests are part of the make check build expect notebookbar. So you can
|
|
just execute it from the root folder or under the
|
|
cypress_test folder to run cypress tests only.
|
|
|
|
make check
|
|
|
|
IMPORTANT: Before stepping under cypress_test folder
|
|
and running any command listed here, make sure you've
|
|
done a top-level make, so everything is up-to-date.
|
|
Running commands from under cypress_test folder won't
|
|
trigger a rebuild.
|
|
|
|
To run cypress test cases selectively, you need to
|
|
go in to the cypress_test folder first and run one of
|
|
the following commands.
|
|
|
|
To run all desktop tests (this includes notebookbar UI related tests):
|
|
|
|
make check-desktop
|
|
|
|
To run all mobile tests:
|
|
|
|
make check-mobile
|
|
|
|
To run one specific test suite of desktop tests,
|
|
use spec argument with a relative path to
|
|
cypress_test/integration_tests/desktop/:
|
|
|
|
make check-desktop spec=writer/form_field_spec.js
|
|
|
|
To run one specific test suite of mobile tests,
|
|
use spec argument with a relative path to
|
|
cypress_test/integration_tests/mobile/:
|
|
|
|
make check-mobile spec=writer/toolbar_spec.js
|
|
|
|
Running one specific test
|
|
-------------------------
|
|
|
|
To run one test case of a test suite you can use Mocha's
|
|
'only' feature. Just replace the it(...) function with
|
|
it.only(...) in the spec file for the selected test case
|
|
and run the test suite using the spec parameter.
|
|
|
|
For example, to run the test with title 'Apply font name.'
|
|
inside apply_font_spec.js file, you need to add it.only():
|
|
|
|
- it('Apply font name.', function() {
|
|
+ it.only('Apply font name.', function() {
|
|
|
|
Then run the test suite with:
|
|
|
|
make check-mobile spec=writer/apply_font_spec.js
|
|
|
|
Or open the test suite in the interactive test runner:
|
|
|
|
make run-mobile spec=writer/apply_font_spec.js
|
|
|
|
Opening interactive test runner
|
|
-------------------------------
|
|
|
|
Cypress has an interactive test runner application which
|
|
runs the test in the browser. So you can see the result of
|
|
the different steps your test makes in the browser. It's useful
|
|
during writing new tests or checking why an existing
|
|
test fails.
|
|
https://docs.cypress.io/guides/core-concepts/test-runner.html
|
|
|
|
To open desktop tests in the test runner:
|
|
|
|
make run-desktop
|
|
|
|
To open mobile tests in the test runner:
|
|
|
|
make run-mobile
|
|
|
|
To open one specific test suite of desktop tests,
|
|
use spec argument with a relative path to
|
|
cypress_test/integration_tests/desktop/:
|
|
|
|
make run-desktop spec=writer/form_field_spec.js
|
|
|
|
To open one specific test suite of mobile tests,
|
|
use spec argument with a relative path to
|
|
cypress_test/integration_tests/mobile/:
|
|
|
|
make run-mobile spec=writer/toolbar_spec.js
|
|
|
|
During the build we run the tests with Chrome browser, so make sure
|
|
you select Chrome browser on the GUI while checking tests.
|
|
We are using different configuration and environment variables for
|
|
mobile and desktop tests, that's why there are two separate commands
|
|
for them and there is no option to open all the tests in the
|
|
test runner.
|
|
|
|
Running tests to update the help dialog screenshots
|
|
--------------------
|
|
There are two spec files which you can to update the screenshots in help dialog
|
|
1. cypress_test/integration_tests/desktop/writer/help_dialog_update_spec.js
|
|
2. cypress_test/integration_tests/desktop/calc/help_dialog_update_spec.js
|
|
|
|
You can run the specfile individually using:
|
|
1. make -C cypress_test UPDATE_SCREENSHOT=true check-desktop spec=writer/help_dialog_update_spec.js
|
|
2. make -C cypress_test UPDATE_SCREENSHOT=true check-desktop spec=calc/help_dialog_update_spec.js
|
|
|
|
If UPDATE_SCREENSHOT is true then only cypress will run the spec file and replace the original screenshot with new screenshot, the default value of UPDATE_SCREENSHOT is false
|
|
|
|
Interference testing
|
|
--------------------
|
|
|
|
The idea is to run an existing cypress test while the same document
|
|
is opened by a second user. The second user does some activity
|
|
(e.g. moving cursor, clicking, etc) and we check whether it makes
|
|
any interference with the first user's view. The first user is doing
|
|
the actual test's steps, so we can see if interference breaks the assertions.
|
|
|
|
In Writer, the interfering user moves the cursor left and right. In Calc,
|
|
the interfering user moves the cellcursor left and right. In Impress, it
|
|
just does mouse-clicking next to the slide. All these activities trigger
|
|
view switches in the core, which can hit issues with multiple users.
|
|
|
|
To run all mobile tests with interference testing:
|
|
|
|
make check-interfer-mobile
|
|
|
|
To run one specific mobile test suit:
|
|
|
|
make check-interfer-mobile spec=writer/toolbar_spec.js
|
|
|
|
We can also run the interactive test runner with interference
|
|
testing. In this case, the test user is visible in the browser
|
|
and the interfering user runs in the background. Yon can see
|
|
the activity of the interfering user as another view appears
|
|
in the document moving its cursor left and right.
|
|
To open one specific mobile test suit in interactive test runner:
|
|
|
|
make run-interfer-mobile spec=writer/toolbar_spec.js
|
|
|
|
To run all desktop tests with interference testing:
|
|
|
|
make check-interfer-desktop
|
|
|
|
To run one specific desktop test suit:
|
|
|
|
make check-interfer-desktop spec=writer/form_field_spec.js
|
|
|
|
To open one specific desktop test suit in interactive test runner:
|
|
|
|
make run-interfer-desktop spec=writer/form_field_spec.js
|
|
|
|
The implementation of interference testing can be found in
|
|
cypress_test/integration_tests/common/interference_user_spec.js
|
|
test file.
|
|
|
|
Running tests in different browsers
|
|
-----------------------------------
|
|
|
|
By default, the tests are run with chrome / chromium. If you need to
|
|
run the tests with a different browser, you can use the CYPRESS_BROWSER
|
|
variable:
|
|
|
|
CYPRESS_BROWSER="firefox" make check
|
|
|
|
This variable can be set to any value which is accepted by cypress
|
|
--browser command line argument:
|
|
https://docs.cypress.io/guides/guides/command-line.html#cypress-run-browser-lt-browser-name-or-path-gt
|
|
|
|
Running tests with nextcloud integration
|
|
----------------------------------------
|
|
|
|
You can run any test runner command in an NC environment with setting
|
|
CYPRESS_INTEGRATION environment variable. For example:
|
|
|
|
CYPRESS_INTEGRATION="nextcloud" make check
|
|
|
|
Prerequisites:
|
|
* Need a local NC 20 installation connected with Collabora Online
|
|
* NC should be available at http://localhost/nextcloud/ (no ssl)
|
|
* Need an NC user with 'cypress_test' as user name and password.
|
|
|
|
Limitations:
|
|
* NC integration uses iframes which makes harder to test with cypress.
|
|
* cy.document() and cy.window() not properly works by now.
|
|
|
|
Running tests with php-proxy
|
|
----------------------------
|
|
|
|
You can run any test runner command with php-proxy with setting
|
|
CYPRESS_INTEGRATION environment variable. For example:
|
|
|
|
CYPRESS_INTEGRATION="php-proxy" make check
|
|
|
|
Prerequisites:
|
|
* Download php-proxy script:
|
|
https://github.com/CollaboraOnline/richdocumentscode/blob/master/proxy.php
|
|
* Put it under /srv/www/htdocs/richproxy/ folder.
|
|
|
|
Other useful flags
|
|
------------------
|
|
|
|
1. ENABLE_VIDEO_REC
|
|
|
|
You can enable video recording during running a test suite in headless mode.
|
|
So you can check visually what happens in the browser during the test.
|
|
Unfortunately this does not work for multi-user tests.
|
|
|
|
ENABLE_VIDEO_REC="1" make check-mobile spec=writer/apply_font_spec.js
|
|
|
|
2. ENABLE_CONSOLE_LOG
|
|
|
|
With this flag we can enable logging for console.error() messages and so they
|
|
will be displayed in the command line. It's useful when we can reproduce a test
|
|
failure only in headless mode and we need to debug the client-side JS code.
|
|
|
|
ENABLE_CONSOLE_LOG="1" make check-mobile spec=writer/apply_font_spec.js
|
|
|
|
Known issues
|
|
------------
|
|
|
|
1. Different builddir and sourcedir with symlinks.
|
|
|
|
Cypress has an issue with symlinks:
|
|
https://github.com/cypress-io/cypress/issues/3482
|
|
|
|
We use the related feature only when the build directory
|
|
is different from the source directory. So to avoid this
|
|
issue with the supportFile, you should build in the source
|
|
directory or you should avoid symlinks in the path of the
|
|
build directory.
|
|
|
|
Code coverage
|
|
-------------
|
|
|
|
We use nyc to instrument the code and then cypress code coverage
|
|
plugin is used to generate coverage numbers. This workflow
|
|
is called by the following command:
|
|
|
|
make run-cov
|
|
|
|
The output is put under cypress_test/coverage folder.
|
|
Open the cypress_test/coverage/lcov-report/index.html
|
|
file to get the summary report.
|
|
|
|
The make command above touches the source files under
|
|
browser/dist/src folder, so after testing the coverage
|
|
doing a clean build is a good idea (e.g. make clean).
|
|
|
|
See also this link:
|
|
https://docs.cypress.io/guides/tooling/code-coverage.html
|
|
|
|
Used Packages
|
|
-------------
|
|
|
|
- cypress:
|
|
Cypress integration test framework.
|
|
|
|
- cypress-failed-log:
|
|
This package makes cypress to dump test logs to the command line
|
|
when a test fails. You can write things to this log by calling
|
|
cy.log() in the test code.
|
|
|
|
- cypress-log-to-output:
|
|
This one can be used to dump console.error() messages to the command
|
|
line. To enable this functionality you need to set ENABLE_CONSOLE_LOG
|
|
environment variable. e.g. `ENABLE_CONSOLE_LOG="1" make check-mobile`.
|
|
|
|
- cypress-select-tests:
|
|
We can filter out tests or test suites before execution using this
|
|
package. Now, it is used to filter out tests based on the core's version.
|
|
See plugins/blacklists.js.
|
|
|
|
- cypress-wait-until:
|
|
Introduces cy.waitUntil() command which can be used as a while loop.
|
|
We can't write loops in a cypress test otherwise. It's useful when
|
|
the cypress tools can't be used to wait on something.
|
|
|
|
- eslint:
|
|
A JS linter tool for identifying patterns in JavaScript code. We use this
|
|
to make sure code conventions are met. Run by make check and make run-*.
|
|
|
|
- eslint-plugin-cypress-rules:
|
|
This is our own eslint plugin to catch cypress specific patterns in the
|
|
test code.
|
|
|
|
- get-port-cli:
|
|
Used by the build system to find an available port to use as coolwsd's
|
|
communication port.
|
|
|
|
- wait-on:
|
|
Used by the build system to wait on coolwsd server to start before
|
|
tests are started.
|