Commit graph

127 commits

Author SHA1 Message Date
Tamás Zolnai
4f9169bf8c cypress: remove content of cypress folder before test run.
Signed-off-by: Tamás Zolnai <tamas.zolnai@collabora.com>
Change-Id: Ic4e637903df6768c42fcd404563455e19fa86cda
2021-01-04 16:30:40 +01:00
Tamás Zolnai
63ee524010 cypress: disable welcome dialog via loolwsd config.
Signed-off-by: Tamás Zolnai <tamas.zolnai@collabora.com>
Change-Id: I15c060e5eac1993116828dfaefe8916030fe2b63
2020-12-30 19:49:42 +01:00
Tamás Zolnai
09fbbf43b0 cypress: extract clean-up code into a function.
Signed-off-by: Tamás Zolnai <tamas.zolnai@collabora.com>
Change-Id: I572b164f89023cd66fca2d392b830417eb6a9d74
2020-12-29 10:42:51 +01:00
Tamás Zolnai
d6663d807f cypress: record the loolwsd's command line output.
So we can catch crashes / debug messages of core code.

Signed-off-by: Tamás Zolnai <tamas.zolnai@collabora.com>
Change-Id: Ib11426e7fdd8f219be4715b3f1e2983ad33d7f2b
2020-12-18 21:20:27 +01:00
Tamás Zolnai
e8bad8ed60 cypress: add 'run-interfer-desktop' make command.
Signed-off-by: Tamás Zolnai <tamas.zolnai@collabora.com>
Change-Id: Ibfb2bd3e6158da72a160cdfdc8507f9887cbdcfa
2020-12-10 13:36:34 +01:00
Tamás Zolnai
f917ece93a cypress: add 'check-interfer-desktop' make command.
To run interference tests for desktop tests.

Signed-off-by: Tamás Zolnai <tamas.zolnai@collabora.com>
Change-Id: Ibe06399846a37345ab08fd830b6350fe12ba2e37
2020-12-10 13:36:34 +01:00
Tamás Zolnai
7a190deb21 cypress: restructure makefile: group interference related code.
Signed-off-by: Tamás Zolnai <tamas.zolnai@collabora.com>
Change-Id: I95bb115e41704d2912f3ed48b7e193ea648364f5
2020-12-10 13:36:34 +01:00
Tamás Zolnai
5f80d8290a cypress: fix 'run-multi' make command.
5 seconds gap between the two users was moved from
js code into the makefile in:
9ff0f3e9fe

We need it for interactive run too.

Signed-off-by: Tamás Zolnai <tamas.zolnai@collabora.com>
Change-Id: Ic8aa541ce30f3146a61875ea66c904e65f0d6cf9
2020-12-08 14:24:18 +01:00
Tamás Zolnai
7422663234 cypress: put loolwsd log under cypress_test/workdir folder.
So it's easier to find the loolwsd.log of the last
called command.

Signed-off-by: Tamás Zolnai <tamas.zolnai@collabora.com>
Change-Id: I4d597a3d6683360ef03f15061a68a30db8ae498e
2020-12-08 14:24:18 +01:00
Tamás Zolnai
bd6cfefc29 cypress: add 'run-interfer-mobile' make command.
For running interference test in interactive mode.
The interference user is run in the background and
the user running the test code is displayed in the
interactive test runner.

Signed-off-by: Tamás Zolnai <tamas.zolnai@collabora.com>
Change-Id: I3595a618f78603ed3e5321969a7395b60555470b
2020-12-08 14:24:18 +01:00
Tamás Zolnai
a95de9a799 cypress: remove this prototype of interference testing.
Signed-off-by: Tamás Zolnai <tamas.zolnai@collabora.com>
Change-Id: Ibb7c84a2c701770cb4064c04223fc1fe041d624e
2020-12-01 13:25:20 +01:00
Tamás Zolnai
52bb718620 cypress: introduce interference testing for mobile tests.
Added `make check-interfer-mobile` command for running
interference tests. Now it works with writer tests.
I'll generalize it later.


Signed-off-by: Tamás Zolnai <tamas.zolnai@collabora.com>
Change-Id: I2181a8653d61cc6ed873e2887a3e42767f46f177
2020-12-01 13:25:20 +01:00
Tamás Zolnai
f399beaf65 cypress: multi-user: avoid running more tests in parallel.
We have more users running already for one test, better
to avoid side-ffects coming from running more multi-user
test in parallel (they are using the same loolwsd).

Change-Id: I1cbed36a3ffe2825c384b525da361c4cfe781b38
Signed-off-by: Tamás Zolnai <tamas.zolnai@collabora.com>
2020-11-19 11:02:29 +01:00
Tamás Zolnai
0f5042f76f cypress: multi-user: move retries config to a shared place.
Change-Id: I7939395b88c7a74a1e0016ec78c95ef46a5a31ec
Signed-off-by: Tamás Zolnai <tamas.zolnai@collabora.com>
2020-11-19 11:02:29 +01:00
Tamás Zolnai
9ff0f3e9fe cypress: multi-user: move this document opening gap to the make file.
Change-Id: Ib37d8d29e8965712975e27a4577a4f1b6a0e35a0
Signed-off-by: Tamás Zolnai <tamas.zolnai@collabora.com>
2020-11-19 11:02:29 +01:00
Tamás Zolnai
7886d63349 cypress: remove php-proxy simulation.
We have a way of running the tests against
the actualy php-proxy script:

CYPRESS_INTEGRATION="php-proxy" make check

So let's remove the C++ implementation, so we don't
need to maintain a duplicated code.

Change-Id: Ie095a70a8175cc8cb4894905826d13762127b148
Signed-off-by: Tamás Zolnai <tamas.zolnai@collabora.com>
2020-10-26 19:28:31 +01:00
Tamás Zolnai
5b0676bc3a cypress: protoype an interference test.
The idea it to run an existing cypress test while
the same document is opened by a second user. The second
user does some activity (e.g. moving cursor, clicking, etc)
and we check whether it makes any interference with the first
user's view. The first user is doing the actual test's steps,
so we can see if interference breaks the assertions.

Now it's just a prototype for testing it in the CI setup.

Change-Id: I3e02c037754862d628c73bbb4b2d82ea84b8a719
2020-10-23 16:09:00 +02:00
Andrea Gelmini
596b26c7d9 Fix typo in code
Change-Id: I0ef8fe7ff382d7cb295aa8054721448d9bf8e477
2020-10-07 12:46:37 +02:00
Tamás Zolnai
55c75144f7 cypress: fail on missing LO installation.
Jenkins sometimes does not work properly and core
installation is not accessible for online build.
In this case the test build was passing without
actually run the tests. Fail on this instead to
avoid getting untested patches in the code repo.

Change-Id: I35c49562dfeaeda5f6c8aa4bd3a746b1f04de3ea
2020-10-05 13:32:12 +02:00
Tamás Zolnai
5c2f5eac2e cypress: use built-in test retries.
It's more effective, since it reruns only the failing test,
not the whole test suite. Also it has a nicer output.

Change-Id: I8a50a20366b76a5cedb4e9383220c53f367c5a68
Reviewed-on: https://gerrit.libreoffice.org/c/online/+/102995
Tested-by: Jenkins CollaboraOffice <jenkinscollaboraoffice@gmail.com>
Reviewed-by: Tamás Zolnai <tamas.zolnai@collabora.com>
2020-09-18 12:50:45 +02:00
Tamás Zolnai
f6d5cf9c3f cypress: support running tests with php-proxy.
Change-Id: I9fe4a974582e0475026f6798a338bae033e6d7e6
Reviewed-on: https://gerrit.libreoffice.org/c/online/+/102733
Tested-by: Jenkins CollaboraOffice <jenkinscollaboraoffice@gmail.com>
Reviewed-by: Tamás Zolnai <tamas.zolnai@collabora.com>
2020-09-15 15:40:22 +02:00
Tamás Zolnai
ebd040cacb cypress: support running test with nextcloud integration.
Change-Id: I0b2255225a6c0eca5317559df33bb12b841dac08
Reviewed-on: https://gerrit.libreoffice.org/c/online/+/102260
Tested-by: Jenkins CollaboraOffice <jenkinscollaboraoffice@gmail.com>
Reviewed-by: Tamás Zolnai <tamas.zolnai@collabora.com>
2020-09-10 14:03:24 +02:00
Tamás Zolnai
0d7a58aa57 cypress: add 'make check-proxy-mobile' command.
To run mobile test with php-proxy simulation.
It's an initial command, should be improved later,
after php proxy simulation nicely works with the
tests.

Change-Id: Id9d2bacfbf09ca34a2c84e0d32808749df4c1909
Reviewed-on: https://gerrit.libreoffice.org/c/online/+/102321
Tested-by: Jenkins CollaboraOffice <jenkinscollaboraoffice@gmail.com>
Reviewed-by: Tamás Zolnai <tamas.zolnai@collabora.com>
2020-09-09 15:50:35 +02:00
Tamás Zolnai
433b108c50 cypress: simplify this.
Change-Id: I4b055f13caebd7f034c4aa8aea724aff300c113d
Reviewed-on: https://gerrit.libreoffice.org/c/online/+/101766
Tested-by: Jenkins
Tested-by: Jenkins CollaboraOffice <jenkinscollaboraoffice@gmail.com>
Reviewed-by: Tamás Zolnai <tamas.zolnai@collabora.com>
2020-09-01 00:20:18 +02:00
Tamás Zolnai
d6005f0947 cypress: add a way to run tests with different browsers.
e.g. CYPRESS_BROWSER="firefox" make check

Change-Id: I3017179152c6e875df667da5081abb0c8cde60b1
Reviewed-on: https://gerrit.libreoffice.org/c/online/+/101733
Tested-by: Jenkins
Tested-by: Jenkins CollaboraOffice <jenkinscollaboraoffice@gmail.com>
Reviewed-by: Tamás Zolnai <tamas.zolnai@collabora.com>
2020-08-31 15:40:44 +02:00
Tamás Zolnai
b56cff6520 cypress: introduce make check-firefox command.
To run all cypress tests inside firefox browser.

Change-Id: I965666e8b6b7aad5a45d1fef4fc9a078aa30615d
Reviewed-on: https://gerrit.libreoffice.org/c/online/+/100441
Tested-by: Jenkins
Tested-by: Jenkins CollaboraOffice <jenkinscollaboraoffice@gmail.com>
Reviewed-by: Tamás Zolnai <tamas.zolnai@collabora.com>
2020-08-11 12:08:58 +02:00
Tamás Zolnai
cfc62f864d cypress: use actual LOOLWSD version hash.
The related issue was fixed upstream.

Change-Id: I8b1191d3c0d5543233ace9322a7d768b3424d62d
Reviewed-on: https://gerrit.libreoffice.org/c/online/+/100435
Tested-by: Jenkins
Tested-by: Jenkins CollaboraOffice <jenkinscollaboraoffice@gmail.com>
Reviewed-by: Tamás Zolnai <tamas.zolnai@collabora.com>
2020-08-10 15:37:46 +02:00
Tamás Zolnai
9c312a4cc6 cypress: group multi-user tests based on component.
Change-Id: Ic2689b8e05c9827807585a4a718f112e23d1e5a5
Reviewed-on: https://gerrit.libreoffice.org/c/online/+/100324
Tested-by: Jenkins
Tested-by: Jenkins CollaboraOffice <jenkinscollaboraoffice@gmail.com>
Reviewed-by: Tamás Zolnai <tamas.zolnai@collabora.com>
2020-08-07 15:37:14 +02:00
Tamás Zolnai
8c6e24b565 cypress: simplify interactive runner rules.
Change-Id: I491eb0c78429981b05a888975380857f7bce8dcd
Reviewed-on: https://gerrit.libreoffice.org/c/online/+/100168
Tested-by: Jenkins
Tested-by: Jenkins CollaboraOffice <jenkinscollaboraoffice@gmail.com>
Reviewed-by: Tamás Zolnai <tamas.zolnai@collabora.com>
2020-08-05 14:56:39 +02:00
Tamás Zolnai
c34aca2588 cypress: add make run-multi command.
To run multi-user tests in interactive test runner.
It has two non-optional parameter. 'spec' defines the
test file and 'user' defines which user is run in the
interactive test runner. The other user is run in the
background.

Change-Id: I8612c201d7d3aeb2bdadd9d3786fb8eb0b60ab55
Reviewed-on: https://gerrit.libreoffice.org/c/online/+/100167
Tested-by: Jenkins
Tested-by: Jenkins CollaboraOffice <jenkinscollaboraoffice@gmail.com>
Reviewed-by: Tamás Zolnai <tamas.zolnai@collabora.com>
2020-08-05 14:56:17 +02:00
Tamás Zolnai
f273888598 cypress: update run-cov command to run multi-user tests too.
Change-Id: If13dea05f5a506331f9f76177655ed65716d391e
Reviewed-on: https://gerrit.libreoffice.org/c/online/+/100165
Tested-by: Jenkins CollaboraOffice <jenkinscollaboraoffice@gmail.com>
Reviewed-by: Tamás Zolnai <tamas.zolnai@collabora.com>
2020-08-05 13:48:16 +02:00
Tamás Zolnai
036cc02d00 cypress: add slide operations multi-user test.
Change-Id: I91a4a1ab581080d4b22c7b03224da4f48e157312
Reviewed-on: https://gerrit.libreoffice.org/c/online/+/100006
Tested-by: Jenkins
Tested-by: Jenkins CollaboraOffice <jenkinscollaboraoffice@gmail.com>
Reviewed-by: Tamás Zolnai <tamas.zolnai@collabora.com>
2020-08-03 18:20:05 +02:00
Tamás Zolnai
e65ae5de93 cypress: add sheet operations multi-user test.
Change-Id: Iea31a01d5c85ada481711f771c3489c6aace6e61
Reviewed-on: https://gerrit.libreoffice.org/c/online/+/99979
Tested-by: Jenkins
Tested-by: Jenkins CollaboraOffice <jenkinscollaboraoffice@gmail.com>
Reviewed-by: Tamás Zolnai <tamas.zolnai@collabora.com>
2020-08-03 15:33:08 +02:00
Tamás Zolnai
1604fa99f8 cypress: update sidebar visibility test.
Test less which might make this test more stable.

Change-Id: Icf11af96bc5d21a14f0a74a641ffed3f91b6e482
Reviewed-on: https://gerrit.libreoffice.org/c/online/+/99783
Tested-by: Jenkins
Tested-by: Jenkins CollaboraOffice <jenkinscollaboraoffice@gmail.com>
Reviewed-by: Tamás Zolnai <tamas.zolnai@collabora.com>
2020-07-31 12:51:22 +02:00
Tamás Zolnai
5b106d9edd cypress: fix-up check-multi command.
We need to introduce separate rules to dump
the log and to dump out the error message.
Otherwise GNU make evaluation breaks the workflow.
It evaluates $(wildcard $(ERROR_LOG)) before the
test were actually executed. So first run the tests,
then dump the error output and then go into error state.

Change-Id: I0f8e5de6800f0691cca14019819972437a741382
Reviewed-on: https://gerrit.libreoffice.org/c/online/+/99781
Tested-by: Jenkins
Tested-by: Jenkins CollaboraOffice <jenkinscollaboraoffice@gmail.com>
Reviewed-by: Tamás Zolnai <tamas.zolnai@collabora.com>
2020-07-30 15:26:19 +02:00
Tamás Zolnai
02b4bd2fd8 cypress: disable this unstable test for now.
Change-Id: I9c339ac23b2ec9a732f4ceb9efe4b66024b70f23
Reviewed-on: https://gerrit.libreoffice.org/c/online/+/99775
Tested-by: Tamás Zolnai <tamas.zolnai@collabora.com>
Reviewed-by: Tamás Zolnai <tamas.zolnai@collabora.com>
2020-07-30 11:29:24 +02:00
Tamás Zolnai
fd6b984a3c cypress: introduce make check-multi command.
To run multi-user tests only.
- make check-multi runs all multi-user tests.
- make check-multi spec=sidebar_visibility runs
only the test pair with the specified name.

Change-Id: Idada04960a6a2d33c7f36ef57bc1d6a1d1cffa99
Reviewed-on: https://gerrit.libreoffice.org/c/online/+/99682
Tested-by: Jenkins
Tested-by: Jenkins CollaboraOffice <jenkinscollaboraoffice@gmail.com>
Reviewed-by: Tamás Zolnai <tamas.zolnai@collabora.com>
2020-07-29 14:26:51 +02:00
Tamás Zolnai
27d3055dcf cypress: add a simultaneous typing multi-user test.
Change-Id: Ib4c410f95443c3775626465d474f52499705da97
Reviewed-on: https://gerrit.libreoffice.org/c/online/+/99676
Tested-by: Jenkins
Tested-by: Jenkins CollaboraOffice <jenkinscollaboraoffice@gmail.com>
Reviewed-by: Tamás Zolnai <tamas.zolnai@collabora.com>
2020-07-29 13:00:28 +02:00
Tamás Zolnai
8ee8f1e1bd cypress: add sidebar visibility multi-user test.
Change-Id: I790a2c149face5fc4899f183833c99471ceb20e1
Reviewed-on: https://gerrit.libreoffice.org/c/online/+/99631
Tested-by: Jenkins
Tested-by: Jenkins CollaboraOffice <jenkinscollaboraoffice@gmail.com>
Reviewed-by: Tamás Zolnai <tamas.zolnai@collabora.com>
2020-07-28 18:31:04 +02:00
Tamás Zolnai
f980eccde2 cypress: update multi-user test case.
Change-Id: I8e73c41df35b92305e31fd3c5e35d5c15ae6e8e8
Reviewed-on: https://gerrit.libreoffice.org/c/online/+/99518
Tested-by: Jenkins CollaboraOffice <jenkinscollaboraoffice@gmail.com>
Tested-by: Jenkins
Reviewed-by: Tamás Zolnai <tamas.zolnai@collabora.com>
2020-07-27 23:19:23 +02:00
Tamás Zolnai
bda83c263b cypress: better way of running multi-user tests.
Before we used the GNU make implicit parallelization
which triggered by recipe structure.
Instead of that we use linux's bakcground process
feature and run one client in the background and
the other one in the foreground.

Change-Id: I9de899686ae67b47eee005f157b6622ae4c5fe33
Reviewed-on: https://gerrit.libreoffice.org/c/online/+/99484
Tested-by: Jenkins
Tested-by: Jenkins CollaboraOffice <jenkinscollaboraoffice@gmail.com>
Reviewed-by: Tamás Zolnai <tamas.zolnai@collabora.com>
2020-07-27 23:19:14 +02:00
Tamás Zolnai
09abe6724f cypress: extract parallel test running code to a shell script file.
Change-Id: Ice856737b5b4bcfc6a202b507a03fdfa16938c25
Reviewed-on: https://gerrit.libreoffice.org/c/online/+/99364
Tested-by: Jenkins CollaboraOffice <jenkinscollaboraoffice@gmail.com>
Tested-by: Jenkins
Reviewed-by: Tamás Zolnai <tamas.zolnai@collabora.com>
2020-07-27 23:18:50 +02:00
Tamás Zolnai
5b70bcd4d3 cypress: avoid spaces in core version string.
It causes issues when passing this string as a
command line argument.

Change-Id: I9af569c8ab941f392800c3c678085c384bec2908
Reviewed-on: https://gerrit.libreoffice.org/c/online/+/99374
Tested-by: Jenkins
Tested-by: Jenkins CollaboraOffice <jenkinscollaboraoffice@gmail.com>
Reviewed-by: Tamás Zolnai <tamas.zolnai@collabora.com>
2020-07-24 15:30:29 +02:00
Tamás Zolnai
514aca9e01 cypress: multi-user test case.
Change-Id: If2ee00c1e43b3cd854b83c762b74b2b8fee6d821
Reviewed-on: https://gerrit.libreoffice.org/c/online/+/99073
Tested-by: Jenkins
Tested-by: Jenkins CollaboraOffice <jenkinscollaboraoffice@gmail.com>
Reviewed-by: Tamás Zolnai <tamas.zolnai@collabora.com>
2020-07-23 17:29:03 +02:00
Tamás Zolnai
d613f8ad1d cypress: make jails folder if does not exist.
Newly jails folder is cleaned up after a run.

Change-Id: Ia4f3eb26f61c5f14304e9ae6a3d5b5f60a0ee592
Reviewed-on: https://gerrit.libreoffice.org/c/online/+/98204
Tested-by: Jenkins CollaboraOffice <jenkinscollaboraoffice@gmail.com>
Reviewed-by: Tamás Zolnai <tamas.zolnai@collabora.com>
2020-07-06 13:08:05 +02:00
Tamás Zolnai
9afe83e306 cypress: don't clear cypress cache by make check.
It's a global cache. Calling make clean on one branch
can break the build on a different because the
shared cache is removed.

Change-Id: Id03fb6b6fd69e899276abc93c3d907538576d8bd
Reviewed-on: https://gerrit.libreoffice.org/c/online/+/97841
Tested-by: Jenkins
Tested-by: Jenkins CollaboraOffice <jenkinscollaboraoffice@gmail.com>
Reviewed-by: Tamás Zolnai <tamas.zolnai@collabora.com>
2020-07-03 14:30:22 +02:00
Tamás Zolnai
9205b4e309 cypress: improve error matcher.
Change-Id: Idad9b6a37b86160c65e0af25bee844652b83f77b
Reviewed-on: https://gerrit.libreoffice.org/c/online/+/97229
Tested-by: Jenkins CollaboraOffice <jenkinscollaboraoffice@gmail.com>
Reviewed-by: Tamás Zolnai <tamas.zolnai@collabora.com>
2020-06-26 15:54:56 +02:00
Tamás Zolnai
e198daae3d cypress: fix build.
Change-Id: Ie8df2a37312c0bee5827cb9378c2147a6b30bffd
Reviewed-on: https://gerrit.libreoffice.org/c/online/+/97228
Tested-by: Jenkins CollaboraOffice <jenkinscollaboraoffice@gmail.com>
Reviewed-by: Tamás Zolnai <tamas.zolnai@collabora.com>
2020-06-26 15:26:12 +02:00
Tamás Zolnai
2a48c97a84 cypress: improve coverage run.
Change-Id: I48637063c998a7c72da8e610ae0c8cc18c61a52a
Reviewed-on: https://gerrit.libreoffice.org/c/online/+/97152
Tested-by: Jenkins CollaboraOffice <jenkinscollaboraoffice@gmail.com>
Tested-by: Jenkins
Reviewed-by: Tamás Zolnai <tamas.zolnai@collabora.com>
2020-06-25 21:35:50 +02:00
Tamás Zolnai
9ac9bc8be4 cypress: add initial code coverage support.
It's not optimal now, beacuse it changes source files.

Change-Id: I01cfb0be37eb0ef21dd1d606d9260b455a1cd585
Reviewed-on: https://gerrit.libreoffice.org/c/online/+/97036
Tested-by: Jenkins CollaboraOffice <jenkinscollaboraoffice@gmail.com>
Tested-by: Jenkins
Reviewed-by: Tamás Zolnai <tamas.zolnai@collabora.com>
2020-06-25 13:27:53 +02:00