Without this, the conversion test in UnitHTTP
fails because the root of the temp directory
hasn't yet been created (mkdir actually fails,
since the root is missing).
This failure occures when UnitHTTP is executed
before other tests.
Change-Id: I9b5d9a48c8ab9735978ecfa2d29944e037cf794d
Signed-off-by: Ashod Nakashian <ashod.nakashian@collabora.co.uk>
In some cases we cannot do a fast bind-mount of the files we want
in our jail since we don't have cap_sys_admin for loolmount inside
eg. docker.
Thus we need to fallback to hard-linking, however various security
systems namespace parts of our tree, such that link() fails with
EXDEV even across the (apparently) same file-system.
As such we need to assemble a copy of what we want to hard-link
close to our jails. However, this needs to be owned by root / the
system to avoid having writable files shared between jails. Hence
we need cap_chown in addition to cap_fowner, to get ownership right
and then hard-link.
Change-Id: Iba0ef46ddbc1c03f3dc7177bc1ec1755624135db
Signed-off-by: Michael Meeks <michael.meeks@collabora.com>
Most C and Posix API clobber errno. By failing to save
it immediately after invoking an API we risk simply
reporting the result of an arbitrary subsequent API call.
This adds LOG_SYS_ERRNO to take errno explicitly.
This is necessary because sometimes logging is not done
immediately after calling the function for which we
want to report errno. Similarly, log macros that log
errno need to save errno before calling any functions.
This is necessary as the argements might contain calls
that clobber errno.
This also converts some LOG_SYS entries to LOG_ERR
because there can be no relevant errno in that context
(f.e. in a catch clause).
A couple of LOG_ macros have been folded into others,
reducing redundancy.
Finally, both of these log macros append errno to the
log message, so there is little point in ending the
messages with a period.
Change-Id: Iecc656f67115fec78b65cad4e7c17a17623ecf43
Signed-off-by: Ashod Nakashian <ashod.nakashian@collabora.co.uk>
These helpers are for testing only. They should
ideally be moved to the test helpers, but because
of dependency on the FileDeleter in FileUtil they
remain in FileUtil.
Change-Id: I93c7e08823edec8f6a53419f0a6596f3255f23f9
Signed-off-by: Ashod Nakashian <ashod.nakashian@collabora.co.uk>
We should have no symlinks in the jail whatsoever, except
those we create to files known to exist in the jail.
Unfortunately, some systems have some of the /etc files
as symlinks. When we create hard-links to these files,
they can't be accessed from the jail, since the path
they point to isn't replicated in systemplate and jails.
First change here is to always link to the source file or,
when copying, to copy the source rather than a symlink.
Next, to detect modifications, we compare not just the
size and timestamp, but also the contents. This way we
can be certain that any modification will be detected.
Finally, when we copy at least one file in the
systemplate/etc directory, we flag it by creating the
'copied' file. This way we have a reliable indicator
and don't need to second guess if the files are
hard-linked or copied.
We also avoid some noisy errors when we fail to update
systemplate when it's read-only by first checking
if systemplate is writable or not and insue a friendly
log instead.
Change-Id: Ie8c3e70ea4ec19ee098309f8666c00639fa7319b
For various reasons, systemplate may be read-only
or under a different owner and therefore impossible
to update the dynamic files in it.
To support such a scenario, we first link the
eight dynamic files in /etc when creating systemplate.
If this fails, we copy the files.
When creating jails, we always check that all the
dynamic files are up-to-date. If they are, nothing
further is necessary and we bind-mount, if enabled
and possible.
However, if the dynamic files are not up-to-date,
we disable bind-mounting and force linking
the files in the jails. Failing that, we copy them,
which is not ideal, but allows us to ensure the
dynamic files are up-to-date as we copy them too.
Ideally, the dynamic files in question would be
hard-link (or at least soft-linked) in systemplate
at creation. From then on we would bind-mount
the jails and everything would work perfectly and
no files would need updating. This patch is fallback
for when this scheme fails, which should be exceedingly
rare anyway, but which still ensures correct operation.
Change-Id: I09c6f057c49396579aaddb1b8bf4af0930dd4247
Reviewed-on: https://gerrit.libreoffice.org/c/online/+/100834
Tested-by: Jenkins CollaboraOffice <jenkinscollaboraoffice@gmail.com>
Tested-by: Jenkins
Reviewed-by: Andras Timar <andras.timar@collabora.com>
When tests are run in parallel, they will all
compete to update and set up the systemplate
directory, which has a handful of files that
need to be up-to-date. This is a source of errors.
Normally, these files are linked (hard- or soft-
link, whichever succeeds). With linking, we
only need to worry about the initial setup,
as the files will never be out-of-date from
then on. However, when linking fails, we need
to copy the files, and update them (by copying
over fresh versions of the files, if necessary)
every time a new kit is forked. Copying over
is tricky, as it's not atomic. To make it
atomic, we copy the files to the destination
directory under a temporary (random) name,
and then rename to the final name (which is
atomic, including replacing the target file,
if it exists).
No such race exists in production, where there
is (or should be) but one instance of loolwsd
(which does the initial setup) and forkit
(which updates systemplate before forking
new kit instances).
This is an issue with parallel tests only.
Change-Id: I6ba1514d00a84da7397d28efeb6378619711d52f
Reviewed-on: https://gerrit.libreoffice.org/c/online/+/97785
Tested-by: Jenkins
Tested-by: Jenkins CollaboraOffice <jenkinscollaboraoffice@gmail.com>
Reviewed-by: Ashod Nakashian <ashnakash@gmail.com>
The new utility is safer and more readable.
Change-Id: I3a86675378d458cb004e5534dbf2b401936d0e57
Reviewed-on: https://gerrit.libreoffice.org/c/online/+/98183
Tested-by: Jenkins
Tested-by: Jenkins CollaboraOffice <jenkinscollaboraoffice@gmail.com>
Reviewed-by: Ashod Nakashian <ashnakash@gmail.com>
loolmount now works and supports mounting and
unmounting, plus numerous improvements,
refactoring, logging, etc.. When enabled,
binding improves the jail setup time by anywhere
from 2x to orders of magnitude (in docker, f.e.).
A new config entry mount_jail_tree controls
whether mounting is used or the old method of
linking/copying of jail contents. It is set to
true by default and falls back to linking/copying.
A test mount is done when the setting is enabled,
and if mounting fails, it's disabled to avoid noise.
Temporarily disabled for unit-tests until we can
cleanup lingering mounts after Jenkins aborts our
build job. In a future patch we will have mount/jail
cleanup as part of make.
The network/system files in /etc that need frequent
refreshing are now updated in systemplate to make
their most recent version available in the jails.
These files can change during the course of loolwsd
lifetime, and are unlikely to be updated in
systemplate after installation at all. We link to
them in the systemplate/etc directory, and if that
fails, we copy them before forking each kit
instance to have the latest.
This reworks the approach used to bind-mount the
jails and the templates such that the total is
now down to only three mounts: systemplate, lo, tmp.
As now systemplate and lotemplate are shared, they
must be mounted as readonly, this means that user/
must now be moved into tmp/user/ which is writable.
The mount-points must be recursive, because we mount
lo/ within the mount-point of systemplate (which is
the root of the jail). But because we (re)bind
recursively, and because both systemplate and
lotemplate are mounted for each jails, we need to
make them unbindable, so they wouldn't multiply the
mount-points for each jails (an explosive growth!)
Contrarywise, we don't want the mount-points to
be shared, because we don't expect to add/remove
mounts after a jail is created.
The random temp directory is now created and set
correctly, plus many logging and other improvements.
Change-Id: Iae3fda5e876cf47d2cae6669a87b5b826a8748df
Reviewed-on: https://gerrit.libreoffice.org/c/online/+/92829
Tested-by: Jenkins
Tested-by: Jenkins CollaboraOffice <jenkinscollaboraoffice@gmail.com>
Reviewed-by: Ashod Nakashian <ashnakash@gmail.com>
More readable and typically more efficient.
Change-Id: I9bd5bfc91f4ac255bb8ae0987708fb8b56b398f8
Reviewed-on: https://gerrit.libreoffice.org/c/online/+/95285
Reviewed-by: Michael Meeks <michael.meeks@collabora.com>
Tested-by: Jenkins
Tested-by: Jenkins CollaboraOffice <jenkinscollaboraoffice@gmail.com>
Getting rid of Util::alertAllUsers() calls in arbitrary places is good
because the semi-obscure way in which it works in the multi-process
web-based Online is potentially hard to fit together with the desire
to make the single-process mobile app able to handle several open
documents.
Change-Id: I6055a993ee31941e3592508aac5f0edf6497a836
Reviewed-on: https://gerrit.libreoffice.org/c/online/+/92571
Tested-by: Jenkins CollaboraOffice <jenkinscollaboraoffice@gmail.com>
Reviewed-by: Tor Lillqvist <tml@collabora.com>
This introduces basic C++17 support, because the functionality needed
here is easy to implement using std::filesystem.
Adds also the necessary checks to ./configure. The code still uses POCO
when C++17 is not available in the compiler.
Change-Id: I03353834d10201bf0a13ea72715560b9b9b16265
Reviewed-on: https://gerrit.libreoffice.org/82294
Reviewed-by: Jan Holesovsky <kendy@collabora.com>
Tested-by: Jan Holesovsky <kendy@collabora.com>
Re-think the plumbing between the different parts of the C++ Online
code. Do try to have it work more like in real Online on all but the
lowest socket level. Except that we don't have multiple processes, but
threads inside the same process. And instead of using actual system
sockets for WebSocket traffic between the threads, we use our own
FakeSocket things, with no WebSocket framing of messages.
Reduce the amount of #ifdef MOBILEAPP a bit also by compiling in the
UnitFoo things. Hardcode that so that no unit testing is ever
attempted, though. We don't try to dlopen any library.
Corresponding changes in the app Objective-C code. Plus fixes and
functionality improvements.
Now it gets so far that the JavaScript code thinks it has the document
tiles presented, and doesn't crash. But it hangs occasionally. And all
tiles show up blank.
Anyway, progress.
Change-Id: I769497c9a46ddb74984bc7af36d132b7b43895d4
Re-think Linux vs mobile ifdefs a bit. Use #ifdef __linux only to
surround code that actually is Linux-specific. Use #ifdef MOBILEAPP
for code that is for a mobile version (with no separste wsd, forkit,
and kit processes, and with no WebSocket protocol used).
Bypass UnitFoo for mobile. Possibly we do want the UnitFoo stuff after
all on mobile, to run in some special testing mode? Hard to say, let's
skipt it for now.
Around 1.5x faster than Poco,
which first enumerates files into
a container, then iterates over
them and stats before unlinking.
Here we enumerate and unlink in
a single pass.
Change-Id: I79d1c0f1b5ec557ccc4f0e2ec7a0609051d8d212
Reviewed-on: https://gerrit.libreoffice.org/33680
Reviewed-by: Ashod Nakashian <ashnakash@gmail.com>
Tested-by: Ashod Nakashian <ashnakash@gmail.com>
This makes debugging much easier as one can
readily match WSD logs with a given test.
Change-Id: I8f2c83d67189038699af3f24dee205bc7efb5c28
Reviewed-on: https://gerrit.libreoffice.org/32860
Reviewed-by: Ashod Nakashian <ashnakash@gmail.com>
Tested-by: Ashod Nakashian <ashnakash@gmail.com>
There is no way to let the user of document currently being
opened, in case of failure, know that disk is low on space.
We check the disk space when forking children after which we try
to alert all users but this would end up doing nothing for
current document because document broker is not registered at
this time (we iterate through doc brokers when alerting). Another
conditional disk check is performed just before opening the
document but this is performed only if last disk check was
performed greater than 60 seconds which would never be the case
because document open is always preceded by a child fork (when
rebalancing children).
Lets not cache the disk check when forking the children to
prevent above mentioned situation while still minimizing the
number of disk checks performed.
Change-Id: Id3add998f94e23f9f8c144f09e5efe9f0b63821c
