2015-04-08 09:22:42 -05:00
|
|
|
#!/bin/bash
|
mount: handle symlinks and improve updating systemplate
We should have no symlinks in the jail whatsoever, except
those we create to files known to exist in the jail.
Unfortunately, some systems have some of the /etc files
as symlinks. When we create hard-links to these files,
they can't be accessed from the jail, since the path
they point to isn't replicated in systemplate and jails.
First change here is to always link to the source file or,
when copying, to copy the source rather than a symlink.
Next, to detect modifications, we compare not just the
size and timestamp, but also the contents. This way we
can be certain that any modification will be detected.
Finally, when we copy at least one file in the
systemplate/etc directory, we flag it by creating the
'copied' file. This way we have a reliable indicator
and don't need to second guess if the files are
hard-linked or copied.
We also avoid some noisy errors when we fail to update
systemplate when it's read-only by first checking
if systemplate is writable or not and insue a friendly
log instead.
Change-Id: Ie8c3e70ea4ec19ee098309f8666c00639fa7319b
2020-10-18 20:56:40 -05:00
|
|
|
# set -x
|
2015-04-08 09:22:42 -05:00
|
|
|
|
2015-06-08 08:58:34 -05:00
|
|
|
test $# -eq 2 || { echo "Usage: $0 <chroot template directory for system libs to create> <LO installation directory>"; exit 1; }
|
2015-04-08 09:22:42 -05:00
|
|
|
|
|
|
|
# No provision for spaces or other weird characters in pathnames. So sue me.
|
|
|
|
|
|
|
|
CHROOT=$1
|
|
|
|
INSTDIR=$2
|
|
|
|
|
2020-12-19 03:47:49 -06:00
|
|
|
test -d "$INSTDIR" || { echo "$0: No such directory: $INSTDIR"; exit 1; }
|
2015-08-14 03:30:45 -05:00
|
|
|
|
2017-09-22 10:04:29 -05:00
|
|
|
mkdir -p $CHROOT || exit 1
|
2015-04-08 09:22:42 -05:00
|
|
|
|
2020-08-23 11:11:23 -05:00
|
|
|
# Resolve the real paths, in case they are relative and/or symlinked.
|
|
|
|
# INSTDIR_LOGICAL will contain the logical path, if there are symlinks,
|
|
|
|
# while INSTDIR is the physical one. Both will most likely be the same,
|
|
|
|
# except on systems that have symlinks in the path. We must create
|
|
|
|
# both paths (if they are different) inside the jail, hence we need both.
|
2015-04-29 07:02:59 -05:00
|
|
|
CHROOT=`cd $CHROOT && /bin/pwd`
|
2020-08-23 11:11:23 -05:00
|
|
|
INSTDIR_LOGICAL=`cd $INSTDIR && /bin/pwd -L`
|
|
|
|
INSTDIR=`cd $INSTDIR && /bin/pwd -P`
|
2015-04-29 07:02:59 -05:00
|
|
|
|
|
|
|
cd / || exit 1
|
2015-04-08 09:22:42 -05:00
|
|
|
|
|
|
|
(
|
2016-01-14 11:14:35 -06:00
|
|
|
# Produce a list of file names, one per line, that will be copied
|
|
|
|
# into the template tree of system files for the chroot jails.
|
|
|
|
|
|
|
|
# First essential files and shared objects
|
wsd: support parallel systemplate setup
When tests are run in parallel, they will all
compete to update and set up the systemplate
directory, which has a handful of files that
need to be up-to-date. This is a source of errors.
Normally, these files are linked (hard- or soft-
link, whichever succeeds). With linking, we
only need to worry about the initial setup,
as the files will never be out-of-date from
then on. However, when linking fails, we need
to copy the files, and update them (by copying
over fresh versions of the files, if necessary)
every time a new kit is forked. Copying over
is tricky, as it's not atomic. To make it
atomic, we copy the files to the destination
directory under a temporary (random) name,
and then rename to the final name (which is
atomic, including replacing the target file,
if it exists).
No such race exists in production, where there
is (or should be) but one instance of loolwsd
(which does the initial setup) and forkit
(which updates systemplate before forking
new kit instances).
This is an issue with parallel tests only.
Change-Id: I6ba1514d00a84da7397d28efeb6378619711d52f
Reviewed-on: https://gerrit.libreoffice.org/c/online/+/97785
Tested-by: Jenkins
Tested-by: Jenkins CollaboraOffice <jenkinscollaboraoffice@gmail.com>
Reviewed-by: Ashod Nakashian <ashnakash@gmail.com>
2020-07-02 16:54:28 -05:00
|
|
|
find etc/ld.so.* \
|
2018-03-10 10:00:57 -06:00
|
|
|
lib/ld-* lib64/ld-* \
|
Fix domain name resolution from jail by adding required libs to systemplate
Originally all was good, then ad21717498cc214d5d5d5c0ff378238552137e2f
removed these files from jail, therefore name resolution broke, then
ac7d67f43e2b82dcc841ca2ecae91e668eb90613 added back libnss_* libs from one location, which
happened to work on developer's computer, then 4e5528e5f8b6abbd9ca715d5ac0ec410cbc783af,
b4f15c714ae5fb5208f4267a76effe8c4f54a4e7, and 9c0ab7215568b53ce4a26de00900f9c6fa36b508
added back etc/hosts etc/nsswitch.conf etc/resolv.conf that were also needed for proper
name resolution, finally this patch adds some more possible locations of Name Service
Switch (NSS) libs, therefore it makes sure that the required libnss_* shared libs
are found e.g. on Debian based systems. Also, testing with libreoffice/online docker image
has shown that libresolv.so.2 is also required in systemplate for name resolution, although
on some other systems it was not needed (preload?).
This fixes two bugs:
1) Linked images in documents were not displayed on certain systems
2) Insert image from WOPI host did not work on certain systems
Change-Id: I5d09a65341050f0f729bdbc46977c0f686ac023c
Reviewed-on: https://gerrit.libreoffice.org/64426
Reviewed-by: Andras Timar <andras.timar@collabora.com>
Tested-by: Andras Timar <andras.timar@collabora.com>
2018-12-02 07:30:28 -06:00
|
|
|
lib/libnss_* lib64/libnss_* lib/*/libnss_* \
|
|
|
|
lib/libresolv* lib64/libresolv* lib/*/libresolv* \
|
2016-01-14 11:14:35 -06:00
|
|
|
var/cache/fontconfig \
|
|
|
|
etc/fonts \
|
|
|
|
usr/lib/locale/en_US.utf8 \
|
|
|
|
usr/lib/locale/C.UTF-8 \
|
|
|
|
usr/lib/locale/locale_archive \
|
2018-11-22 07:23:57 -06:00
|
|
|
usr/lib/*/nss/*.so \
|
|
|
|
usr/lib/*/libsqlite* \
|
2016-01-14 11:14:35 -06:00
|
|
|
usr/share/zoneinfo/* \
|
|
|
|
usr/share/liblangtag \
|
2017-10-30 05:01:14 -05:00
|
|
|
usr/share/hyphen \
|
2018-06-15 08:04:25 -05:00
|
|
|
-type f 2>/dev/null
|
2016-01-14 11:14:35 -06:00
|
|
|
|
|
|
|
find etc/fonts \
|
|
|
|
lib/ld-* lib64/ld-* \
|
Fix domain name resolution from jail by adding required libs to systemplate
Originally all was good, then ad21717498cc214d5d5d5c0ff378238552137e2f
removed these files from jail, therefore name resolution broke, then
ac7d67f43e2b82dcc841ca2ecae91e668eb90613 added back libnss_* libs from one location, which
happened to work on developer's computer, then 4e5528e5f8b6abbd9ca715d5ac0ec410cbc783af,
b4f15c714ae5fb5208f4267a76effe8c4f54a4e7, and 9c0ab7215568b53ce4a26de00900f9c6fa36b508
added back etc/hosts etc/nsswitch.conf etc/resolv.conf that were also needed for proper
name resolution, finally this patch adds some more possible locations of Name Service
Switch (NSS) libs, therefore it makes sure that the required libnss_* shared libs
are found e.g. on Debian based systems. Also, testing with libreoffice/online docker image
has shown that libresolv.so.2 is also required in systemplate for name resolution, although
on some other systems it was not needed (preload?).
This fixes two bugs:
1) Linked images in documents were not displayed on certain systems
2) Insert image from WOPI host did not work on certain systems
Change-Id: I5d09a65341050f0f729bdbc46977c0f686ac023c
Reviewed-on: https://gerrit.libreoffice.org/64426
Reviewed-by: Andras Timar <andras.timar@collabora.com>
Tested-by: Andras Timar <andras.timar@collabora.com>
2018-12-02 07:30:28 -06:00
|
|
|
lib/libnss_* lib64/libnss_* lib/*/libnss_* \
|
|
|
|
lib/libresolv* lib64/libresolv* lib/*/libresolv* \
|
2018-11-22 07:23:57 -06:00
|
|
|
usr/lib/*/libsqlite* \
|
2018-06-15 08:04:25 -05:00
|
|
|
-type l 2>/dev/null
|
2016-01-14 11:14:35 -06:00
|
|
|
|
|
|
|
# Go through the LO shared objects and check what system libraries
|
|
|
|
# they link to.
|
2018-03-10 10:00:57 -06:00
|
|
|
find $INSTDIR -name 'xpdfimport' |
|
2016-01-14 11:14:35 -06:00
|
|
|
while read file; do
|
|
|
|
ldd $file 2>/dev/null
|
|
|
|
done |
|
|
|
|
grep -v dynamic | cut -d " " -f 3 | grep -E '^(/lib|/usr)' | sort -u | sed -e 's,^/,,'
|
|
|
|
|
2015-04-08 09:22:42 -05:00
|
|
|
) |
|
|
|
|
|
2015-04-29 07:02:59 -05:00
|
|
|
# Can't use -l because then symlinks won't be handled well enough.
|
|
|
|
# This will now copy the file a symlink points to, but whatever.
|
|
|
|
cpio -p -d -L $CHROOT
|
2015-04-27 07:49:25 -05:00
|
|
|
|
wsd: support read-only systemplate
For various reasons, systemplate may be read-only
or under a different owner and therefore impossible
to update the dynamic files in it.
To support such a scenario, we first link the
eight dynamic files in /etc when creating systemplate.
If this fails, we copy the files.
When creating jails, we always check that all the
dynamic files are up-to-date. If they are, nothing
further is necessary and we bind-mount, if enabled
and possible.
However, if the dynamic files are not up-to-date,
we disable bind-mounting and force linking
the files in the jails. Failing that, we copy them,
which is not ideal, but allows us to ensure the
dynamic files are up-to-date as we copy them too.
Ideally, the dynamic files in question would be
hard-link (or at least soft-linked) in systemplate
at creation. From then on we would bind-mount
the jails and everything would work perfectly and
no files would need updating. This patch is fallback
for when this scheme fails, which should be exceedingly
rare anyway, but which still ensures correct operation.
Change-Id: I09c6f057c49396579aaddb1b8bf4af0930dd4247
Reviewed-on: https://gerrit.libreoffice.org/c/online/+/100834
Tested-by: Jenkins CollaboraOffice <jenkinscollaboraoffice@gmail.com>
Tested-by: Jenkins
Reviewed-by: Andras Timar <andras.timar@collabora.com>
2020-08-16 21:46:33 -05:00
|
|
|
# Link the dynamic files, replacing any existing.
|
mount: handle symlinks and improve updating systemplate
We should have no symlinks in the jail whatsoever, except
those we create to files known to exist in the jail.
Unfortunately, some systems have some of the /etc files
as symlinks. When we create hard-links to these files,
they can't be accessed from the jail, since the path
they point to isn't replicated in systemplate and jails.
First change here is to always link to the source file or,
when copying, to copy the source rather than a symlink.
Next, to detect modifications, we compare not just the
size and timestamp, but also the contents. This way we
can be certain that any modification will be detected.
Finally, when we copy at least one file in the
systemplate/etc directory, we flag it by creating the
'copied' file. This way we have a reliable indicator
and don't need to second guess if the files are
hard-linked or copied.
We also avoid some noisy errors when we fail to update
systemplate when it's read-only by first checking
if systemplate is writable or not and insue a friendly
log instead.
Change-Id: Ie8c3e70ea4ec19ee098309f8666c00639fa7319b
2020-10-18 20:56:40 -05:00
|
|
|
rm -f $CHROOT/etc/copied
|
wsd: support read-only systemplate
For various reasons, systemplate may be read-only
or under a different owner and therefore impossible
to update the dynamic files in it.
To support such a scenario, we first link the
eight dynamic files in /etc when creating systemplate.
If this fails, we copy the files.
When creating jails, we always check that all the
dynamic files are up-to-date. If they are, nothing
further is necessary and we bind-mount, if enabled
and possible.
However, if the dynamic files are not up-to-date,
we disable bind-mounting and force linking
the files in the jails. Failing that, we copy them,
which is not ideal, but allows us to ensure the
dynamic files are up-to-date as we copy them too.
Ideally, the dynamic files in question would be
hard-link (or at least soft-linked) in systemplate
at creation. From then on we would bind-mount
the jails and everything would work perfectly and
no files would need updating. This patch is fallback
for when this scheme fails, which should be exceedingly
rare anyway, but which still ensures correct operation.
Change-Id: I09c6f057c49396579aaddb1b8bf4af0930dd4247
Reviewed-on: https://gerrit.libreoffice.org/c/online/+/100834
Tested-by: Jenkins CollaboraOffice <jenkinscollaboraoffice@gmail.com>
Tested-by: Jenkins
Reviewed-by: Andras Timar <andras.timar@collabora.com>
2020-08-16 21:46:33 -05:00
|
|
|
for file in hosts nsswitch.conf resolv.conf passwd group host.conf timezone localtime
|
|
|
|
do
|
mount: handle symlinks and improve updating systemplate
We should have no symlinks in the jail whatsoever, except
those we create to files known to exist in the jail.
Unfortunately, some systems have some of the /etc files
as symlinks. When we create hard-links to these files,
they can't be accessed from the jail, since the path
they point to isn't replicated in systemplate and jails.
First change here is to always link to the source file or,
when copying, to copy the source rather than a symlink.
Next, to detect modifications, we compare not just the
size and timestamp, but also the contents. This way we
can be certain that any modification will be detected.
Finally, when we copy at least one file in the
systemplate/etc directory, we flag it by creating the
'copied' file. This way we have a reliable indicator
and don't need to second guess if the files are
hard-linked or copied.
We also avoid some noisy errors when we fail to update
systemplate when it's read-only by first checking
if systemplate is writable or not and insue a friendly
log instead.
Change-Id: Ie8c3e70ea4ec19ee098309f8666c00639fa7319b
2020-10-18 20:56:40 -05:00
|
|
|
# echo "Linking/Copying /etc/$file"
|
2020-08-26 10:52:39 -05:00
|
|
|
# Prefer hard-linking, fallback to just copying (do *not* use soft-linking because that would be relative to the jail).
|
mount: handle symlinks and improve updating systemplate
We should have no symlinks in the jail whatsoever, except
those we create to files known to exist in the jail.
Unfortunately, some systems have some of the /etc files
as symlinks. When we create hard-links to these files,
they can't be accessed from the jail, since the path
they point to isn't replicated in systemplate and jails.
First change here is to always link to the source file or,
when copying, to copy the source rather than a symlink.
Next, to detect modifications, we compare not just the
size and timestamp, but also the contents. This way we
can be certain that any modification will be detected.
Finally, when we copy at least one file in the
systemplate/etc directory, we flag it by creating the
'copied' file. This way we have a reliable indicator
and don't need to second guess if the files are
hard-linked or copied.
We also avoid some noisy errors when we fail to update
systemplate when it's read-only by first checking
if systemplate is writable or not and insue a friendly
log instead.
Change-Id: Ie8c3e70ea4ec19ee098309f8666c00639fa7319b
2020-10-18 20:56:40 -05:00
|
|
|
# When copying, we must make sure that we copy the source and not a symlink. Otherwise, the source won't be accessible from the jail.
|
|
|
|
# In addition, we flag that at least one file is copied by creating the 'copied' file, so that we do check for updates.
|
2020-12-19 03:47:49 -06:00
|
|
|
ln -f `realpath /etc/$file` $CHROOT/etc/$file 2> /dev/null || (cp --dereference --preserve=all /etc/$file $CHROOT/etc/$file && touch $CHROOT/etc/copied) || echo "$0: Failed to link or copy /etc/$file"
|
wsd: support read-only systemplate
For various reasons, systemplate may be read-only
or under a different owner and therefore impossible
to update the dynamic files in it.
To support such a scenario, we first link the
eight dynamic files in /etc when creating systemplate.
If this fails, we copy the files.
When creating jails, we always check that all the
dynamic files are up-to-date. If they are, nothing
further is necessary and we bind-mount, if enabled
and possible.
However, if the dynamic files are not up-to-date,
we disable bind-mounting and force linking
the files in the jails. Failing that, we copy them,
which is not ideal, but allows us to ensure the
dynamic files are up-to-date as we copy them too.
Ideally, the dynamic files in question would be
hard-link (or at least soft-linked) in systemplate
at creation. From then on we would bind-mount
the jails and everything would work perfectly and
no files would need updating. This patch is fallback
for when this scheme fails, which should be exceedingly
rare anyway, but which still ensures correct operation.
Change-Id: I09c6f057c49396579aaddb1b8bf4af0930dd4247
Reviewed-on: https://gerrit.libreoffice.org/c/online/+/100834
Tested-by: Jenkins CollaboraOffice <jenkinscollaboraoffice@gmail.com>
Tested-by: Jenkins
Reviewed-by: Andras Timar <andras.timar@collabora.com>
2020-08-16 21:46:33 -05:00
|
|
|
done
|
|
|
|
|
2020-08-23 11:11:23 -05:00
|
|
|
# Link dev/random and dev/urandom to ../tmp/dev/.
|
|
|
|
# The jail then creates the random device nodes in its /tmp/dev/.
|
|
|
|
mkdir -p $CHROOT/dev
|
|
|
|
mkdir -p $CHROOT/tmp/dev
|
|
|
|
for file in random urandom
|
|
|
|
do
|
2020-08-26 10:52:39 -05:00
|
|
|
# This link is relative anyway, so can be soft.
|
2020-12-19 03:47:49 -06:00
|
|
|
ln -f ../tmp/dev/$file $CHROOT/dev/ 2> /dev/null || ln -f -s ../tmp/dev/$file $CHROOT/dev/ || echo "$0: Failed to link dev/$file"
|
2020-08-23 11:11:23 -05:00
|
|
|
done
|
|
|
|
|
|
|
|
# Create a relative symbolic link within systemplate that points from
|
|
|
|
# the path of $INSTDIR (as seen from the jail as an absolute path)
|
|
|
|
# to the /lo path, where the instdir of LO will really reside.
|
|
|
|
mkdir -p $CHROOT/lo
|
|
|
|
# In case the original path is different from
|
|
|
|
for path in $INSTDIR $INSTDIR_LOGICAL
|
|
|
|
do
|
2020-08-26 10:52:39 -05:00
|
|
|
# Create a soft-link, as it's a relative directory path (can't be a hard-link).
|
2020-08-23 11:11:23 -05:00
|
|
|
INSTDIR_PARENT="$(dirname "$CHROOT/$path")"
|
|
|
|
mkdir -p $INSTDIR_PARENT
|
|
|
|
ln -f -s `realpath --relative-to=$INSTDIR_PARENT $CHROOT/lo` $CHROOT/$path
|
|
|
|
done
|
|
|
|
|
2015-04-27 07:49:25 -05:00
|
|
|
# /usr/share/fonts needs to be taken care of separately because the
|
2016-04-15 08:23:44 -05:00
|
|
|
# directory time stamps must be preserved for fontconfig to trust
|
2015-04-27 07:49:25 -05:00
|
|
|
# its cache.
|
|
|
|
|
2015-04-29 07:02:59 -05:00
|
|
|
cd $CHROOT || exit 1
|
|
|
|
|
2016-02-29 06:25:12 -06:00
|
|
|
mkdir -p usr/share || exit 1
|
2016-06-29 09:01:26 -05:00
|
|
|
cp -r -p -L /usr/share/fonts usr/share
|
2016-02-29 06:25:12 -06:00
|
|
|
|
|
|
|
if [ -h usr/share/fonts/ghostscript ]; then
|
|
|
|
mkdir usr/share/ghostscript || exit 1
|
2016-06-29 09:01:26 -05:00
|
|
|
cp -r -p -L /usr/share/ghostscript/fonts usr/share/ghostscript
|
2015-04-29 07:02:59 -05:00
|
|
|
fi
|
2016-04-15 08:23:44 -05:00
|
|
|
|
2018-03-10 07:50:43 -06:00
|
|
|
# Remove obsolete & unused bitmap fonts
|
|
|
|
find usr/share -name '*.pcf' | xargs rm -f
|
|
|
|
find usr/share -name '*.pcf.gz' | xargs rm -f
|
|
|
|
|
2016-04-15 08:34:23 -05:00
|
|
|
# Debugging only hackery to avoid confusion.
|
2016-04-15 08:23:44 -05:00
|
|
|
if test "z$ENABLE_DEBUG" != "z" -a "z$HOME" != "z"; then
|
2020-12-19 03:47:49 -06:00
|
|
|
echo "$0: Copying development users's fonts into systemplate"
|
2016-04-15 08:23:44 -05:00
|
|
|
mkdir -p $CHROOT/$HOME
|
2016-04-15 09:19:04 -05:00
|
|
|
test -d $HOME/.fonts && cp -r -p -L $HOME/.fonts $CHROOT/$HOME
|
2016-04-15 08:23:44 -05:00
|
|
|
fi
|
2016-04-15 09:19:04 -05:00
|
|
|
|
|
|
|
exit 0
|