2015-04-08 09:22:42 -05:00
|
|
|
#!/bin/bash
|
|
|
|
|
2015-06-08 08:58:34 -05:00
|
|
|
test $# -eq 2 || { echo "Usage: $0 <chroot template directory for system libs to create> <LO installation directory>"; exit 1; }
|
2015-04-08 09:22:42 -05:00
|
|
|
|
|
|
|
# No provision for spaces or other weird characters in pathnames. So sue me.
|
|
|
|
|
|
|
|
CHROOT=$1
|
|
|
|
INSTDIR=$2
|
|
|
|
|
2015-08-14 03:30:45 -05:00
|
|
|
test -d "$INSTDIR" || { echo "No such directory: $INSTDIR"; exit 1; }
|
|
|
|
|
2017-09-22 10:04:29 -05:00
|
|
|
mkdir -p $CHROOT || exit 1
|
2015-04-08 09:22:42 -05:00
|
|
|
|
2015-04-29 07:02:59 -05:00
|
|
|
CHROOT=`cd $CHROOT && /bin/pwd`
|
|
|
|
INSTDIR=`cd $INSTDIR && /bin/pwd`
|
|
|
|
|
|
|
|
cd / || exit 1
|
2015-04-08 09:22:42 -05:00
|
|
|
|
|
|
|
(
|
2016-01-14 11:14:35 -06:00
|
|
|
# Produce a list of file names, one per line, that will be copied
|
|
|
|
# into the template tree of system files for the chroot jails.
|
|
|
|
|
|
|
|
# First essential files and shared objects
|
wsd: support parallel systemplate setup
When tests are run in parallel, they will all
compete to update and set up the systemplate
directory, which has a handful of files that
need to be up-to-date. This is a source of errors.
Normally, these files are linked (hard- or soft-
link, whichever succeeds). With linking, we
only need to worry about the initial setup,
as the files will never be out-of-date from
then on. However, when linking fails, we need
to copy the files, and update them (by copying
over fresh versions of the files, if necessary)
every time a new kit is forked. Copying over
is tricky, as it's not atomic. To make it
atomic, we copy the files to the destination
directory under a temporary (random) name,
and then rename to the final name (which is
atomic, including replacing the target file,
if it exists).
No such race exists in production, where there
is (or should be) but one instance of loolwsd
(which does the initial setup) and forkit
(which updates systemplate before forking
new kit instances).
This is an issue with parallel tests only.
Change-Id: I6ba1514d00a84da7397d28efeb6378619711d52f
Reviewed-on: https://gerrit.libreoffice.org/c/online/+/97785
Tested-by: Jenkins
Tested-by: Jenkins CollaboraOffice <jenkinscollaboraoffice@gmail.com>
Reviewed-by: Ashod Nakashian <ashnakash@gmail.com>
2020-07-02 16:54:28 -05:00
|
|
|
find etc/ld.so.* \
|
2018-03-10 10:00:57 -06:00
|
|
|
lib/ld-* lib64/ld-* \
|
Fix domain name resolution from jail by adding required libs to systemplate
Originally all was good, then ad21717498cc214d5d5d5c0ff378238552137e2f
removed these files from jail, therefore name resolution broke, then
ac7d67f43e2b82dcc841ca2ecae91e668eb90613 added back libnss_* libs from one location, which
happened to work on developer's computer, then 4e5528e5f8b6abbd9ca715d5ac0ec410cbc783af,
b4f15c714ae5fb5208f4267a76effe8c4f54a4e7, and 9c0ab7215568b53ce4a26de00900f9c6fa36b508
added back etc/hosts etc/nsswitch.conf etc/resolv.conf that were also needed for proper
name resolution, finally this patch adds some more possible locations of Name Service
Switch (NSS) libs, therefore it makes sure that the required libnss_* shared libs
are found e.g. on Debian based systems. Also, testing with libreoffice/online docker image
has shown that libresolv.so.2 is also required in systemplate for name resolution, although
on some other systems it was not needed (preload?).
This fixes two bugs:
1) Linked images in documents were not displayed on certain systems
2) Insert image from WOPI host did not work on certain systems
Change-Id: I5d09a65341050f0f729bdbc46977c0f686ac023c
Reviewed-on: https://gerrit.libreoffice.org/64426
Reviewed-by: Andras Timar <andras.timar@collabora.com>
Tested-by: Andras Timar <andras.timar@collabora.com>
2018-12-02 07:30:28 -06:00
|
|
|
lib/libnss_* lib64/libnss_* lib/*/libnss_* \
|
|
|
|
lib/libresolv* lib64/libresolv* lib/*/libresolv* \
|
2016-01-14 11:14:35 -06:00
|
|
|
var/cache/fontconfig \
|
|
|
|
etc/fonts \
|
|
|
|
usr/lib/locale/en_US.utf8 \
|
|
|
|
usr/lib/locale/C.UTF-8 \
|
|
|
|
usr/lib/locale/locale_archive \
|
2018-11-22 07:23:57 -06:00
|
|
|
usr/lib/*/nss/*.so \
|
|
|
|
usr/lib/*/libsqlite* \
|
2016-01-14 11:14:35 -06:00
|
|
|
usr/share/zoneinfo/* \
|
|
|
|
usr/share/liblangtag \
|
2017-10-30 05:01:14 -05:00
|
|
|
usr/share/hyphen \
|
2018-06-15 08:04:25 -05:00
|
|
|
-type f 2>/dev/null
|
2016-01-14 11:14:35 -06:00
|
|
|
|
|
|
|
find etc/fonts \
|
|
|
|
lib/ld-* lib64/ld-* \
|
Fix domain name resolution from jail by adding required libs to systemplate
Originally all was good, then ad21717498cc214d5d5d5c0ff378238552137e2f
removed these files from jail, therefore name resolution broke, then
ac7d67f43e2b82dcc841ca2ecae91e668eb90613 added back libnss_* libs from one location, which
happened to work on developer's computer, then 4e5528e5f8b6abbd9ca715d5ac0ec410cbc783af,
b4f15c714ae5fb5208f4267a76effe8c4f54a4e7, and 9c0ab7215568b53ce4a26de00900f9c6fa36b508
added back etc/hosts etc/nsswitch.conf etc/resolv.conf that were also needed for proper
name resolution, finally this patch adds some more possible locations of Name Service
Switch (NSS) libs, therefore it makes sure that the required libnss_* shared libs
are found e.g. on Debian based systems. Also, testing with libreoffice/online docker image
has shown that libresolv.so.2 is also required in systemplate for name resolution, although
on some other systems it was not needed (preload?).
This fixes two bugs:
1) Linked images in documents were not displayed on certain systems
2) Insert image from WOPI host did not work on certain systems
Change-Id: I5d09a65341050f0f729bdbc46977c0f686ac023c
Reviewed-on: https://gerrit.libreoffice.org/64426
Reviewed-by: Andras Timar <andras.timar@collabora.com>
Tested-by: Andras Timar <andras.timar@collabora.com>
2018-12-02 07:30:28 -06:00
|
|
|
lib/libnss_* lib64/libnss_* lib/*/libnss_* \
|
|
|
|
lib/libresolv* lib64/libresolv* lib/*/libresolv* \
|
2018-11-22 07:23:57 -06:00
|
|
|
usr/lib/*/libsqlite* \
|
2018-06-15 08:04:25 -05:00
|
|
|
-type l 2>/dev/null
|
2016-01-14 11:14:35 -06:00
|
|
|
|
|
|
|
# Go through the LO shared objects and check what system libraries
|
|
|
|
# they link to.
|
2018-03-10 10:00:57 -06:00
|
|
|
find $INSTDIR -name 'xpdfimport' |
|
2016-01-14 11:14:35 -06:00
|
|
|
while read file; do
|
|
|
|
ldd $file 2>/dev/null
|
|
|
|
done |
|
|
|
|
grep -v dynamic | cut -d " " -f 3 | grep -E '^(/lib|/usr)' | sort -u | sed -e 's,^/,,'
|
|
|
|
|
2015-04-08 09:22:42 -05:00
|
|
|
) |
|
|
|
|
|
2015-04-29 07:02:59 -05:00
|
|
|
# Can't use -l because then symlinks won't be handled well enough.
|
|
|
|
# This will now copy the file a symlink points to, but whatever.
|
|
|
|
cpio -p -d -L $CHROOT
|
2015-04-27 07:49:25 -05:00
|
|
|
|
wsd: support parallel systemplate setup
When tests are run in parallel, they will all
compete to update and set up the systemplate
directory, which has a handful of files that
need to be up-to-date. This is a source of errors.
Normally, these files are linked (hard- or soft-
link, whichever succeeds). With linking, we
only need to worry about the initial setup,
as the files will never be out-of-date from
then on. However, when linking fails, we need
to copy the files, and update them (by copying
over fresh versions of the files, if necessary)
every time a new kit is forked. Copying over
is tricky, as it's not atomic. To make it
atomic, we copy the files to the destination
directory under a temporary (random) name,
and then rename to the final name (which is
atomic, including replacing the target file,
if it exists).
No such race exists in production, where there
is (or should be) but one instance of loolwsd
(which does the initial setup) and forkit
(which updates systemplate before forking
new kit instances).
This is an issue with parallel tests only.
Change-Id: I6ba1514d00a84da7397d28efeb6378619711d52f
Reviewed-on: https://gerrit.libreoffice.org/c/online/+/97785
Tested-by: Jenkins
Tested-by: Jenkins CollaboraOffice <jenkinscollaboraoffice@gmail.com>
Reviewed-by: Ashod Nakashian <ashnakash@gmail.com>
2020-07-02 16:54:28 -05:00
|
|
|
# Remove the dynamic files, which are linked by loolwsd.
|
|
|
|
rm -f $CHROOT/etc/{hosts,nsswitch.conf,resolv.conf,passwd,group,host.conf,timezone,localtime}
|
|
|
|
|
wsd: faster jail setup via bind-mount
loolmount now works and supports mounting and
unmounting, plus numerous improvements,
refactoring, logging, etc.. When enabled,
binding improves the jail setup time by anywhere
from 2x to orders of magnitude (in docker, f.e.).
A new config entry mount_jail_tree controls
whether mounting is used or the old method of
linking/copying of jail contents. It is set to
true by default and falls back to linking/copying.
A test mount is done when the setting is enabled,
and if mounting fails, it's disabled to avoid noise.
Temporarily disabled for unit-tests until we can
cleanup lingering mounts after Jenkins aborts our
build job. In a future patch we will have mount/jail
cleanup as part of make.
The network/system files in /etc that need frequent
refreshing are now updated in systemplate to make
their most recent version available in the jails.
These files can change during the course of loolwsd
lifetime, and are unlikely to be updated in
systemplate after installation at all. We link to
them in the systemplate/etc directory, and if that
fails, we copy them before forking each kit
instance to have the latest.
This reworks the approach used to bind-mount the
jails and the templates such that the total is
now down to only three mounts: systemplate, lo, tmp.
As now systemplate and lotemplate are shared, they
must be mounted as readonly, this means that user/
must now be moved into tmp/user/ which is writable.
The mount-points must be recursive, because we mount
lo/ within the mount-point of systemplate (which is
the root of the jail). But because we (re)bind
recursively, and because both systemplate and
lotemplate are mounted for each jails, we need to
make them unbindable, so they wouldn't multiply the
mount-points for each jails (an explosive growth!)
Contrarywise, we don't want the mount-points to
be shared, because we don't expect to add/remove
mounts after a jail is created.
The random temp directory is now created and set
correctly, plus many logging and other improvements.
Change-Id: Iae3fda5e876cf47d2cae6669a87b5b826a8748df
Reviewed-on: https://gerrit.libreoffice.org/c/online/+/92829
Tested-by: Jenkins
Tested-by: Jenkins CollaboraOffice <jenkinscollaboraoffice@gmail.com>
Reviewed-by: Ashod Nakashian <ashnakash@gmail.com>
2020-04-09 08:02:58 -05:00
|
|
|
mkdir -p $CHROOT/lo
|
|
|
|
mkdir -p $CHROOT/dev
|
|
|
|
mkdir -p $CHROOT/tmp/dev
|
2015-07-18 13:28:12 -05:00
|
|
|
|
2015-04-27 07:49:25 -05:00
|
|
|
# /usr/share/fonts needs to be taken care of separately because the
|
2016-04-15 08:23:44 -05:00
|
|
|
# directory time stamps must be preserved for fontconfig to trust
|
2015-04-27 07:49:25 -05:00
|
|
|
# its cache.
|
|
|
|
|
2015-04-29 07:02:59 -05:00
|
|
|
cd $CHROOT || exit 1
|
|
|
|
|
2016-02-29 06:25:12 -06:00
|
|
|
mkdir -p usr/share || exit 1
|
2016-06-29 09:01:26 -05:00
|
|
|
cp -r -p -L /usr/share/fonts usr/share
|
2016-02-29 06:25:12 -06:00
|
|
|
|
|
|
|
if [ -h usr/share/fonts/ghostscript ]; then
|
|
|
|
mkdir usr/share/ghostscript || exit 1
|
2016-06-29 09:01:26 -05:00
|
|
|
cp -r -p -L /usr/share/ghostscript/fonts usr/share/ghostscript
|
2015-04-29 07:02:59 -05:00
|
|
|
fi
|
2016-04-15 08:23:44 -05:00
|
|
|
|
2018-03-10 07:50:43 -06:00
|
|
|
# Remove obsolete & unused bitmap fonts
|
|
|
|
find usr/share -name '*.pcf' | xargs rm -f
|
|
|
|
find usr/share -name '*.pcf.gz' | xargs rm -f
|
|
|
|
|
2016-04-15 08:34:23 -05:00
|
|
|
# Debugging only hackery to avoid confusion.
|
2016-04-15 08:23:44 -05:00
|
|
|
if test "z$ENABLE_DEBUG" != "z" -a "z$HOME" != "z"; then
|
|
|
|
echo "Copying development users's fonts into systemplate"
|
|
|
|
mkdir -p $CHROOT/$HOME
|
2016-04-15 09:19:04 -05:00
|
|
|
test -d $HOME/.fonts && cp -r -p -L $HOME/.fonts $CHROOT/$HOME
|
2016-04-15 08:23:44 -05:00
|
|
|
fi
|
2016-04-15 09:19:04 -05:00
|
|
|
|
|
|
|
exit 0
|