libreoffice-online/common/FileUtil.hpp

270 lines
12 KiB
C++
Raw Normal View History

/* -*- Mode: C++; tab-width: 4; indent-tabs-mode: nil; c-basic-offset: 4; fill-column: 100 -*- */
/*
* This Source Code Form is subject to the terms of the Mozilla Public
* License, v. 2.0. If a copy of the MPL was not distributed with this
* file, You can obtain one at http://mozilla.org/MPL/2.0/.
*/
#pragma once
wsd: faster jail setup via bind-mount loolmount now works and supports mounting and unmounting, plus numerous improvements, refactoring, logging, etc.. When enabled, binding improves the jail setup time by anywhere from 2x to orders of magnitude (in docker, f.e.). A new config entry mount_jail_tree controls whether mounting is used or the old method of linking/copying of jail contents. It is set to true by default and falls back to linking/copying. A test mount is done when the setting is enabled, and if mounting fails, it's disabled to avoid noise. Temporarily disabled for unit-tests until we can cleanup lingering mounts after Jenkins aborts our build job. In a future patch we will have mount/jail cleanup as part of make. The network/system files in /etc that need frequent refreshing are now updated in systemplate to make their most recent version available in the jails. These files can change during the course of loolwsd lifetime, and are unlikely to be updated in systemplate after installation at all. We link to them in the systemplate/etc directory, and if that fails, we copy them before forking each kit instance to have the latest. This reworks the approach used to bind-mount the jails and the templates such that the total is now down to only three mounts: systemplate, lo, tmp. As now systemplate and lotemplate are shared, they must be mounted as readonly, this means that user/ must now be moved into tmp/user/ which is writable. The mount-points must be recursive, because we mount lo/ within the mount-point of systemplate (which is the root of the jail). But because we (re)bind recursively, and because both systemplate and lotemplate are mounted for each jails, we need to make them unbindable, so they wouldn't multiply the mount-points for each jails (an explosive growth!) Contrarywise, we don't want the mount-points to be shared, because we don't expect to add/remove mounts after a jail is created. The random temp directory is now created and set correctly, plus many logging and other improvements. Change-Id: Iae3fda5e876cf47d2cae6669a87b5b826a8748df Reviewed-on: https://gerrit.libreoffice.org/c/online/+/92829 Tested-by: Jenkins Tested-by: Jenkins CollaboraOffice <jenkinscollaboraoffice@gmail.com> Reviewed-by: Ashod Nakashian <ashnakash@gmail.com>
2020-04-09 08:02:58 -05:00
#include <cerrno>
#include <chrono>
#include <string>
wsd: faster jail setup via bind-mount loolmount now works and supports mounting and unmounting, plus numerous improvements, refactoring, logging, etc.. When enabled, binding improves the jail setup time by anywhere from 2x to orders of magnitude (in docker, f.e.). A new config entry mount_jail_tree controls whether mounting is used or the old method of linking/copying of jail contents. It is set to true by default and falls back to linking/copying. A test mount is done when the setting is enabled, and if mounting fails, it's disabled to avoid noise. Temporarily disabled for unit-tests until we can cleanup lingering mounts after Jenkins aborts our build job. In a future patch we will have mount/jail cleanup as part of make. The network/system files in /etc that need frequent refreshing are now updated in systemplate to make their most recent version available in the jails. These files can change during the course of loolwsd lifetime, and are unlikely to be updated in systemplate after installation at all. We link to them in the systemplate/etc directory, and if that fails, we copy them before forking each kit instance to have the latest. This reworks the approach used to bind-mount the jails and the templates such that the total is now down to only three mounts: systemplate, lo, tmp. As now systemplate and lotemplate are shared, they must be mounted as readonly, this means that user/ must now be moved into tmp/user/ which is writable. The mount-points must be recursive, because we mount lo/ within the mount-point of systemplate (which is the root of the jail). But because we (re)bind recursively, and because both systemplate and lotemplate are mounted for each jails, we need to make them unbindable, so they wouldn't multiply the mount-points for each jails (an explosive growth!) Contrarywise, we don't want the mount-points to be shared, because we don't expect to add/remove mounts after a jail is created. The random temp directory is now created and set correctly, plus many logging and other improvements. Change-Id: Iae3fda5e876cf47d2cae6669a87b5b826a8748df Reviewed-on: https://gerrit.libreoffice.org/c/online/+/92829 Tested-by: Jenkins Tested-by: Jenkins CollaboraOffice <jenkinscollaboraoffice@gmail.com> Reviewed-by: Ashod Nakashian <ashnakash@gmail.com>
2020-04-09 08:02:58 -05:00
#include <sys/stat.h>
#include <Poco/Path.h>
#include "Log.hpp"
namespace FileUtil
{
/// Used for anonymizing URLs
void setUrlAnonymization(bool anonymize, const std::uint64_t salt);
/// Anonymize the basename of filenames, preserving the path and extension.
std::string anonymizeUrl(const std::string& url);
/// Anonymize user names and IDs.
/// Will use the Obfuscated User ID if one is provided via WOPI.
std::string anonymizeUsername(const std::string& username);
/// Create a secure, random directory path.
std::string createRandomDir(const std::string& path);
// Save data to a file (overwriting an existing file if necessary) with checks for errors. Write
// to a temporary file in the same directory that is then atomically renamed to the desired name
// if everything goes well. In case of any error, both the destination file (if it already
// exists) and the temporary file (if was created, or existed already) are removed. Return true
// if everything succeeded.
bool saveDataToFileSafely(const std::string& fileName, const char* data, std::size_t size);
// We work around some of the mess of using the same sources both on the server side and in unit
// tests with conditional compilation based on BUILDING_TESTS.
// Add the file system that 'path' is located on to a list of file systems that are periodically
// checked for available space. The list is initially empty.
void registerFileSystemForDiskSpaceChecks(const std::string& path);
// Perform the check. If the free space on any of the registered file systems is below 5%, call
// 'alertAllUsers("internal", "diskfull")'. The check will be made no more often than once a
// minute if cacheLastCheck is set to true.
std::string checkDiskSpaceOnRegisteredFileSystems(const bool cacheLastCheck = true);
// Check disk space on a specific file system, the one where 'path' is located. This does not
// add that file system to the list used by 'registerFileSystemForDiskSpaceChecks'. If the free
// space on the file system is below 5%, return false, otherwise true. Note that this function
// does not call 'alertAllUsers'.
bool checkDiskSpace(const std::string& path);
/// Safely remove a file or directory.
/// Suppresses exception when the file is already removed.
/// This can happen when there is a race (unavoidable) or when
/// we don't care to check before we remove (when no race exists).
void removeFile(const std::string& path, const bool recursive = false);
inline void removeFile(const Poco::Path& path, const bool recursive = false)
{
removeFile(path.toString(), recursive);
}
wsd: faster jail setup via bind-mount loolmount now works and supports mounting and unmounting, plus numerous improvements, refactoring, logging, etc.. When enabled, binding improves the jail setup time by anywhere from 2x to orders of magnitude (in docker, f.e.). A new config entry mount_jail_tree controls whether mounting is used or the old method of linking/copying of jail contents. It is set to true by default and falls back to linking/copying. A test mount is done when the setting is enabled, and if mounting fails, it's disabled to avoid noise. Temporarily disabled for unit-tests until we can cleanup lingering mounts after Jenkins aborts our build job. In a future patch we will have mount/jail cleanup as part of make. The network/system files in /etc that need frequent refreshing are now updated in systemplate to make their most recent version available in the jails. These files can change during the course of loolwsd lifetime, and are unlikely to be updated in systemplate after installation at all. We link to them in the systemplate/etc directory, and if that fails, we copy them before forking each kit instance to have the latest. This reworks the approach used to bind-mount the jails and the templates such that the total is now down to only three mounts: systemplate, lo, tmp. As now systemplate and lotemplate are shared, they must be mounted as readonly, this means that user/ must now be moved into tmp/user/ which is writable. The mount-points must be recursive, because we mount lo/ within the mount-point of systemplate (which is the root of the jail). But because we (re)bind recursively, and because both systemplate and lotemplate are mounted for each jails, we need to make them unbindable, so they wouldn't multiply the mount-points for each jails (an explosive growth!) Contrarywise, we don't want the mount-points to be shared, because we don't expect to add/remove mounts after a jail is created. The random temp directory is now created and set correctly, plus many logging and other improvements. Change-Id: Iae3fda5e876cf47d2cae6669a87b5b826a8748df Reviewed-on: https://gerrit.libreoffice.org/c/online/+/92829 Tested-by: Jenkins Tested-by: Jenkins CollaboraOffice <jenkinscollaboraoffice@gmail.com> Reviewed-by: Ashod Nakashian <ashnakash@gmail.com>
2020-04-09 08:02:58 -05:00
/// Returns true iff the directory is empty (or doesn't exist).
bool isEmptyDirectory(const char* path);
inline bool isEmptyDirectory(const std::string& path) { return isEmptyDirectory(path.c_str()); }
/// Returns true iff the path given is writable by our *real* UID.
bool isWritable(const char* path);
inline bool isWritable(const std::string& path) { return isWritable(path.c_str()); }
/// Update the access-time and modified-time metadata for the given file.
bool updateTimestamps(const std::string& filename, timespec tsAccess, timespec tsModified);
/// Copy the source file to the target.
bool copy(const std::string& fromPath, const std::string& toPath, bool log,
bool throw_on_error);
/// Atomically copy a file and optionally preserve its timestamps.
/// The file is copied with a temporary name, and then atomically renamed.
/// NOTE: toPath must be a valid filename, not a directory.
/// Does not log (except errors), does not throw. Returns true on success.
bool copyAtomic(const std::string& fromPath, const std::string& toPath,
bool preserveTimestamps);
/// Copy a file from @fromPath to @toPath, throws on failure.
inline void copyFileTo(const std::string& fromPath, const std::string& toPath)
{
copy(fromPath, toPath, /*log=*/true, /*throw_on_error=*/true);
}
/// Try to hard-link, and fallback to copying it linking fails.
/// Returns true iff either linking or copying succeeds.
inline bool linkOrCopyFile(const std::string& source, const std::string& newPath)
{
// first try a simple hard-link
if (link(source.c_str(), newPath.c_str()) == 0)
return true;
const auto onrre = errno;
LOG_DBG("Failed to link [" << source << "] to [" << newPath << "] ("
<< Util::symbolicErrno(onrre) << ": " << std::strerror(onrre)
<< "), will try to copy");
return FileUtil::copy(source, newPath, /*log=*/true, /*throw_on_error=*/false);
}
/// Returns the system temporary directory.
std::string getSysTempDirectoryPath();
/// Create randomized temporary directory in the root provided
/// with S_IRWXU (read, write, and execute by owner) permissions.
/// If root is empty, the current system temp directory is used.
std::string createRandomTmpDir(std::string root = std::string());
/// Create a temporary directory in the root provided
std::string createTmpDir(std::string dirName, std::string root = std::string());
test: thread-safe common shared test data This protects against memory corruptions, and a cascade of issues, such as the following: Attaching to process 56245 [New LWP 56246] [New LWP 56252] [New LWP 56253] [New LWP 56254] [New LWP 56362] [New LWP 56364] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". 0x00007fda228f7d2d in __GI___pthread_timedjoin_ex (threadid=140574279595776, thread_return=0x0, abstime=0x0, block=<optimized out>) at pthread_join_common.c:89 89 pthread_join_common.c: No such file or directory. Thread 7 (Thread 0x7fda197fa700 (LWP 56364)): #0 __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135 #1 0x00007fda228f9023 in __GI___pthread_mutex_lock (mutex=0x55e9937567e0 <SigHandlerTrap>) at ../nptl/pthread_mutex_lock.c:78 #2 0x000055e9934cd4fe in __gthread_mutex_lock (__mutex=0x55e9937567e0 <SigHandlerTrap>) at /usr/include/x86_64-linux-gnu/c++/7/bits/gthr-default.h:748 #3 std::mutex::lock (this=0x55e9937567e0 <SigHandlerTrap>) at /usr/include/c++/7/bits/std_mutex.h:103 #4 std::unique_lock<std::mutex>::lock (this=<synthetic pointer>) at /usr/include/c++/7/bits/std_mutex.h:267 #5 std::unique_lock<std::mutex>::unique_lock (__m=..., this=<synthetic pointer>) at /usr/include/c++/7/bits/std_mutex.h:197 #6 SigUtil::handleFatalSignal (signal=11) at common/SigUtil.cpp:214 #7 <signal handler called> #8 std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_capacity (this=<optimized out>, __capacity=0) at /usr/include/c++/7/bits/basic_string.h:200 #9 std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string (__str=..., this=<optimized out>) at /usr/include/c++/7/bits/basic_string.h:542 #10 std::_Construct<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > ( __p=<optimized out>) at /usr/include/c++/7/bits/stl_construct.h:75 #11 std::__uninitialized_copy<false>::__uninit_copy<std::move_iterator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*> (__result=0x7fd9f40017b0, __last=..., __first=...) at /usr/include/c++/7/bits/stl_uninitialized.h:83 #12 std::uninitialized_copy<std::move_iterator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*> (__result=<optimized out>, __last=..., __first=...) at /usr/include/c++/7/bits/stl_uninitialized.h:134 #13 std::__uninitialized_copy_a<std::move_iterator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > (__result=<optimized out>, __last=..., __first=...) at /usr/include/c++/7/bits/stl_uninitialized.h:289 #14 std::__uninitialized_move_if_noexcept_a<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > (__alloc=..., __result=<optimized out>, __last=0x55e995662850, __first=<optimized out>) at /usr/include/c++/7/bits/stl_uninitialized.h:312 #15 std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >::_M_realloc_insert<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&> ( this=this@entry=0x55e9937566f0 <FileUtil::getTempFilePath(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)::fileDeleter>, __position=<error: Cannot access memory at address 0x3735003238323a>, __args#0="/tmp/loadTorture_viewcursor.odp_725a6013_viewcursor.odp") at /usr/include/c++/7/bits/vector.tcc:424 #16 0x000055e9934aa829 in std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >::push_back (__x="/tmp/loadTorture_viewcursor.odp_725a6013_viewcursor.odp", this=0x55e9937566f0 <FileUtil::getTempFilePath(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)::fileDeleter>) at /usr/include/c++/7/bits/stl_vector.h:948 #17 (anonymous namespace)::FileDeleter::registerForDeletion (file="/tmp/loadTorture_viewcursor.odp_725a6013_viewcursor.odp", this=0x55e9937566f0 <FileUtil::getTempFilePath(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)::fileDeleter>) at common/FileUtil.cpp:69 #18 FileUtil::getTempFilePath (srcDir=..., srcFilename=..., dstFilenamePrefix=...) at common/FileUtil.cpp:97 #19 0x00007fda216b51c9 in helpers::getDocumentPathAndURL (docFilename="viewcursor.odp", documentPath="", documentURL="", prefix="loadTorture_viewcursor.odp_") at ./helpers.hpp:151 #20 0x00007fda216ad007 in UnitLoadTorture::loadTorture (this=<optimized out>, testname="loadTorture_viewcursor.odp ", docName="viewcursor.odp", thread_count=thread_count@entry=3, max_jitter_ms=<optimized out>, max_jitter_ms@entry=75) at UnitLoadTorture.cpp:41 #21 0x00007fda216ad7dd in UnitLoadTorture::<lambda()>::operator() (__closure=0x55e994dbc758) at UnitLoadTorture.cpp:186 #22 std::__invoke_impl<void, UnitLoadTorture::testLoadTorture()::<lambda()> > (__f=...) at /usr/include/c++/7/bits/invoke.h:60 #23 std::__invoke<UnitLoadTorture::testLoadTorture()::<lambda()> > (__fn=...) at /usr/include/c++/7/bits/invoke.h:95 #24 std::thread::_Invoker<std::tuple<UnitLoadTorture::testLoadTorture()::<lambda()> > >::_M_invoke<0> (this=0x55e994dbc758) at /usr/include/c++/7/thread:234 #25 std::thread::_Invoker<std::tuple<UnitLoadTorture::testLoadTorture()::<lambda()> > >::operator() (this=0x55e994dbc758) at /usr/include/c++/7/thread:243 #26 std::thread::_State_impl<std::thread::_Invoker<std::tuple<UnitLoadTorture::testLoadTorture()::<lambda()> > > >::_M_run(void) (this=0x55e994dbc750) at /usr/include/c++/7/thread:186 #27 0x00007fda22de366f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6 #28 0x00007fda228f66db in start_thread (arg=0x7fda197fa700) at pthread_create.c:463 #29 0x00007fda2261f88f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 Thread 6 (Thread 0x7fd9fffff700 (LWP 56362)): #0 __lll_lock_wait_private () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:95 #1 0x00007fda225952db in __GI___libc_malloc (bytes=bytes@entry=3635) at malloc.c:3063 #2 0x00007fda2262f364 in __backtrace_symbols (array=array@entry=0x7fd9ffffc540, size=size@entry=26) at backtracesyms.c:69 #3 0x000055e9934cc439 in SigUtil::dumpBacktrace () at common/SigUtil.cpp:253 #4 0x000055e9934cd5ae in SigUtil::handleFatalSignal (signal=6) at common/SigUtil.cpp:236 #5 <signal handler called> #6 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51 #7 0x00007fda2253e801 in __GI_abort () at abort.c:79 #8 0x00007fda22587897 in __libc_message (action=action@entry=do_abort, fmt=fmt@entry=0x7fda226b4b9a "%s ") at ../sysdeps/posix/libc_fatal.c:181 #9 0x00007fda2258e90a in malloc_printerr (str=str@entry=0x7fda226b2e0e "malloc(): memory corruption") at malloc.c:5350 #10 0x00007fda22592994 in _int_malloc (av=av@entry=0x7fda00000020, bytes=bytes@entry=32) at malloc.c:3738 #11 0x00007fda225952ed in __GI___libc_malloc (bytes=32) at malloc.c:3065 #12 0x00007fda22db9258 in operator new(unsigned long) () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6 #13 0x00007fda244adaae in __gnu_cxx::new_allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >::allocate (this=0x7fd9ffffe740, __n=1) at /usr/include/c++/5/ext/new_allocator.h:104 #14 0x00007fda244ad300 in __gnu_cxx::__alloc_traits<std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >::allocate (__a=..., __n=1) at /usr/include/c++/5/ext/alloc_traits.h:182 #15 0x00007fda244ac858 in std::_Vector_base<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >::_M_allocate (this=0x7fd9ffffe740, __n=1) at /usr/include/c++/5/bits/stl_vector.h:170 #16 0x00007fda244ab7c7 in std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >::_M_insert_aux (this=0x7fd9ffffe740, __position=non-dereferenceable iterator for std::vector, __x="home") at /usr/include/c++/5/bits/vector.tcc:353 #17 0x00007fda244aada2 in std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >::push_back (this=0x7fd9ffffe740, __x="home") at /usr/include/c++/5/bits/stl_vector.h:925 #18 0x00007fda2350ddcb in Poco::Path::pushDirectory (this=0x7fd9ffffe6c0, dir="home") at src/Path.cpp:471 #19 0x00007fda2350ea27 in Poco::Path::parseUnix (this=0x7fd9ffffe6c0, path="/home/ash/prj/lo/online/test/") at src/Path.cpp:671 #20 0x00007fda2350cd6b in Poco::Path::assign (this=0x7fd9ffffe6c0, path="/home/ash/prj/lo/online/test/") at src/Path.cpp:182 #21 0x00007fda2350c1be in Poco::Path::Path (this=0x7fd9ffffe6c0, path="/home/ash/prj/lo/online/test/") at src/Path.cpp:54 #22 0x00007fda2350d33a in Poco::Path::makeAbsolute (this=0x7fd9ffffe8f0) at src/Path.cpp:318 #23 0x00007fda216b52ca in helpers::getDocumentPathAndURL (docFilename="setclientpart.ods", documentPath="/tmp/loadTorture_setclientpart.ods_ae70d2e_setclientpart.ods", documentURL="", prefix="loadTorture_setclientpart.ods_") at ./helpers.hpp:153 #24 0x00007fda216ad007 in UnitLoadTorture::loadTorture (this=<optimized out>, testname="loadTorture_setclientpart.ods ", docName="setclientpart.ods", thread_count=thread_count@entry=3, max_jitter_ms=<optimized out>, max_jitter_ms@entry=75) at UnitLoadTorture.cpp:41 #25 0x00007fda216ad7dd in UnitLoadTorture::<lambda()>::operator() (__closure=0x55e994cffe18) at UnitLoadTorture.cpp:186 #26 std::__invoke_impl<void, UnitLoadTorture::testLoadTorture()::<lambda()> > (__f=...) at /usr/include/c++/7/bits/invoke.h:60 #27 std::__invoke<UnitLoadTorture::testLoadTorture()::<lambda()> > (__fn=...) at /usr/include/c++/7/bits/invoke.h:95 #28 std::thread::_Invoker<std::tuple<UnitLoadTorture::testLoadTorture()::<lambda()> > >::_M_invoke<0> (this=0x55e994cffe18) at /usr/include/c++/7/thread:234 #29 std::thread::_Invoker<std::tuple<UnitLoadTorture::testLoadTorture()::<lambda()> > >::operator() (this=0x55e994cffe18) at /usr/include/c++/7/thread:243 #30 std::thread::_State_impl<std::thread::_Invoker<std::tuple<UnitLoadTorture::testLoadTorture()::<lambda()> > > >::_M_run(void) (this=0x55e994cffe10) at /usr/include/c++/7/thread:186 #31 0x00007fda22de366f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6 #32 0x00007fda228f66db in start_thread (arg=0x7fd9fffff700) at pthread_create.c:463 #33 0x00007fda2261f88f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 Change-Id: Ib60954e49e692082e686cb3fdc5fc9d5cfbc83c9 Reviewed-on: https://gerrit.libreoffice.org/83047 Reviewed-by: Michael Meeks <michael.meeks@collabora.com> Tested-by: Michael Meeks <michael.meeks@collabora.com>
2019-11-17 19:32:54 -06:00
/// Make a temp copy of a file, and prepend it with a prefix.
/// Used by tests to avoid tainting the originals.
std::string getTempFileCopyPath(const std::string& srcDir, const std::string& srcFilename,
test: thread-safe common shared test data This protects against memory corruptions, and a cascade of issues, such as the following: Attaching to process 56245 [New LWP 56246] [New LWP 56252] [New LWP 56253] [New LWP 56254] [New LWP 56362] [New LWP 56364] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". 0x00007fda228f7d2d in __GI___pthread_timedjoin_ex (threadid=140574279595776, thread_return=0x0, abstime=0x0, block=<optimized out>) at pthread_join_common.c:89 89 pthread_join_common.c: No such file or directory. Thread 7 (Thread 0x7fda197fa700 (LWP 56364)): #0 __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135 #1 0x00007fda228f9023 in __GI___pthread_mutex_lock (mutex=0x55e9937567e0 <SigHandlerTrap>) at ../nptl/pthread_mutex_lock.c:78 #2 0x000055e9934cd4fe in __gthread_mutex_lock (__mutex=0x55e9937567e0 <SigHandlerTrap>) at /usr/include/x86_64-linux-gnu/c++/7/bits/gthr-default.h:748 #3 std::mutex::lock (this=0x55e9937567e0 <SigHandlerTrap>) at /usr/include/c++/7/bits/std_mutex.h:103 #4 std::unique_lock<std::mutex>::lock (this=<synthetic pointer>) at /usr/include/c++/7/bits/std_mutex.h:267 #5 std::unique_lock<std::mutex>::unique_lock (__m=..., this=<synthetic pointer>) at /usr/include/c++/7/bits/std_mutex.h:197 #6 SigUtil::handleFatalSignal (signal=11) at common/SigUtil.cpp:214 #7 <signal handler called> #8 std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_capacity (this=<optimized out>, __capacity=0) at /usr/include/c++/7/bits/basic_string.h:200 #9 std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string (__str=..., this=<optimized out>) at /usr/include/c++/7/bits/basic_string.h:542 #10 std::_Construct<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > ( __p=<optimized out>) at /usr/include/c++/7/bits/stl_construct.h:75 #11 std::__uninitialized_copy<false>::__uninit_copy<std::move_iterator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*> (__result=0x7fd9f40017b0, __last=..., __first=...) at /usr/include/c++/7/bits/stl_uninitialized.h:83 #12 std::uninitialized_copy<std::move_iterator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*> (__result=<optimized out>, __last=..., __first=...) at /usr/include/c++/7/bits/stl_uninitialized.h:134 #13 std::__uninitialized_copy_a<std::move_iterator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > (__result=<optimized out>, __last=..., __first=...) at /usr/include/c++/7/bits/stl_uninitialized.h:289 #14 std::__uninitialized_move_if_noexcept_a<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > (__alloc=..., __result=<optimized out>, __last=0x55e995662850, __first=<optimized out>) at /usr/include/c++/7/bits/stl_uninitialized.h:312 #15 std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >::_M_realloc_insert<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&> ( this=this@entry=0x55e9937566f0 <FileUtil::getTempFilePath(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)::fileDeleter>, __position=<error: Cannot access memory at address 0x3735003238323a>, __args#0="/tmp/loadTorture_viewcursor.odp_725a6013_viewcursor.odp") at /usr/include/c++/7/bits/vector.tcc:424 #16 0x000055e9934aa829 in std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >::push_back (__x="/tmp/loadTorture_viewcursor.odp_725a6013_viewcursor.odp", this=0x55e9937566f0 <FileUtil::getTempFilePath(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)::fileDeleter>) at /usr/include/c++/7/bits/stl_vector.h:948 #17 (anonymous namespace)::FileDeleter::registerForDeletion (file="/tmp/loadTorture_viewcursor.odp_725a6013_viewcursor.odp", this=0x55e9937566f0 <FileUtil::getTempFilePath(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)::fileDeleter>) at common/FileUtil.cpp:69 #18 FileUtil::getTempFilePath (srcDir=..., srcFilename=..., dstFilenamePrefix=...) at common/FileUtil.cpp:97 #19 0x00007fda216b51c9 in helpers::getDocumentPathAndURL (docFilename="viewcursor.odp", documentPath="", documentURL="", prefix="loadTorture_viewcursor.odp_") at ./helpers.hpp:151 #20 0x00007fda216ad007 in UnitLoadTorture::loadTorture (this=<optimized out>, testname="loadTorture_viewcursor.odp ", docName="viewcursor.odp", thread_count=thread_count@entry=3, max_jitter_ms=<optimized out>, max_jitter_ms@entry=75) at UnitLoadTorture.cpp:41 #21 0x00007fda216ad7dd in UnitLoadTorture::<lambda()>::operator() (__closure=0x55e994dbc758) at UnitLoadTorture.cpp:186 #22 std::__invoke_impl<void, UnitLoadTorture::testLoadTorture()::<lambda()> > (__f=...) at /usr/include/c++/7/bits/invoke.h:60 #23 std::__invoke<UnitLoadTorture::testLoadTorture()::<lambda()> > (__fn=...) at /usr/include/c++/7/bits/invoke.h:95 #24 std::thread::_Invoker<std::tuple<UnitLoadTorture::testLoadTorture()::<lambda()> > >::_M_invoke<0> (this=0x55e994dbc758) at /usr/include/c++/7/thread:234 #25 std::thread::_Invoker<std::tuple<UnitLoadTorture::testLoadTorture()::<lambda()> > >::operator() (this=0x55e994dbc758) at /usr/include/c++/7/thread:243 #26 std::thread::_State_impl<std::thread::_Invoker<std::tuple<UnitLoadTorture::testLoadTorture()::<lambda()> > > >::_M_run(void) (this=0x55e994dbc750) at /usr/include/c++/7/thread:186 #27 0x00007fda22de366f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6 #28 0x00007fda228f66db in start_thread (arg=0x7fda197fa700) at pthread_create.c:463 #29 0x00007fda2261f88f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 Thread 6 (Thread 0x7fd9fffff700 (LWP 56362)): #0 __lll_lock_wait_private () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:95 #1 0x00007fda225952db in __GI___libc_malloc (bytes=bytes@entry=3635) at malloc.c:3063 #2 0x00007fda2262f364 in __backtrace_symbols (array=array@entry=0x7fd9ffffc540, size=size@entry=26) at backtracesyms.c:69 #3 0x000055e9934cc439 in SigUtil::dumpBacktrace () at common/SigUtil.cpp:253 #4 0x000055e9934cd5ae in SigUtil::handleFatalSignal (signal=6) at common/SigUtil.cpp:236 #5 <signal handler called> #6 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51 #7 0x00007fda2253e801 in __GI_abort () at abort.c:79 #8 0x00007fda22587897 in __libc_message (action=action@entry=do_abort, fmt=fmt@entry=0x7fda226b4b9a "%s ") at ../sysdeps/posix/libc_fatal.c:181 #9 0x00007fda2258e90a in malloc_printerr (str=str@entry=0x7fda226b2e0e "malloc(): memory corruption") at malloc.c:5350 #10 0x00007fda22592994 in _int_malloc (av=av@entry=0x7fda00000020, bytes=bytes@entry=32) at malloc.c:3738 #11 0x00007fda225952ed in __GI___libc_malloc (bytes=32) at malloc.c:3065 #12 0x00007fda22db9258 in operator new(unsigned long) () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6 #13 0x00007fda244adaae in __gnu_cxx::new_allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >::allocate (this=0x7fd9ffffe740, __n=1) at /usr/include/c++/5/ext/new_allocator.h:104 #14 0x00007fda244ad300 in __gnu_cxx::__alloc_traits<std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >::allocate (__a=..., __n=1) at /usr/include/c++/5/ext/alloc_traits.h:182 #15 0x00007fda244ac858 in std::_Vector_base<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >::_M_allocate (this=0x7fd9ffffe740, __n=1) at /usr/include/c++/5/bits/stl_vector.h:170 #16 0x00007fda244ab7c7 in std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >::_M_insert_aux (this=0x7fd9ffffe740, __position=non-dereferenceable iterator for std::vector, __x="home") at /usr/include/c++/5/bits/vector.tcc:353 #17 0x00007fda244aada2 in std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >::push_back (this=0x7fd9ffffe740, __x="home") at /usr/include/c++/5/bits/stl_vector.h:925 #18 0x00007fda2350ddcb in Poco::Path::pushDirectory (this=0x7fd9ffffe6c0, dir="home") at src/Path.cpp:471 #19 0x00007fda2350ea27 in Poco::Path::parseUnix (this=0x7fd9ffffe6c0, path="/home/ash/prj/lo/online/test/") at src/Path.cpp:671 #20 0x00007fda2350cd6b in Poco::Path::assign (this=0x7fd9ffffe6c0, path="/home/ash/prj/lo/online/test/") at src/Path.cpp:182 #21 0x00007fda2350c1be in Poco::Path::Path (this=0x7fd9ffffe6c0, path="/home/ash/prj/lo/online/test/") at src/Path.cpp:54 #22 0x00007fda2350d33a in Poco::Path::makeAbsolute (this=0x7fd9ffffe8f0) at src/Path.cpp:318 #23 0x00007fda216b52ca in helpers::getDocumentPathAndURL (docFilename="setclientpart.ods", documentPath="/tmp/loadTorture_setclientpart.ods_ae70d2e_setclientpart.ods", documentURL="", prefix="loadTorture_setclientpart.ods_") at ./helpers.hpp:153 #24 0x00007fda216ad007 in UnitLoadTorture::loadTorture (this=<optimized out>, testname="loadTorture_setclientpart.ods ", docName="setclientpart.ods", thread_count=thread_count@entry=3, max_jitter_ms=<optimized out>, max_jitter_ms@entry=75) at UnitLoadTorture.cpp:41 #25 0x00007fda216ad7dd in UnitLoadTorture::<lambda()>::operator() (__closure=0x55e994cffe18) at UnitLoadTorture.cpp:186 #26 std::__invoke_impl<void, UnitLoadTorture::testLoadTorture()::<lambda()> > (__f=...) at /usr/include/c++/7/bits/invoke.h:60 #27 std::__invoke<UnitLoadTorture::testLoadTorture()::<lambda()> > (__fn=...) at /usr/include/c++/7/bits/invoke.h:95 #28 std::thread::_Invoker<std::tuple<UnitLoadTorture::testLoadTorture()::<lambda()> > >::_M_invoke<0> (this=0x55e994cffe18) at /usr/include/c++/7/thread:234 #29 std::thread::_Invoker<std::tuple<UnitLoadTorture::testLoadTorture()::<lambda()> > >::operator() (this=0x55e994cffe18) at /usr/include/c++/7/thread:243 #30 std::thread::_State_impl<std::thread::_Invoker<std::tuple<UnitLoadTorture::testLoadTorture()::<lambda()> > > >::_M_run(void) (this=0x55e994cffe10) at /usr/include/c++/7/thread:186 #31 0x00007fda22de366f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6 #32 0x00007fda228f66db in start_thread (arg=0x7fd9fffff700) at pthread_create.c:463 #33 0x00007fda2261f88f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 Change-Id: Ib60954e49e692082e686cb3fdc5fc9d5cfbc83c9 Reviewed-on: https://gerrit.libreoffice.org/83047 Reviewed-by: Michael Meeks <michael.meeks@collabora.com> Tested-by: Michael Meeks <michael.meeks@collabora.com>
2019-11-17 19:32:54 -06:00
const std::string& dstFilenamePrefix);
/// Make a temp copy of a file.
/// Used by tests to avoid tainting the originals.
/// srcDir shouldn't end with '/' and srcFilename shouldn't contain '/'.
/// Returns the created file path.
inline std::string getTempFileCopyPath(const std::string& srcDir, const std::string& srcFilename)
test: thread-safe common shared test data This protects against memory corruptions, and a cascade of issues, such as the following: Attaching to process 56245 [New LWP 56246] [New LWP 56252] [New LWP 56253] [New LWP 56254] [New LWP 56362] [New LWP 56364] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". 0x00007fda228f7d2d in __GI___pthread_timedjoin_ex (threadid=140574279595776, thread_return=0x0, abstime=0x0, block=<optimized out>) at pthread_join_common.c:89 89 pthread_join_common.c: No such file or directory. Thread 7 (Thread 0x7fda197fa700 (LWP 56364)): #0 __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135 #1 0x00007fda228f9023 in __GI___pthread_mutex_lock (mutex=0x55e9937567e0 <SigHandlerTrap>) at ../nptl/pthread_mutex_lock.c:78 #2 0x000055e9934cd4fe in __gthread_mutex_lock (__mutex=0x55e9937567e0 <SigHandlerTrap>) at /usr/include/x86_64-linux-gnu/c++/7/bits/gthr-default.h:748 #3 std::mutex::lock (this=0x55e9937567e0 <SigHandlerTrap>) at /usr/include/c++/7/bits/std_mutex.h:103 #4 std::unique_lock<std::mutex>::lock (this=<synthetic pointer>) at /usr/include/c++/7/bits/std_mutex.h:267 #5 std::unique_lock<std::mutex>::unique_lock (__m=..., this=<synthetic pointer>) at /usr/include/c++/7/bits/std_mutex.h:197 #6 SigUtil::handleFatalSignal (signal=11) at common/SigUtil.cpp:214 #7 <signal handler called> #8 std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_capacity (this=<optimized out>, __capacity=0) at /usr/include/c++/7/bits/basic_string.h:200 #9 std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string (__str=..., this=<optimized out>) at /usr/include/c++/7/bits/basic_string.h:542 #10 std::_Construct<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > ( __p=<optimized out>) at /usr/include/c++/7/bits/stl_construct.h:75 #11 std::__uninitialized_copy<false>::__uninit_copy<std::move_iterator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*> (__result=0x7fd9f40017b0, __last=..., __first=...) at /usr/include/c++/7/bits/stl_uninitialized.h:83 #12 std::uninitialized_copy<std::move_iterator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*> (__result=<optimized out>, __last=..., __first=...) at /usr/include/c++/7/bits/stl_uninitialized.h:134 #13 std::__uninitialized_copy_a<std::move_iterator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > (__result=<optimized out>, __last=..., __first=...) at /usr/include/c++/7/bits/stl_uninitialized.h:289 #14 std::__uninitialized_move_if_noexcept_a<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > (__alloc=..., __result=<optimized out>, __last=0x55e995662850, __first=<optimized out>) at /usr/include/c++/7/bits/stl_uninitialized.h:312 #15 std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >::_M_realloc_insert<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&> ( this=this@entry=0x55e9937566f0 <FileUtil::getTempFilePath(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)::fileDeleter>, __position=<error: Cannot access memory at address 0x3735003238323a>, __args#0="/tmp/loadTorture_viewcursor.odp_725a6013_viewcursor.odp") at /usr/include/c++/7/bits/vector.tcc:424 #16 0x000055e9934aa829 in std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >::push_back (__x="/tmp/loadTorture_viewcursor.odp_725a6013_viewcursor.odp", this=0x55e9937566f0 <FileUtil::getTempFilePath(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)::fileDeleter>) at /usr/include/c++/7/bits/stl_vector.h:948 #17 (anonymous namespace)::FileDeleter::registerForDeletion (file="/tmp/loadTorture_viewcursor.odp_725a6013_viewcursor.odp", this=0x55e9937566f0 <FileUtil::getTempFilePath(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)::fileDeleter>) at common/FileUtil.cpp:69 #18 FileUtil::getTempFilePath (srcDir=..., srcFilename=..., dstFilenamePrefix=...) at common/FileUtil.cpp:97 #19 0x00007fda216b51c9 in helpers::getDocumentPathAndURL (docFilename="viewcursor.odp", documentPath="", documentURL="", prefix="loadTorture_viewcursor.odp_") at ./helpers.hpp:151 #20 0x00007fda216ad007 in UnitLoadTorture::loadTorture (this=<optimized out>, testname="loadTorture_viewcursor.odp ", docName="viewcursor.odp", thread_count=thread_count@entry=3, max_jitter_ms=<optimized out>, max_jitter_ms@entry=75) at UnitLoadTorture.cpp:41 #21 0x00007fda216ad7dd in UnitLoadTorture::<lambda()>::operator() (__closure=0x55e994dbc758) at UnitLoadTorture.cpp:186 #22 std::__invoke_impl<void, UnitLoadTorture::testLoadTorture()::<lambda()> > (__f=...) at /usr/include/c++/7/bits/invoke.h:60 #23 std::__invoke<UnitLoadTorture::testLoadTorture()::<lambda()> > (__fn=...) at /usr/include/c++/7/bits/invoke.h:95 #24 std::thread::_Invoker<std::tuple<UnitLoadTorture::testLoadTorture()::<lambda()> > >::_M_invoke<0> (this=0x55e994dbc758) at /usr/include/c++/7/thread:234 #25 std::thread::_Invoker<std::tuple<UnitLoadTorture::testLoadTorture()::<lambda()> > >::operator() (this=0x55e994dbc758) at /usr/include/c++/7/thread:243 #26 std::thread::_State_impl<std::thread::_Invoker<std::tuple<UnitLoadTorture::testLoadTorture()::<lambda()> > > >::_M_run(void) (this=0x55e994dbc750) at /usr/include/c++/7/thread:186 #27 0x00007fda22de366f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6 #28 0x00007fda228f66db in start_thread (arg=0x7fda197fa700) at pthread_create.c:463 #29 0x00007fda2261f88f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 Thread 6 (Thread 0x7fd9fffff700 (LWP 56362)): #0 __lll_lock_wait_private () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:95 #1 0x00007fda225952db in __GI___libc_malloc (bytes=bytes@entry=3635) at malloc.c:3063 #2 0x00007fda2262f364 in __backtrace_symbols (array=array@entry=0x7fd9ffffc540, size=size@entry=26) at backtracesyms.c:69 #3 0x000055e9934cc439 in SigUtil::dumpBacktrace () at common/SigUtil.cpp:253 #4 0x000055e9934cd5ae in SigUtil::handleFatalSignal (signal=6) at common/SigUtil.cpp:236 #5 <signal handler called> #6 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51 #7 0x00007fda2253e801 in __GI_abort () at abort.c:79 #8 0x00007fda22587897 in __libc_message (action=action@entry=do_abort, fmt=fmt@entry=0x7fda226b4b9a "%s ") at ../sysdeps/posix/libc_fatal.c:181 #9 0x00007fda2258e90a in malloc_printerr (str=str@entry=0x7fda226b2e0e "malloc(): memory corruption") at malloc.c:5350 #10 0x00007fda22592994 in _int_malloc (av=av@entry=0x7fda00000020, bytes=bytes@entry=32) at malloc.c:3738 #11 0x00007fda225952ed in __GI___libc_malloc (bytes=32) at malloc.c:3065 #12 0x00007fda22db9258 in operator new(unsigned long) () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6 #13 0x00007fda244adaae in __gnu_cxx::new_allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >::allocate (this=0x7fd9ffffe740, __n=1) at /usr/include/c++/5/ext/new_allocator.h:104 #14 0x00007fda244ad300 in __gnu_cxx::__alloc_traits<std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >::allocate (__a=..., __n=1) at /usr/include/c++/5/ext/alloc_traits.h:182 #15 0x00007fda244ac858 in std::_Vector_base<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >::_M_allocate (this=0x7fd9ffffe740, __n=1) at /usr/include/c++/5/bits/stl_vector.h:170 #16 0x00007fda244ab7c7 in std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >::_M_insert_aux (this=0x7fd9ffffe740, __position=non-dereferenceable iterator for std::vector, __x="home") at /usr/include/c++/5/bits/vector.tcc:353 #17 0x00007fda244aada2 in std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >::push_back (this=0x7fd9ffffe740, __x="home") at /usr/include/c++/5/bits/stl_vector.h:925 #18 0x00007fda2350ddcb in Poco::Path::pushDirectory (this=0x7fd9ffffe6c0, dir="home") at src/Path.cpp:471 #19 0x00007fda2350ea27 in Poco::Path::parseUnix (this=0x7fd9ffffe6c0, path="/home/ash/prj/lo/online/test/") at src/Path.cpp:671 #20 0x00007fda2350cd6b in Poco::Path::assign (this=0x7fd9ffffe6c0, path="/home/ash/prj/lo/online/test/") at src/Path.cpp:182 #21 0x00007fda2350c1be in Poco::Path::Path (this=0x7fd9ffffe6c0, path="/home/ash/prj/lo/online/test/") at src/Path.cpp:54 #22 0x00007fda2350d33a in Poco::Path::makeAbsolute (this=0x7fd9ffffe8f0) at src/Path.cpp:318 #23 0x00007fda216b52ca in helpers::getDocumentPathAndURL (docFilename="setclientpart.ods", documentPath="/tmp/loadTorture_setclientpart.ods_ae70d2e_setclientpart.ods", documentURL="", prefix="loadTorture_setclientpart.ods_") at ./helpers.hpp:153 #24 0x00007fda216ad007 in UnitLoadTorture::loadTorture (this=<optimized out>, testname="loadTorture_setclientpart.ods ", docName="setclientpart.ods", thread_count=thread_count@entry=3, max_jitter_ms=<optimized out>, max_jitter_ms@entry=75) at UnitLoadTorture.cpp:41 #25 0x00007fda216ad7dd in UnitLoadTorture::<lambda()>::operator() (__closure=0x55e994cffe18) at UnitLoadTorture.cpp:186 #26 std::__invoke_impl<void, UnitLoadTorture::testLoadTorture()::<lambda()> > (__f=...) at /usr/include/c++/7/bits/invoke.h:60 #27 std::__invoke<UnitLoadTorture::testLoadTorture()::<lambda()> > (__fn=...) at /usr/include/c++/7/bits/invoke.h:95 #28 std::thread::_Invoker<std::tuple<UnitLoadTorture::testLoadTorture()::<lambda()> > >::_M_invoke<0> (this=0x55e994cffe18) at /usr/include/c++/7/thread:234 #29 std::thread::_Invoker<std::tuple<UnitLoadTorture::testLoadTorture()::<lambda()> > >::operator() (this=0x55e994cffe18) at /usr/include/c++/7/thread:243 #30 std::thread::_State_impl<std::thread::_Invoker<std::tuple<UnitLoadTorture::testLoadTorture()::<lambda()> > > >::_M_run(void) (this=0x55e994cffe10) at /usr/include/c++/7/thread:186 #31 0x00007fda22de366f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6 #32 0x00007fda228f66db in start_thread (arg=0x7fd9fffff700) at pthread_create.c:463 #33 0x00007fda2261f88f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 Change-Id: Ib60954e49e692082e686cb3fdc5fc9d5cfbc83c9 Reviewed-on: https://gerrit.libreoffice.org/83047 Reviewed-by: Michael Meeks <michael.meeks@collabora.com> Tested-by: Michael Meeks <michael.meeks@collabora.com>
2019-11-17 19:32:54 -06:00
{
return getTempFileCopyPath(srcDir, srcFilename, std::string());
test: thread-safe common shared test data This protects against memory corruptions, and a cascade of issues, such as the following: Attaching to process 56245 [New LWP 56246] [New LWP 56252] [New LWP 56253] [New LWP 56254] [New LWP 56362] [New LWP 56364] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". 0x00007fda228f7d2d in __GI___pthread_timedjoin_ex (threadid=140574279595776, thread_return=0x0, abstime=0x0, block=<optimized out>) at pthread_join_common.c:89 89 pthread_join_common.c: No such file or directory. Thread 7 (Thread 0x7fda197fa700 (LWP 56364)): #0 __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135 #1 0x00007fda228f9023 in __GI___pthread_mutex_lock (mutex=0x55e9937567e0 <SigHandlerTrap>) at ../nptl/pthread_mutex_lock.c:78 #2 0x000055e9934cd4fe in __gthread_mutex_lock (__mutex=0x55e9937567e0 <SigHandlerTrap>) at /usr/include/x86_64-linux-gnu/c++/7/bits/gthr-default.h:748 #3 std::mutex::lock (this=0x55e9937567e0 <SigHandlerTrap>) at /usr/include/c++/7/bits/std_mutex.h:103 #4 std::unique_lock<std::mutex>::lock (this=<synthetic pointer>) at /usr/include/c++/7/bits/std_mutex.h:267 #5 std::unique_lock<std::mutex>::unique_lock (__m=..., this=<synthetic pointer>) at /usr/include/c++/7/bits/std_mutex.h:197 #6 SigUtil::handleFatalSignal (signal=11) at common/SigUtil.cpp:214 #7 <signal handler called> #8 std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_capacity (this=<optimized out>, __capacity=0) at /usr/include/c++/7/bits/basic_string.h:200 #9 std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string (__str=..., this=<optimized out>) at /usr/include/c++/7/bits/basic_string.h:542 #10 std::_Construct<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > ( __p=<optimized out>) at /usr/include/c++/7/bits/stl_construct.h:75 #11 std::__uninitialized_copy<false>::__uninit_copy<std::move_iterator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*> (__result=0x7fd9f40017b0, __last=..., __first=...) at /usr/include/c++/7/bits/stl_uninitialized.h:83 #12 std::uninitialized_copy<std::move_iterator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*> (__result=<optimized out>, __last=..., __first=...) at /usr/include/c++/7/bits/stl_uninitialized.h:134 #13 std::__uninitialized_copy_a<std::move_iterator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > (__result=<optimized out>, __last=..., __first=...) at /usr/include/c++/7/bits/stl_uninitialized.h:289 #14 std::__uninitialized_move_if_noexcept_a<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > (__alloc=..., __result=<optimized out>, __last=0x55e995662850, __first=<optimized out>) at /usr/include/c++/7/bits/stl_uninitialized.h:312 #15 std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >::_M_realloc_insert<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&> ( this=this@entry=0x55e9937566f0 <FileUtil::getTempFilePath(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)::fileDeleter>, __position=<error: Cannot access memory at address 0x3735003238323a>, __args#0="/tmp/loadTorture_viewcursor.odp_725a6013_viewcursor.odp") at /usr/include/c++/7/bits/vector.tcc:424 #16 0x000055e9934aa829 in std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >::push_back (__x="/tmp/loadTorture_viewcursor.odp_725a6013_viewcursor.odp", this=0x55e9937566f0 <FileUtil::getTempFilePath(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)::fileDeleter>) at /usr/include/c++/7/bits/stl_vector.h:948 #17 (anonymous namespace)::FileDeleter::registerForDeletion (file="/tmp/loadTorture_viewcursor.odp_725a6013_viewcursor.odp", this=0x55e9937566f0 <FileUtil::getTempFilePath(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)::fileDeleter>) at common/FileUtil.cpp:69 #18 FileUtil::getTempFilePath (srcDir=..., srcFilename=..., dstFilenamePrefix=...) at common/FileUtil.cpp:97 #19 0x00007fda216b51c9 in helpers::getDocumentPathAndURL (docFilename="viewcursor.odp", documentPath="", documentURL="", prefix="loadTorture_viewcursor.odp_") at ./helpers.hpp:151 #20 0x00007fda216ad007 in UnitLoadTorture::loadTorture (this=<optimized out>, testname="loadTorture_viewcursor.odp ", docName="viewcursor.odp", thread_count=thread_count@entry=3, max_jitter_ms=<optimized out>, max_jitter_ms@entry=75) at UnitLoadTorture.cpp:41 #21 0x00007fda216ad7dd in UnitLoadTorture::<lambda()>::operator() (__closure=0x55e994dbc758) at UnitLoadTorture.cpp:186 #22 std::__invoke_impl<void, UnitLoadTorture::testLoadTorture()::<lambda()> > (__f=...) at /usr/include/c++/7/bits/invoke.h:60 #23 std::__invoke<UnitLoadTorture::testLoadTorture()::<lambda()> > (__fn=...) at /usr/include/c++/7/bits/invoke.h:95 #24 std::thread::_Invoker<std::tuple<UnitLoadTorture::testLoadTorture()::<lambda()> > >::_M_invoke<0> (this=0x55e994dbc758) at /usr/include/c++/7/thread:234 #25 std::thread::_Invoker<std::tuple<UnitLoadTorture::testLoadTorture()::<lambda()> > >::operator() (this=0x55e994dbc758) at /usr/include/c++/7/thread:243 #26 std::thread::_State_impl<std::thread::_Invoker<std::tuple<UnitLoadTorture::testLoadTorture()::<lambda()> > > >::_M_run(void) (this=0x55e994dbc750) at /usr/include/c++/7/thread:186 #27 0x00007fda22de366f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6 #28 0x00007fda228f66db in start_thread (arg=0x7fda197fa700) at pthread_create.c:463 #29 0x00007fda2261f88f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 Thread 6 (Thread 0x7fd9fffff700 (LWP 56362)): #0 __lll_lock_wait_private () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:95 #1 0x00007fda225952db in __GI___libc_malloc (bytes=bytes@entry=3635) at malloc.c:3063 #2 0x00007fda2262f364 in __backtrace_symbols (array=array@entry=0x7fd9ffffc540, size=size@entry=26) at backtracesyms.c:69 #3 0x000055e9934cc439 in SigUtil::dumpBacktrace () at common/SigUtil.cpp:253 #4 0x000055e9934cd5ae in SigUtil::handleFatalSignal (signal=6) at common/SigUtil.cpp:236 #5 <signal handler called> #6 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51 #7 0x00007fda2253e801 in __GI_abort () at abort.c:79 #8 0x00007fda22587897 in __libc_message (action=action@entry=do_abort, fmt=fmt@entry=0x7fda226b4b9a "%s ") at ../sysdeps/posix/libc_fatal.c:181 #9 0x00007fda2258e90a in malloc_printerr (str=str@entry=0x7fda226b2e0e "malloc(): memory corruption") at malloc.c:5350 #10 0x00007fda22592994 in _int_malloc (av=av@entry=0x7fda00000020, bytes=bytes@entry=32) at malloc.c:3738 #11 0x00007fda225952ed in __GI___libc_malloc (bytes=32) at malloc.c:3065 #12 0x00007fda22db9258 in operator new(unsigned long) () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6 #13 0x00007fda244adaae in __gnu_cxx::new_allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >::allocate (this=0x7fd9ffffe740, __n=1) at /usr/include/c++/5/ext/new_allocator.h:104 #14 0x00007fda244ad300 in __gnu_cxx::__alloc_traits<std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >::allocate (__a=..., __n=1) at /usr/include/c++/5/ext/alloc_traits.h:182 #15 0x00007fda244ac858 in std::_Vector_base<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >::_M_allocate (this=0x7fd9ffffe740, __n=1) at /usr/include/c++/5/bits/stl_vector.h:170 #16 0x00007fda244ab7c7 in std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >::_M_insert_aux (this=0x7fd9ffffe740, __position=non-dereferenceable iterator for std::vector, __x="home") at /usr/include/c++/5/bits/vector.tcc:353 #17 0x00007fda244aada2 in std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >::push_back (this=0x7fd9ffffe740, __x="home") at /usr/include/c++/5/bits/stl_vector.h:925 #18 0x00007fda2350ddcb in Poco::Path::pushDirectory (this=0x7fd9ffffe6c0, dir="home") at src/Path.cpp:471 #19 0x00007fda2350ea27 in Poco::Path::parseUnix (this=0x7fd9ffffe6c0, path="/home/ash/prj/lo/online/test/") at src/Path.cpp:671 #20 0x00007fda2350cd6b in Poco::Path::assign (this=0x7fd9ffffe6c0, path="/home/ash/prj/lo/online/test/") at src/Path.cpp:182 #21 0x00007fda2350c1be in Poco::Path::Path (this=0x7fd9ffffe6c0, path="/home/ash/prj/lo/online/test/") at src/Path.cpp:54 #22 0x00007fda2350d33a in Poco::Path::makeAbsolute (this=0x7fd9ffffe8f0) at src/Path.cpp:318 #23 0x00007fda216b52ca in helpers::getDocumentPathAndURL (docFilename="setclientpart.ods", documentPath="/tmp/loadTorture_setclientpart.ods_ae70d2e_setclientpart.ods", documentURL="", prefix="loadTorture_setclientpart.ods_") at ./helpers.hpp:153 #24 0x00007fda216ad007 in UnitLoadTorture::loadTorture (this=<optimized out>, testname="loadTorture_setclientpart.ods ", docName="setclientpart.ods", thread_count=thread_count@entry=3, max_jitter_ms=<optimized out>, max_jitter_ms@entry=75) at UnitLoadTorture.cpp:41 #25 0x00007fda216ad7dd in UnitLoadTorture::<lambda()>::operator() (__closure=0x55e994cffe18) at UnitLoadTorture.cpp:186 #26 std::__invoke_impl<void, UnitLoadTorture::testLoadTorture()::<lambda()> > (__f=...) at /usr/include/c++/7/bits/invoke.h:60 #27 std::__invoke<UnitLoadTorture::testLoadTorture()::<lambda()> > (__fn=...) at /usr/include/c++/7/bits/invoke.h:95 #28 std::thread::_Invoker<std::tuple<UnitLoadTorture::testLoadTorture()::<lambda()> > >::_M_invoke<0> (this=0x55e994cffe18) at /usr/include/c++/7/thread:234 #29 std::thread::_Invoker<std::tuple<UnitLoadTorture::testLoadTorture()::<lambda()> > >::operator() (this=0x55e994cffe18) at /usr/include/c++/7/thread:243 #30 std::thread::_State_impl<std::thread::_Invoker<std::tuple<UnitLoadTorture::testLoadTorture()::<lambda()> > > >::_M_run(void) (this=0x55e994cffe10) at /usr/include/c++/7/thread:186 #31 0x00007fda22de366f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6 #32 0x00007fda228f66db in start_thread (arg=0x7fd9fffff700) at pthread_create.c:463 #33 0x00007fda2261f88f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 Change-Id: Ib60954e49e692082e686cb3fdc5fc9d5cfbc83c9 Reviewed-on: https://gerrit.libreoffice.org/83047 Reviewed-by: Michael Meeks <michael.meeks@collabora.com> Tested-by: Michael Meeks <michael.meeks@collabora.com>
2019-11-17 19:32:54 -06:00
}
/// Returns the realpath(3) of the provided path.
std::string realpath(const char* path);
inline std::string realpath(const std::string& path)
{
return realpath(path.c_str());
}
/// Returns file extension from the path
std::string extractFileExtension(const std::string& path);
/// Returns true iff the two files both exist, can be read,
/// have equal size and every byte of their contents match.
bool compareFileContents(const std::string& rhsPath, const std::string& lhsPath);
/// Reads the whole file to memory. Only for small files.
std::unique_ptr<std::vector<char>> readFile(const std::string& path, int maxSize = 256 * 1024);
wsd: faster jail setup via bind-mount loolmount now works and supports mounting and unmounting, plus numerous improvements, refactoring, logging, etc.. When enabled, binding improves the jail setup time by anywhere from 2x to orders of magnitude (in docker, f.e.). A new config entry mount_jail_tree controls whether mounting is used or the old method of linking/copying of jail contents. It is set to true by default and falls back to linking/copying. A test mount is done when the setting is enabled, and if mounting fails, it's disabled to avoid noise. Temporarily disabled for unit-tests until we can cleanup lingering mounts after Jenkins aborts our build job. In a future patch we will have mount/jail cleanup as part of make. The network/system files in /etc that need frequent refreshing are now updated in systemplate to make their most recent version available in the jails. These files can change during the course of loolwsd lifetime, and are unlikely to be updated in systemplate after installation at all. We link to them in the systemplate/etc directory, and if that fails, we copy them before forking each kit instance to have the latest. This reworks the approach used to bind-mount the jails and the templates such that the total is now down to only three mounts: systemplate, lo, tmp. As now systemplate and lotemplate are shared, they must be mounted as readonly, this means that user/ must now be moved into tmp/user/ which is writable. The mount-points must be recursive, because we mount lo/ within the mount-point of systemplate (which is the root of the jail). But because we (re)bind recursively, and because both systemplate and lotemplate are mounted for each jails, we need to make them unbindable, so they wouldn't multiply the mount-points for each jails (an explosive growth!) Contrarywise, we don't want the mount-points to be shared, because we don't expect to add/remove mounts after a jail is created. The random temp directory is now created and set correctly, plus many logging and other improvements. Change-Id: Iae3fda5e876cf47d2cae6669a87b5b826a8748df Reviewed-on: https://gerrit.libreoffice.org/c/online/+/92829 Tested-by: Jenkins Tested-by: Jenkins CollaboraOffice <jenkinscollaboraoffice@gmail.com> Reviewed-by: Ashod Nakashian <ashnakash@gmail.com>
2020-04-09 08:02:58 -05:00
/// File/Directory stat helper.
class Stat
{
int clearStat() { memset (&_sb, 0, sizeof(_sb)); return 0; }
wsd: faster jail setup via bind-mount loolmount now works and supports mounting and unmounting, plus numerous improvements, refactoring, logging, etc.. When enabled, binding improves the jail setup time by anywhere from 2x to orders of magnitude (in docker, f.e.). A new config entry mount_jail_tree controls whether mounting is used or the old method of linking/copying of jail contents. It is set to true by default and falls back to linking/copying. A test mount is done when the setting is enabled, and if mounting fails, it's disabled to avoid noise. Temporarily disabled for unit-tests until we can cleanup lingering mounts after Jenkins aborts our build job. In a future patch we will have mount/jail cleanup as part of make. The network/system files in /etc that need frequent refreshing are now updated in systemplate to make their most recent version available in the jails. These files can change during the course of loolwsd lifetime, and are unlikely to be updated in systemplate after installation at all. We link to them in the systemplate/etc directory, and if that fails, we copy them before forking each kit instance to have the latest. This reworks the approach used to bind-mount the jails and the templates such that the total is now down to only three mounts: systemplate, lo, tmp. As now systemplate and lotemplate are shared, they must be mounted as readonly, this means that user/ must now be moved into tmp/user/ which is writable. The mount-points must be recursive, because we mount lo/ within the mount-point of systemplate (which is the root of the jail). But because we (re)bind recursively, and because both systemplate and lotemplate are mounted for each jails, we need to make them unbindable, so they wouldn't multiply the mount-points for each jails (an explosive growth!) Contrarywise, we don't want the mount-points to be shared, because we don't expect to add/remove mounts after a jail is created. The random temp directory is now created and set correctly, plus many logging and other improvements. Change-Id: Iae3fda5e876cf47d2cae6669a87b5b826a8748df Reviewed-on: https://gerrit.libreoffice.org/c/online/+/92829 Tested-by: Jenkins Tested-by: Jenkins CollaboraOffice <jenkinscollaboraoffice@gmail.com> Reviewed-by: Ashod Nakashian <ashnakash@gmail.com>
2020-04-09 08:02:58 -05:00
public:
/// Stat the given path. Symbolic links are stat'ed when @link is true.
wsd: faster jail setup via bind-mount loolmount now works and supports mounting and unmounting, plus numerous improvements, refactoring, logging, etc.. When enabled, binding improves the jail setup time by anywhere from 2x to orders of magnitude (in docker, f.e.). A new config entry mount_jail_tree controls whether mounting is used or the old method of linking/copying of jail contents. It is set to true by default and falls back to linking/copying. A test mount is done when the setting is enabled, and if mounting fails, it's disabled to avoid noise. Temporarily disabled for unit-tests until we can cleanup lingering mounts after Jenkins aborts our build job. In a future patch we will have mount/jail cleanup as part of make. The network/system files in /etc that need frequent refreshing are now updated in systemplate to make their most recent version available in the jails. These files can change during the course of loolwsd lifetime, and are unlikely to be updated in systemplate after installation at all. We link to them in the systemplate/etc directory, and if that fails, we copy them before forking each kit instance to have the latest. This reworks the approach used to bind-mount the jails and the templates such that the total is now down to only three mounts: systemplate, lo, tmp. As now systemplate and lotemplate are shared, they must be mounted as readonly, this means that user/ must now be moved into tmp/user/ which is writable. The mount-points must be recursive, because we mount lo/ within the mount-point of systemplate (which is the root of the jail). But because we (re)bind recursively, and because both systemplate and lotemplate are mounted for each jails, we need to make them unbindable, so they wouldn't multiply the mount-points for each jails (an explosive growth!) Contrarywise, we don't want the mount-points to be shared, because we don't expect to add/remove mounts after a jail is created. The random temp directory is now created and set correctly, plus many logging and other improvements. Change-Id: Iae3fda5e876cf47d2cae6669a87b5b826a8748df Reviewed-on: https://gerrit.libreoffice.org/c/online/+/92829 Tested-by: Jenkins Tested-by: Jenkins CollaboraOffice <jenkinscollaboraoffice@gmail.com> Reviewed-by: Ashod Nakashian <ashnakash@gmail.com>
2020-04-09 08:02:58 -05:00
Stat(const std::string& file, bool link = false)
: _path(file)
, _res(clearStat() | (link ? lstat(file.c_str(), &_sb) : stat(file.c_str(), &_sb)))
wsd: faster jail setup via bind-mount loolmount now works and supports mounting and unmounting, plus numerous improvements, refactoring, logging, etc.. When enabled, binding improves the jail setup time by anywhere from 2x to orders of magnitude (in docker, f.e.). A new config entry mount_jail_tree controls whether mounting is used or the old method of linking/copying of jail contents. It is set to true by default and falls back to linking/copying. A test mount is done when the setting is enabled, and if mounting fails, it's disabled to avoid noise. Temporarily disabled for unit-tests until we can cleanup lingering mounts after Jenkins aborts our build job. In a future patch we will have mount/jail cleanup as part of make. The network/system files in /etc that need frequent refreshing are now updated in systemplate to make their most recent version available in the jails. These files can change during the course of loolwsd lifetime, and are unlikely to be updated in systemplate after installation at all. We link to them in the systemplate/etc directory, and if that fails, we copy them before forking each kit instance to have the latest. This reworks the approach used to bind-mount the jails and the templates such that the total is now down to only three mounts: systemplate, lo, tmp. As now systemplate and lotemplate are shared, they must be mounted as readonly, this means that user/ must now be moved into tmp/user/ which is writable. The mount-points must be recursive, because we mount lo/ within the mount-point of systemplate (which is the root of the jail). But because we (re)bind recursively, and because both systemplate and lotemplate are mounted for each jails, we need to make them unbindable, so they wouldn't multiply the mount-points for each jails (an explosive growth!) Contrarywise, we don't want the mount-points to be shared, because we don't expect to add/remove mounts after a jail is created. The random temp directory is now created and set correctly, plus many logging and other improvements. Change-Id: Iae3fda5e876cf47d2cae6669a87b5b826a8748df Reviewed-on: https://gerrit.libreoffice.org/c/online/+/92829 Tested-by: Jenkins Tested-by: Jenkins CollaboraOffice <jenkinscollaboraoffice@gmail.com> Reviewed-by: Ashod Nakashian <ashnakash@gmail.com>
2020-04-09 08:02:58 -05:00
, _errno(errno)
{
}
bool good() const { return _res == 0; }
bool bad() const { return !good(); }
bool erno() const { return _errno; }
const struct ::stat& sb() const { return _sb; }
const std::string path() const { return _path; }
bool isDirectory() const { return S_ISDIR(_sb.st_mode); }
bool isFile() const { return S_ISREG(_sb.st_mode); }
bool isLink() const { return S_ISLNK(_sb.st_mode); }
std::size_t hardLinkCount() const { return _sb.st_nlink; }
ino_t inodeNumber() const { return _sb.st_ino; }
wsd: faster jail setup via bind-mount loolmount now works and supports mounting and unmounting, plus numerous improvements, refactoring, logging, etc.. When enabled, binding improves the jail setup time by anywhere from 2x to orders of magnitude (in docker, f.e.). A new config entry mount_jail_tree controls whether mounting is used or the old method of linking/copying of jail contents. It is set to true by default and falls back to linking/copying. A test mount is done when the setting is enabled, and if mounting fails, it's disabled to avoid noise. Temporarily disabled for unit-tests until we can cleanup lingering mounts after Jenkins aborts our build job. In a future patch we will have mount/jail cleanup as part of make. The network/system files in /etc that need frequent refreshing are now updated in systemplate to make their most recent version available in the jails. These files can change during the course of loolwsd lifetime, and are unlikely to be updated in systemplate after installation at all. We link to them in the systemplate/etc directory, and if that fails, we copy them before forking each kit instance to have the latest. This reworks the approach used to bind-mount the jails and the templates such that the total is now down to only three mounts: systemplate, lo, tmp. As now systemplate and lotemplate are shared, they must be mounted as readonly, this means that user/ must now be moved into tmp/user/ which is writable. The mount-points must be recursive, because we mount lo/ within the mount-point of systemplate (which is the root of the jail). But because we (re)bind recursively, and because both systemplate and lotemplate are mounted for each jails, we need to make them unbindable, so they wouldn't multiply the mount-points for each jails (an explosive growth!) Contrarywise, we don't want the mount-points to be shared, because we don't expect to add/remove mounts after a jail is created. The random temp directory is now created and set correctly, plus many logging and other improvements. Change-Id: Iae3fda5e876cf47d2cae6669a87b5b826a8748df Reviewed-on: https://gerrit.libreoffice.org/c/online/+/92829 Tested-by: Jenkins Tested-by: Jenkins CollaboraOffice <jenkinscollaboraoffice@gmail.com> Reviewed-by: Ashod Nakashian <ashnakash@gmail.com>
2020-04-09 08:02:58 -05:00
/// Returns the filesize in bytes.
std::size_t size() const { return _sb.st_size; }
/// Returns the modified unix-time as timespec since epoch with
/// nanosecond precision, if/when the filesystem supports it.
timespec modifiedTime() const
{
#ifdef IOS
return _sb.st_mtimespec;
#else
return _sb.st_mtim;
#endif
}
/// Returns the modified unix-time in microseconds since epoch.
int64_t modifiedTimeUs() const
{
// cast to make sure the calculation happens with enough bits
return (static_cast<int64_t>(modifiedTime().tv_sec) * 1000 * 1000) + (modifiedTime().tv_nsec / 1000);
}
/// Returns the modified unix-time in milliseconds since epoch.
std::size_t modifiedTimeMs() const
{
return (modifiedTime().tv_sec * 1000) + (modifiedTime().tv_nsec / 1000000);
}
/// Returns the modified unix-time as time_point (in microsecond precision, if available).
/// The units is system-dependent, but it's 100% safe as time_point does the conversion
/// to whatever we request, remembering the original units.
std::chrono::system_clock::time_point modifiedTimepoint() const
{
// The time in microseconds.
const std::chrono::microseconds us{ modifiedTimeUs() };
// Convert to the precision of the system_clock::time_point,
// which can be different from microseconds.
return std::chrono::system_clock::time_point(
std::chrono::duration_cast<std::chrono::system_clock::duration>(us));
}
/// Returns true iff the path exists, regardless of access permission.
wsd: faster jail setup via bind-mount loolmount now works and supports mounting and unmounting, plus numerous improvements, refactoring, logging, etc.. When enabled, binding improves the jail setup time by anywhere from 2x to orders of magnitude (in docker, f.e.). A new config entry mount_jail_tree controls whether mounting is used or the old method of linking/copying of jail contents. It is set to true by default and falls back to linking/copying. A test mount is done when the setting is enabled, and if mounting fails, it's disabled to avoid noise. Temporarily disabled for unit-tests until we can cleanup lingering mounts after Jenkins aborts our build job. In a future patch we will have mount/jail cleanup as part of make. The network/system files in /etc that need frequent refreshing are now updated in systemplate to make their most recent version available in the jails. These files can change during the course of loolwsd lifetime, and are unlikely to be updated in systemplate after installation at all. We link to them in the systemplate/etc directory, and if that fails, we copy them before forking each kit instance to have the latest. This reworks the approach used to bind-mount the jails and the templates such that the total is now down to only three mounts: systemplate, lo, tmp. As now systemplate and lotemplate are shared, they must be mounted as readonly, this means that user/ must now be moved into tmp/user/ which is writable. The mount-points must be recursive, because we mount lo/ within the mount-point of systemplate (which is the root of the jail). But because we (re)bind recursively, and because both systemplate and lotemplate are mounted for each jails, we need to make them unbindable, so they wouldn't multiply the mount-points for each jails (an explosive growth!) Contrarywise, we don't want the mount-points to be shared, because we don't expect to add/remove mounts after a jail is created. The random temp directory is now created and set correctly, plus many logging and other improvements. Change-Id: Iae3fda5e876cf47d2cae6669a87b5b826a8748df Reviewed-on: https://gerrit.libreoffice.org/c/online/+/92829 Tested-by: Jenkins Tested-by: Jenkins CollaboraOffice <jenkinscollaboraoffice@gmail.com> Reviewed-by: Ashod Nakashian <ashnakash@gmail.com>
2020-04-09 08:02:58 -05:00
bool exists() const { return good() || (_errno != ENOENT && _errno != ENOTDIR); }
/// Returns true if both files exist and have
/// the same size and same contents.
bool isIdenticalTo(const Stat& other) const
{
// No need to check whether they are linked or not,
// since if they are, the following check will match,
// and if they aren't, we still need to rely on the following.
// Finally, compare the contents, to avoid costly copying if we fail to update.
return (exists() && other.exists() && !isDirectory() && !other.isDirectory() &&
size() == other.size() && compareFileContents(_path, other._path));
}
/// Returns true if both files exist and have
/// the same size and modified timestamp.
bool isUpToDate(const Stat& other) const
{
// No need to check whether they are linked or not,
// since if they are, the following check will match,
// and if they aren't, we still need to rely on the following.
// Finally, compare the contents, to avoid costly copying if we fail to update.
if (isIdenticalTo(other))
{
return true;
}
// Clearly, no match. Log something informative.
LOG_DBG("File contents mismatch: ["
<< _path << "] " << (exists() ? "exists" : "missing") << ", " << size()
<< " bytes, modified at " << modifiedTime().tv_sec << " =/= [" << other._path
<< "]: " << (other.exists() ? "exists" : "missing") << ", " << other.size()
<< " bytes, modified at " << other.modifiedTime().tv_sec);
return false;
}
wsd: faster jail setup via bind-mount loolmount now works and supports mounting and unmounting, plus numerous improvements, refactoring, logging, etc.. When enabled, binding improves the jail setup time by anywhere from 2x to orders of magnitude (in docker, f.e.). A new config entry mount_jail_tree controls whether mounting is used or the old method of linking/copying of jail contents. It is set to true by default and falls back to linking/copying. A test mount is done when the setting is enabled, and if mounting fails, it's disabled to avoid noise. Temporarily disabled for unit-tests until we can cleanup lingering mounts after Jenkins aborts our build job. In a future patch we will have mount/jail cleanup as part of make. The network/system files in /etc that need frequent refreshing are now updated in systemplate to make their most recent version available in the jails. These files can change during the course of loolwsd lifetime, and are unlikely to be updated in systemplate after installation at all. We link to them in the systemplate/etc directory, and if that fails, we copy them before forking each kit instance to have the latest. This reworks the approach used to bind-mount the jails and the templates such that the total is now down to only three mounts: systemplate, lo, tmp. As now systemplate and lotemplate are shared, they must be mounted as readonly, this means that user/ must now be moved into tmp/user/ which is writable. The mount-points must be recursive, because we mount lo/ within the mount-point of systemplate (which is the root of the jail). But because we (re)bind recursively, and because both systemplate and lotemplate are mounted for each jails, we need to make them unbindable, so they wouldn't multiply the mount-points for each jails (an explosive growth!) Contrarywise, we don't want the mount-points to be shared, because we don't expect to add/remove mounts after a jail is created. The random temp directory is now created and set correctly, plus many logging and other improvements. Change-Id: Iae3fda5e876cf47d2cae6669a87b5b826a8748df Reviewed-on: https://gerrit.libreoffice.org/c/online/+/92829 Tested-by: Jenkins Tested-by: Jenkins CollaboraOffice <jenkinscollaboraoffice@gmail.com> Reviewed-by: Ashod Nakashian <ashnakash@gmail.com>
2020-04-09 08:02:58 -05:00
private:
const std::string _path;
struct ::stat _sb;
const int _res;
const int _errno;
};
} // end namespace FileUtil
/* vim:set shiftwidth=4 softtabstop=4 expandtab: */