In the instdir/program of an --enable-mergelibs build of LibreOffice
there is also a dummy libsofficeapp.so. Obviously we don't want to
even try that. So look for libmergedlo.so first.
I easily got the thing into a state where it kept forking new
processes without limit even if no client was doing anything. The log
output contained lines like:
wsd-09526-04 00:00:48.302029 [client_ws_001b ] MasterToBroker: spawn -1
brk-09528-00 00:00:48.279945 [loolbroker ] Broker command: [spawn -1].
brk-09528-00 00:00:48.279964 [loolbroker ] Spawning -1 childs per request.
brk-09528-00 00:00:48.279974 [loolbroker ] Creating -1 new child.
This fixes that, but there are still several issues remaining.
So just use DEFAULT_CLIENT_PORT_NUMBER and don't confusingly include
ChildProcessSession.hpp in the test program sources just to define a
static (file-local) ClientPortNumber variable that never gets set to
anything except its initialized value. ChildProcessSession is specific
to the internals of loolwsd and loolbroker and should not be used in
the test programs.
Preiniting LibreOfficeKit and forking kit processes (instead of
spawning) has worked fine for a while, and has been the default way
this works.
No 'loolkit' program gets built any more.
Convert-to is broken after re-designing Broker
and simplifying it. Temporarily disabling it
to help `make check` pass until it gets fixed.
Change-Id: Id49f86d8a1a25367233a09b865630ed3a210a4fd
Reviewed-on: https://gerrit.libreoffice.org/23793
Reviewed-by: Ashod Nakashian <ashnakash@gmail.com>
Tested-by: Ashod Nakashian <ashnakash@gmail.com>
Broker is now just a simple spawn-machine.
It only gets requests from WSD to spawn
new kit instances, which it doesn't even
track.
Once a kit instance is initialized, it
connects to WSD with a control WS.
From there on, it's up to WSD to manage
the kit process.
Also was removed the benchmark logic
since it can no longer function.
Change-Id: I1bf56bc6416c9eadafba637276bbb8b3107e5727
Reviewed-on: https://gerrit.libreoffice.org/23790
Reviewed-by: Ashod Nakashian <ashnakash@gmail.com>
Tested-by: Ashod Nakashian <ashnakash@gmail.com>
WSD now communicates on a WebSocket directly
with kit processes. ChildProcess encapsulates
kit processes and the control WS, which itself
is owned by DocumentBroker.
Change-Id: Ica209aaa07974739b8e51a14e11325d084e193f6
Reviewed-on: https://gerrit.libreoffice.org/23789
Reviewed-by: Ashod Nakashian <ashnakash@gmail.com>
Tested-by: Ashod Nakashian <ashnakash@gmail.com>
In face of exceptions, the lock was not released
and the condition variable was not signalled,
thereby causing all subsequent views on the
same document to fail loading.
Change-Id: I18d3cefcc74a158facefe1e74a9c802ee048b014
Reviewed-on: https://gerrit.libreoffice.org/23785
Reviewed-by: Ashod Nakashian <ashnakash@gmail.com>
Tested-by: Ashod Nakashian <ashnakash@gmail.com>
of the 10k files still linked into the jail; 5700 are from usr/
so bind mount just that directory, also set noatime, ro, and
some other helpful looking options.
Change-Id: I28d2d5cbbdf33fb57ea0f0c0915cb267603ee16d
Reviewed-on: https://gerrit.libreoffice.org/23777
Reviewed-by: Ashod Nakashian <ashnakash@gmail.com>
Tested-by: Ashod Nakashian <ashnakash@gmail.com>
Create directories top-down and not bottom up for more efficiency too.
Skip the sdk (if we have it) - ~20k files, and misc. other
pieces that we don't need; still more to go there.
Change-Id: Iccd9ebac495fba414d268b339ef82a161d98a9ca
Reviewed-on: https://gerrit.libreoffice.org/23770
Reviewed-by: Ashod Nakashian <ashnakash@gmail.com>
Tested-by: Ashod Nakashian <ashnakash@gmail.com>
This reverts commit 97c8f35ddf.
Since the Broker design has been extremely simplified,
all communication between Broker <-> Kit are gone.
Only a pipe between WSD and Broker remain.
Temporarily reverting this to apply the Broker redesign,
after which this patch can be reviewed and merged.
This will be easier than trying to merge the redesigned
Broker on top of this.
Change-Id: Ia901fad604008654c01841df62e88918adad45e1
Reviewed-on: https://gerrit.libreoffice.org/23769
Reviewed-by: Ashod Nakashian <ashnakash@gmail.com>
Tested-by: Ashod Nakashian <ashnakash@gmail.com>
There is nothing that says a client has even requested any tiles, so
there might be none to persist. Don't let an exception thrown by the
DirectoryIterator propagate upwards and cause potential
issues.
Noticed the issue when testing using the 'connect' test program,
giving it input that did not request any tiles.
Had to add a shared pointer to the BasicTileQueue for the session to
the MasterProcessSession object, and restructure the coe a a bit to
allocate BasicTileQueue objects dynamically. Possibly just passing a
reference to a BasicTileQueue in the stack would have worked, but why
risk it?
The actual logic when to do auto / idle save is not quite right still,
did not change that in this commit.
Time increment is handled on client-side, server only sends the
elapsed time during first page-load.
Change-Id: I73e98fd95ca9f391b625a8dcfc7e3490878c6a40
... which can be taken only one at a time. Others can only view,
not edit. When a session with edit lock exits, the edit lock is
handed over to the next alive session.
Change-Id: I712a4e70369f1d07c1d83af416a0f5c288b05c7d
Don't embed newlines in "lines" written to the log. When logging stuff
read from or written to the fifos, translate newlines to " / " for
clarity.
(If we would want complete, exact verbose logging, we should be really
pedantic and log all non-printable bytes in hex anyway, etc, so
displaying newlines as space-separated slashes should be OK. It isn't
as if there would be totally arbitary data passed through the fifos
anyway.)
We will switch to reading all these values as soon as wip
configuration file object is accessible globally.
Change-Id: I16eff339349683050be1985eefabc34854cccba3
We act as a client when we communicate with owncloud server.
For time being, just accept whatever certificates owncloud server
gives us. We might want to get more strict here in future.
Change-Id: I4813d19412b66ecf57d6cdef9c3ac94fbbaa521f
Spare child processes are now in a separate
container. A FIFO that gives older instances
priority to avoid using instances too young
to have initialized fully.
In addition, spare instances are now
proactively spawned such that there
is at least a minimum number of
spares at any given time.
Change-Id: Ibdb206d88473adb306c274f4af39798c784258a0
Reviewed-on: https://gerrit.libreoffice.org/23647
Reviewed-by: Ashod Nakashian <ashnakash@gmail.com>
Tested-by: Ashod Nakashian <ashnakash@gmail.com>
All messages now pass through the queue.
This resolves a race between single-line
messages and multi-line ones.
Previously, single-line messages were
processed on the queue (on a background
thread) while multi-line ones were handled
immediatly. This resulted in order-inversion
due to a race between the queue thread and the
next multi-line message, which caused stability
issues every so often.
Change-Id: Ia220791d1d75c4f3e3e0965dd0c6f81bae63a296
Reviewed-on: https://gerrit.libreoffice.org/23583
Reviewed-by: Ashod Nakashian <ashnakash@gmail.com>
Tested-by: Ashod Nakashian <ashnakash@gmail.com>
It is not a good idea to have the same string somewhat arbitrarily
both as a static const members of the LOOLWSD class and then as a
file-local static const in another file. Or defined as a separate
local const static in each compilation unit that includes
Common.hpp. Use constexpr instead, in Common.hpp.
This is C++, not Java. Or is there a school of thought for C++ style
that says one should avoid plain "C-style" file-local static
variables, and instead put everything always in a class, even as
static members? Do we want to follow that?
Dumping of registered tests in preparation
to allow for selective test running.
Change-Id: I83af1c9be211804f757918c326602a0b53815704
Reviewed-on: https://gerrit.libreoffice.org/23531
Reviewed-by: Ashod Nakashian <ashnakash@gmail.com>
Tested-by: Ashod Nakashian <ashnakash@gmail.com>
The plan was to keep the document's open and close history right
from the moment the server started up to the end, so we would
only expire() the document. For now, just destroy the document
as soon as user session is closed.
Change-Id: Id8f5b550a2b9bda217f7f8f8f1f82f85b1aa3502
CollaboraCloudSuiteCA_ca-chain.cert.pem is the CA chain that should
be trusted. Move the file to directory of trusted certs, such as
/usr/share/pki/trust.
Configuration XML is added with SSL as sample use-case.
A 'desc' attribute can be used to describe the fields,
and another 'type' to help define the corresponding data
type in the code.
Since Poco allows accessing group nodes (that have the
same name) by index, order can be preserved.
SSL initialization refactored and cert/key file
paths moved to the config file.
Change-Id: I259826a19697bd851587bebcc4f0cd233ab6848b
Reviewed-on: https://gerrit.libreoffice.org/23464
Reviewed-by: Ashod Nakashian <ashnakash@gmail.com>
Tested-by: Ashod Nakashian <ashnakash@gmail.com>
When the Document lock cannot be taken
purging doesn't block (which would block
the kit-broker pipe). Instead, purging
is done only when the lock is taken,
otherwise we try again later.
Change-Id: Id201f1c67803d9b1e765e8c55f85206795fe53c0
Reviewed-on: https://gerrit.libreoffice.org/23448
Reviewed-by: Ashod Nakashian <ashnakash@gmail.com>
Tested-by: Ashod Nakashian <ashnakash@gmail.com>
Tests can modify the test documents they use.
Currently there is data-loss protection that
saves an open doc if connection is lost with
the client. For tests this means modification
are saved when a connection is terminated
ungracefully and this both adds noise
to the git checkout and makes subsequent
tests fail.
This patch makes temp copies of the original
doc before a test is run and deletes them
afterwards.
Change-Id: I1dd6ff2b839701e85c8bd502ba75170c01fa106e
Reviewed-on: https://gerrit.libreoffice.org/23447
Reviewed-by: Ashod Nakashian <ashnakash@gmail.com>
Tested-by: Ashod Nakashian <ashnakash@gmail.com>
Automake already have rules, make tags and make ctags, for emacs
and vim style tags respectively. We can pass our custom flags to
them using the AM_ macros.
https://www.gnu.org/software/automake/manual/html_node/Tags.html
This commit also adds support to create emacs style tags using
automake's pre-generated tag rules.
Change-Id: I4f6ed997fab6964b3c1f6637e3fd0365f8d4c8b8
Reviewed-on: https://gerrit.libreoffice.org/23442
Reviewed-by: pranavk <pranavk@collabora.com>
Tested-by: pranavk <pranavk@collabora.com>
Poco/SharedPtr.h: In instantiation of ‘static void Poco::ReleasePolicy<C>::release(C*) [with C = Poco::Net::PrivateKeyPassphraseHandler]’:
Poco/SharedPtr.h:130:14: required from ‘Poco::SharedPtr<C, RC, RP>::SharedPtr(C*) [with C = Poco::Net::PrivateKeyPassphraseHandler; RC = Poco::ReferenceCounter; RP = Poco::ReleasePolicy<Poco::Net::PrivateKeyPassphraseHandler>]’
Poco/SharedPtr.h:70:3: warning: possible problem detected in invocation of delete operator: [-Wdelete-incomplete]
delete pObj;
Poco/SharedPtr.h:66:25: warning: ‘pObj’ has incomplete type
static void release(C* pObj)
Poco/Net/PrivateKeyFactory.h:30:7: note: forward declaration of ‘class Poco::Net::PrivateKeyPassphraseHandler’
class PrivateKeyPassphraseHandler;
Just a skeleton, actual saving not yet implemented. Also, not sure
the logic when to trigger save is as intended.
Note that no separate timer classes or objects are used. The existing
watpid/sleep loop that wakes up once every two seconds currently is
used. If that loop is re-factored to be less silly, the auto/idle save code
must be implemented differently.
When no child process has died, I don't see the point in calling
waitpid() eleven times in quick succession (with WNOHANG), doing
nothing else, and then sleeping for a bit. Let's call waitpid() just
once, and sleep only if the return value indicates that no child
process has died.
Use same port (9989) for all client connections. This includes
admin panel, static file serving and normal client websocket
connections.
Change-Id: Idcfd7dd8925523c36e884717c41a3b6a827f6ff3
File server serves the admin html file after successfull
authentication, and sets the cookie in client which would be sent
for all subsequent connections by client to connect to admin websocket.
Change-Id: I0ee3bbfca7eefc428020d29612374410556b1e27
Only purpose, at the moment, is to create
Poco::HTTPRequestHandler which would be passed on the serving
handling the static file requests.
Change-Id: I97c3fc0c73da077d3efee919416098b880c9c2ad
... and use SSL for client connections. Also fix our test suite
to use HTTPS now.
Change-Id: Id396a7c2d1830da8d3b0ce446522403363ac17c1
Reviewed-on: https://gerrit.libreoffice.org/23395
Reviewed-by: Tor Lillqvist <tml@collabora.com>
Tested-by: Tor Lillqvist <tml@collabora.com>
File server serves the admin html file after successfull
authentication, and sets the cookie in client which would be sent
for all subsequent connections by client to connect to admin websocket.
Change-Id: I0ee3bbfca7eefc428020d29612374410556b1e27
Only purpose, at the moment, is to create
Poco::HTTPRequestHandler which would be passed on the serving
handling the static file requests.
Change-Id: I97c3fc0c73da077d3efee919416098b880c9c2ad
... and use SSL for client connections.
Change-Id: Id396a7c2d1830da8d3b0ce446522403363ac17c1
Reviewed-on: https://gerrit.libreoffice.org/23395
Reviewed-by: Tor Lillqvist <tml@collabora.com>
Tested-by: Tor Lillqvist <tml@collabora.com>
To make room for other classes deriving from AuthBase that do not
require any authorization code, such as JWT authentication.
Change-Id: I69a35dd6f775badd7377949df2ca326c910d4021
This is to distinguish the deliberate close of connection, and timeout,
connection drop, or forced close.
When the last session is closed non-deliberately, force a save so that the
edits are not lost.
Ideally, we will have a randomized path for the jails.
Unfortunately, this will make it harder to cleanup
after an ungraceful exit of a child, including recovery
of docs etc.
Having a PID for the jailId makes this issue easier by
implicitly implying the jail path for a given child.
To prevent security leaks, we should at least randomize
the doc directory within the jail, as such:
/chroot/<pid>/user/docs/<rand>/
For now we use jailId=pid=rand.
Change-Id: I948fba0aaef725c9c059780df0a184a86569d898
Reviewed-on: https://gerrit.libreoffice.org/23223
Reviewed-by: Ashod Nakashian <ashnakash@gmail.com>
Tested-by: Ashod Nakashian <ashnakash@gmail.com>
Renamed DocumentStoreManager to DocumentBroker and
restructured the handshake process.
Currently, at first client connection to a given doc
a DocumentBroker is created to serve as the clearing house
of all client-side activities on the document.
Prime goals is loading and saving of the document, but
also to guarantee race-free management of the doc.
Each doc has a unique DocKey based on the URL (the path,
without queries). This DocKey is used as key into a map
of all DocumentBrokers. The latter is shared among
MasterProcessSession instances.
Change-Id: I569f2d235676e88ddc690147f3cb89faa60388c2
Reviewed-on: https://gerrit.libreoffice.org/23216
Reviewed-by: Ashod Nakashian <ashnakash@gmail.com>
Tested-by: Ashod Nakashian <ashnakash@gmail.com>
The current format is more flexible and standard.
/loolwsd/child?sessionId=xxx&jailId=yyy
The sessionId is the client-specific connection ID (which
is originally passed to the child via the Broker.
The jailId is the PID of the child.
Change-Id: I69c88e84114f9678addf795896ca2da15ca1221b
Reviewed-on: https://gerrit.libreoffice.org/23211
Reviewed-by: Ashod Nakashian <ashnakash@gmail.com>
Tested-by: Ashod Nakashian <ashnakash@gmail.com>
By default, queries and stores the total memory usage in
AdminModel every 5 seconds, and caches the last 100 such values.
Both cache size and interval can be changed by simple commands
from the clients.
Change-Id: I86cf8228d0129dc8aab0a03856c12dfeb240b169
Well, it was already there.
e7f75a3e56 was though correct
introduced it again. Remove duplicate such call.
Change-Id: Iddacbee53aaec58340db489e7830af75354aee34
It is desirable to have colored logs when running loolwsd
in a terminal, but not redirecting its output to a file.
Outputting to a terminal is now detected and colored logs
are automatically enabled.
To force colored logs in files as well, define
LOOL_LOGCOLOR in the environ. The output color codes
can then be processed using, f.e., `less -r`.
Change-Id: I09fbee4441f210d814ac5ad23dd99d1c33b560b7
Reviewed-on: https://gerrit.libreoffice.org/23080
Reviewed-by: Ashod Nakashian <ashnakash@gmail.com>
Tested-by: Ashod Nakashian <ashnakash@gmail.com>
... for which they want to be notified.
Needed to add some missing header files to Storage and Auth class
because of conflict raised when LOOLWSD.hpp is included in
Admin.cpp
Change-Id: Ia1c8ed82f8cd979eaf93267ae5dfa347acf895f4
Further changes/refactoring to make it possible:
* Add broker pid to Admin class
* Move getMemoryUsage for process to Util
* Change variable name to accurately reflect *active* items
_nViews -> _nActiveViews, etc.
Change-Id: I4c9206c49ab829b73ebfe226874bfbbcc8f95342
Reviewed-on: https://gerrit.libreoffice.org/22989
Reviewed-by: Tor Lillqvist <tml@collabora.com>
Tested-by: Tor Lillqvist <tml@collabora.com>
Otherwise the process could terminate while admin sessions are still
being served.
Change-Id: Id91d0989e264e8294827e8e2ad8dd2d44b6006ec
Reviewed-on: https://gerrit.libreoffice.org/23021
Reviewed-by: Tor Lillqvist <tml@collabora.com>
Tested-by: Tor Lillqvist <tml@collabora.com>
Much simpler.
Also, don't duplicate the code informing that LD_BIND_NOW or
LOK_VIEW_CALLBACK are not set. Not that I understand why we need to
inform about that? If the "normal" thing should be that they are set,
why don't make it so by default then?
I occasionally saw a deadlock when running 'make check' where one
thread holds the ChildProcessSession::Mutex and wants the _mutex for a
Document, while another thread holds that _mutex and wants the
Mutex. In particular, it is the Document::onUnload() that wants the
_mutex. So avoid the deadlock by having Document::onUnload() first
take the ChildProcessSession::Mutex.
The callbacks from documentLoad() are made in the same thread.
Sure, as such it is not a good thing to use recursive mutexes. If we
switch back to non-recursive mutexes, we will have to stop taking the
lock in callbacks from documentLoad(), i.e. make sure we know those
functions aren't used elsewhere, in places where a lock would be
needed. Or something.
If a client session closes just after sending a load message to load a
document, and another session then fairly immediately connects and
sends a load message for the same document, the latter session gets
handled by the same kit process. Also, the same Document object is
apparently used. In that kit process, the first documentLoad() might
easily still be in progress. The handler for the new session still
calls onLoad(), too, and as the first onLoad() had dropped the lock
for the duration of the documentLoad() call, the new onLoad can take
the lock and call documentLoad(), too, while the first documentLoad()
call in the other thread still is in progress. This leads to
interesting problems.
It is loolwsd that spawns loolbroker so we control what arguments it
gets, so no need to give verbose errors if our own code is
inconsistent. That is what assert() is for.
Admin web sessions are added as subscribers to AdminModel. Live
notification fill up the AdminModel, and notifies to
subscribers, if present any. AdminModel can also be queried to
fetch any previous data since the start of the server including
expired documents/views with timestamps for analysis.
There is lot of stuff that can be added in future. This commit
just lays the foundation of appropriate classes.
Change-Id: Ifcf6c2896ef46b33935802e79cd28240fd4f980e
Reviewed-on: https://gerrit.libreoffice.org/22869
Reviewed-by: Tor Lillqvist <tml@collabora.com>
Tested-by: Tor Lillqvist <tml@collabora.com>
As a test, add command to fetch documents from AdminModel.
Change-Id: I3cb7097ba7dde049f3b2478fe7b6b6c309da1d92
Reviewed-on: https://gerrit.libreoffice.org/22781
Reviewed-by: Tor Lillqvist <tml@collabora.com>
Tested-by: Tor Lillqvist <tml@collabora.com>
Turns out the callbacks from documentLoad() are in general done on
another thread, not necessarily the same one that did the
documentLoad() call. Not sure why it happened to be the same thread
for me, but oh well. Need to fix the problem described in
3671abf89b in some other way then.
This reverts commit 2fab757462.
The callbacks from documentLoad() are made in the same thread.
Sure, as such it is not a good thing to use recursive mutexes. If we
switch back to non-recursive mutexes, we will have to stop taking the
lock in callbacks from documentLoad(), i.e. make sure we know those
functions aren't used elsewhere, in places where a lock would be
needed. Or something.
... and Admin and AdminModel containing all the required data
that we need to expose to Admin panel.
Admin processor will keep listening to any data on this
notification pipe and update AdminModel accordingly.
Change-Id: I0dd6f07ae60158733c34d17f53a35def70600513
Reviewed-on: https://gerrit.libreoffice.org/22780
Reviewed-by: Ashod Nakashian <ashnakash@gmail.com>
Tested-by: Ashod Nakashian <ashnakash@gmail.com>
It does quite a lot of work that can produce very many lines of log,
so it is good to be able to quickly find when that is done when
perusing a log file.
Not needed after all. It was a red herring. The device files work fine
even if not owned by root:root and with mode 664. The actual problem
was that I used a file system mounted with nodev when testing loolwsd.
This reverts commit 509314d559
That is probably what was the intent. As originally written, in case
the function encountered partial writers, and had to do several
write() calls, only the number of bytes written by the last one was
returned.
Luckily the actual return value of writeFIFO() is not used
anywhere. It is just tested for being negative.
Still there is the problem that if at first one or several write()
calls succeed but don't write the whole buffer, and then a write()
fails, the caller has no way to know that the buffer has been
partially written. But that is hopefully highly theoretical and there
is no sane way to handle such a situation anyway.
The getChildStatus() and getSignalStatus() functions returned the
latter, still the returned values were compared against
Poco::Util::Application::EXIT_OK. (Sure, both Poco's
Application::EXIT_SUCCESS and stdlib.h's EXIT_OK are zero, but that
doesn't mean one should mix them up.)
Also add two comments pondering the meaning of the code.
We are not interested in the variable being assigned an incremented
value. Its value is not used any more. We are just interested in the
value of the variable plus one. Using pre-increment gives the wrong
impression.
Sure, this is nit-picking.
So don't give it any then.
Remove the --uid option and related attempts to handle running loolwsd
under sudo, to be able to debug it. Now with loolwsd not having
capabilities, it should work fine to just run it under a debugger
normally. (For the loolbroker and loolkit processes, attaching to an
already started process is the way to debug.)
