libreoffice-online/wasm
Tor Lillqvist fc0b538a19 Increase PTHREAD_POOL_SIZE
Just 4 is very likely not nearly enough. I think I did see some error
that could be related to running out of threads. Does
PTHREAD_POOL_SIZE limit how many threads can exist simultaneously?

Signed-off-by: Tor Lillqvist <tml@collabora.com>
Change-Id: I5da637b5660e89655f7049b6754f70c74dff6da2
2023-01-15 19:46:37 +01:00
..
Makefile.am Increase PTHREAD_POOL_SIZE 2023-01-15 19:46:37 +01:00
poco-1.12.4-emscripten.patch
poco-devel-emscripten.patch
README WASM add a note about memory required for linking 2023-01-10 22:02:22 +02:00
wasmapp.cpp Trying to init LOKit cause mysterious runtime error 2023-01-15 19:46:37 +01:00
wasmapp.hpp Trying to init LOKit cause mysterious runtime error 2023-01-15 19:46:37 +01:00

= WASM with emscripten =

Building for WASM with emscripten is still a bit immature and many projects
need patching to work.

Note that there are wrapper tools like "emconfigure", "emmake", "emcmake" that
tend to set up environment variables like CC properly, although some project's
build system unfortunately override that.

Linking WASM executable uses a lot of memory; without optimisations or with -O1
in LDFLAGS, /usr/bin/time prints "1660068maxresident" which should work
anywhere, but with -O2 in LDFLAGS (which is the default) some LTO happens that
uses about 8x as much RAM, "12261016maxresident" or ~12 GiB... so append
CXXFLAGS='-g -O1' or similar to the configure invocation if out of memory.

For convenience it's recommended to build with a docker container that has the
required build tools.

    podman pull public.ecr.aws/allotropia/libo-builders/wasm

== Build LO core ==

First you need to build LibreOffice core with emscripten.
Currently best to use "feature/wasm" branch.

These examples assume that the git repo is in $HOME/lo/feature_wasm, the build
directory in /data/lo/build_feature_wasm and the tarball download cache in
$HOME/lo/ext_sources - adjust these to your preferences.

    podman run -v $HOME/lo/feature_wasm:$HOME/lo/feature_wasm:ro -v /data/lo/build_feature_wasm:/data/lo/build_feature_wasm -v $HOME/lo/ext_sources:/ext_sources:rw --security-opt=label=disable -ti public.ecr.aws/allotropia/libo-builders/wasm /bin/bash -c "source /home/builder/emsdk/emsdk_env.sh && cd /data/lo/build_feature_wasm/ && $HOME/lo/feature_wasm/autogen.sh --with-external-tar=/ext_sources --with-distro=LibreOfficeWASM32"

    podman run -v $HOME/lo/feature_wasm:$HOME/lo/feature_wasm:ro -v /data/lo/build_feature_wasm:/data/lo/build_feature_wasm -v $HOME/lo/ext_sources:/ext_sources:rw --security-opt=label=disable -ti public.ecr.aws/allotropia/libo-builders/wasm /bin/bash -c "source /home/builder/emsdk/emsdk_env.sh && cd /data/lo/build_feature_wasm/ && make -rj8"

== Build Online using dependencies from container ==

The container contains prebuilt libzstd and POCO for convenience.

Note that here the LO core (and builddir, if separate) must be mapped to the
same directory in the container as it was when building core, due to files
containing absolute paths.

    podman run -v $HOME/lo/feature_wasm:$HOME/lo/feature_wasm:ro -v /data/lo/build_feature_wasm:/data/lo/build_feature_wasm:ro -v $HOME/lo/online:$HOME/lo/online:rw -v /data/lo/build_online_wasm:/data/lo/build_online_wasm:rw --security-opt=label=disable -ti public.ecr.aws/allotropia/libo-builders/wasm /bin/bash -c "source /home/builder/emsdk/emsdk_env.sh && cd /data/lo/build_online_wasm/ && $HOME/lo/online/autogen.sh && emconfigure $HOME/lo/online/configure --with-lokit-path=$HOME/lo/feature_wasm/include --with-lo-path=/data/lo/build_feature_wasm/instdir --with-lo-builddir=/data/lo/build_feature_wasm --with-zstd-includes=/usr/local/include --with-zstd-libs=/usr/local/lib --with-poco-includes=/usr/local/include --with-poco-libs=/usr/local/lib --host=wasm32-local-emscripten"
    podman run -v $HOME/lo/feature_wasm:$HOME/lo/feature_wasm:ro -v /data/lo/build_feature_wasm:/data/lo/build_feature_wasm:ro -v $HOME/lo/online:$HOME/lo/online:rw -v /data/lo/build_online_wasm:/data/lo/build_online_wasm:rw --security-opt=label=disable -ti public.ecr.aws/allotropia/libo-builders/wasm /bin/bash -c "source /home/builder/emsdk/emsdk_env.sh && cd /data/lo/build_online_wasm/ && emmake make -rj8"

== Build Online dependencies manually ==

Then build libzstd:
* with assembly code disabled
* using the Makefile (didn't try its other build systems)

    tar -xzvf ~/Downloads/zstd-1.5.2.tar.gz
    cd zstd-1.5.2/
    podman run -v /data/lo/zstd-1.5.2:/data/lo/zstd-1.5.2:rw --security-opt=label=disable -ti public.ecr.aws/allotropia/libo-builders/wasm /bin/bash -c 'source /home/builder/emsdk/emsdk_env.sh && cd /data/lo/zstd-1.5.2/ && emmake make -j8 lib-nomt ZSTD_NO_ASM=1'

Then build POCO:
* this requires a patch (currently exists patch for 1.12.4 and untested "devel"
  branch)
* note that the header Poco/Platform.h maps EMSCRIPTEN to POCO_OS_LINUX
* it has both Makefiles and CMake but i couldn't get CMake to use CXXFLAGS to
  disable some Linux-specific code; i tried variations on:
    emcmake cmake -S .. -DENABLE_MONGODB=off -DENABLE_DATA_SQLITE=off -DENABLE_REDIS=off -DENABLE_PROMETHEUS=off -DENABLE_PDF=off -DENABLE_SEVENZIP=off -DENABLE_ZIP=off -DENABLE_CPPPARSER=off -DENABLE_POCODOC=off -DENABLE_PAGECOMPILER=off -DENABLE_PAGECOMPILER_FILE2PAGE=off -DENABLE_ACTIVERECORD=off -DENABLE_ACTIVERECORD_COMPILER=off -DENABLE_DATA=off -DENABLE_DATA_ODBC=off -DENABLE_DATA_POSTGRESQL=off -DENABLE_DATA_MYSQL=off -DENABLE_APACHECONNECTOR=off -DENABLE_JWT=off CXXFLAGS=-DPOCO_NO_LINUX_IF_PACKET_H
* Online requires a single include directory so "make install" must be used

    tar -xjvf ~/Downloads/poco-1.12.4-all.tar.bz2
    cd poco-1.12.4-all
    patch -p1 < $HOME/lo/online/wasm/poco-1.12.4-emscripten.patch
    podman run -v /data/lo/poco-1.12.4-all:/data/lo/poco-1.12.4-all:rw --security-opt=label=disable -ti public.ecr.aws/allotropia/libo-builders/wasm /bin/bash -c 'source /home/builder/emsdk/emsdk_env.sh && cd /data/lo/poco-1.12.4-all && emconfigure ./configure --static --no-samples --no-tests --omit=Crypto,NetSSL_OpenSSL,JWT,Data,Data/SQLite,Data/ODBC,Data/MySQL,Data/PostgreSQL,Zip,PageCompiler,PageCompiler/File2Page,MongoDB,Redis,ActiveRecord,ActiveRecord/Compiler,Prometheus && emmake make -j8 CC=/home/builder/emsdk/upstream/emscripten/emcc CXX=/home/builder/emsdk/upstream/emscripten/em++ LD=/home/builder/emsdk/upstream/emscripten/em++ CXXFLAGS="-DPOCO_NO_LINUX_IF_PACKET_H -pthread -s USE_PTHREADS=1 -s DISABLE_EXCEPTION_CATCHING=0" && make -j8 install INSTALLDIR=/data/lo/poco-1.12.4-all/install'

== Build Online with manually built dependencies ==

Then build Online itself:

Note that here the LO core (and builddir, if separate) must be mapped to the
same directory in the container as it was when building core, due to files
containing absolute paths.

    podman run -v /data/lo/zstd-1.5.2:/data/lo/zstd-1.5.2:ro -v /data/lo/poco-1.12.4-all:/data/lo/poco-1.12.4-all:ro -v $HOME/lo/feature_wasm:$HOME/lo/feature_wasm:ro -v /data/lo/build_feature_wasm:/data/lo/build_feature_wasm:ro -v $HOME/lo/online:$HOME/lo/online:rw -v /data/lo/build_online_wasm:/data/lo/build_online_wasm:rw --security-opt=label=disable -ti public.ecr.aws/allotropia/libo-builders/wasm /bin/bash -c "source /home/builder/emsdk/emsdk_env.sh && cd /data/lo/build_online_wasm/ && $HOME/lo/online/autogen.sh && emconfigure $HOME/lo/online/configure --with-lokit-path=$HOME/lo/feature_wasm/include --with-lo-path=/data/lo/build_feature_wasm/instdir --with-lo-builddir=/data/lo/build_feature_wasm --with-zstd-includes=/data/lo/zstd-1.5.2/lib --with-zstd-libs=/data/lo/zstd-1.5.2/lib --with-poco-includes=/data/lo/poco-1.12.4-all/install/include --with-poco-libs=/data/lo/poco-1.12.4-all/install/lib --host=wasm32-local-emscripten"
    podman run -v /data/lo/zstd-1.5.2:/data/lo/zstd-1.5.2:ro -v /data/lo/poco-1.12.4-all:/data/lo/poco-1.12.4-all:ro -v $HOME/lo/feature_wasm:$HOME/lo/feature_wasm:ro -v /data/lo/build_feature_wasm:/data/lo/build_feature_wasm:ro -v $HOME/lo/online:$HOME/lo/online:rw -v /data/lo/build_online_wasm:/data/lo/build_online_wasm:rw --security-opt=label=disable -ti public.ecr.aws/allotropia/libo-builders/wasm /bin/bash -c "source /home/builder/emsdk/emsdk_env.sh && cd /data/lo/build_online_wasm/ && emmake make"