We need a WebAssembly binding for zstandard

[naskarsubhajit]

I see you have compiled zstandard in c to javascript by emscripten.
We need same to get a webassembly (.wasm) . Any lead on this will be really helpful.

[yoshihitoh]

Hello

The latest Emscripten generates WebAssembly by default, and older versions can build with -s WASM=1 switch.
I'm trying to enable wasm on feature/support-wasm branch. Generating wasm is done, but I need to change API signatures to wait async-compilation of wasm.

[naskarsubhajit]

Sure that will help. Thank you. Let me go through.

[naskarsubhajit]

I see you have updated required changes for webassembly. It will be helpful if you can create a guide doc to do so.
Thank you.

[naskarsubhajit]

Hi,
I need a help, while running update-zstd-binding.sh it fails saying

Error downloading URL 'https://github.com/kripken/emscripten-fastcomp/archive/1.37.36.tar.gz': <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:661)>
Installation failed!

can you help on this.

Thanks.

[yoshihitoh]

Hi @naskarsubhajit,
I tried to install Emscripten 1.37.36, but could not repro the problem...

Installing SDK 'sdk-1.37.36-64bit'..
Installing tool 'clang-e1.37.36-64bit'..
Downloading: /emscripten/emsdk-portable/zips/emscripten-llvm-e1.37.36.tar.gz from https://s3.amazonaws.com/mozilla-games/emscripten/packages/llvm/tag/linux_64bit/emscripten-llvm-e1.37.36.tar.gz, 308409525 Bytes
Unpacking '/emscripten/emsdk-portable/zips/emscripten-llvm-e1.37.36.tar.gz' to '/emscripten/emsdk-portable/clang/e1.37.36_64bit'
Done installing tool 'clang-e1.37.36-64bit'.
Installing tool 'node-8.9.1-64bit'..
Downloading: /emscripten/emsdk-portable/zips/node-v8.9.1-linux-x64.tar.xz from https://s3.amazonaws.com/mozilla-games/emscripten/packages/node-v8.9.1-linux-x64.tar.xz, 11387108 Bytes
Unpacking '/emscripten/emsdk-portable/zips/node-v8.9.1-linux-x64.tar.xz' to '/emscripten/emsdk-portable/node/8.9.1_64bit'
Done installing tool 'node-8.9.1-64bit'.
Installing tool 'emscripten-1.37.36'..
Downloading: /emscripten/emsdk-portable/zips/1.37.36.tar.gz from https://github.com/kripken/emscripten/archive/1.37.36.tar.gz
Unpacking '/emscripten/emsdk-portable/zips/1.37.36.tar.gz' to '/emscripten/emsdk-portable/emscripten/1.37.36'
Done installing tool 'emscripten-1.37.36'.
Done installing SDK 'sdk-1.37.36-64bit'.
Writing .emscripten configuration file to user home directory /root/
The Emscripten configuration file /root/.emscripten has been rewritten with the following contents:

I don't know why the problem occurs.

I used https://github.com/yoshihitoh/zstd-codec/blob/develop/Dockerfile , and change EMCC_SDK_VERSION to 1.37.36.

[naskarsubhajit]

Thanks for your help and sorry for late reply. I was able to compile it with EMCC_SDK_VERSION to 1.37.36. I changed in premake5.lua for
project "zstd-codec-binding-wasm"

linkoptions {
"--bind",
"--memory-init-file 0",
"-s DEMANGLE_SUPPORT=1",
"-s 'EXTRA_EXPORTED_RUNTIME_METHODS=["FS"]'",
"-s MODULARIZE=1",
"-s WASM=1",
"-s SINGLE_FILE=1",
"-s BINARYEN_ASYNC_COMPILATION=1",
"-s NO_EXIT_RUNTIME=1",
"-s ALLOW_MEMORY_GROWTH=1",
"-s TOTAL_MEMORY=2147483648"
}

I see ALLOW_MEMORY_GROWTH=1 allows to expand memory runtime
but still throws below error while trying to compress 35 mb file.

abort("Cannot enlarge memory arrays. Either (1) compile with -s TOTAL_MEMORY=X with X higher than the current value 16777216, (2) compile with -s ALLOW_MEMORY_GROWTH=1 which allows increasing the size at runtime, or (3) if you want malloc to return NULL (0) instead of this abort, compile with -s ABORTING_MALLOC=0 "). Build with -s ASSERTIONS=1 for more info.

[yoshihitoh]

@naskarsubhajit Hello, sorry for response delay.
I have not tried 35 MiB file, but compilation option seems enough to compress the file.
(because TOTAL_MEMORY indicates INT_MAX=2GiB= value)

I don't know why compression method needs more memory, and cannot enlarge memory arrays.

IIRC you are using simple API ( zstd.Simple() ), is it correct?
You can use streaming API ( zstd.Streaming() ) to suppress huge memory consumption.