CRYPTO-172: Add support for Linux-riscv64

[luhenry]

Java supports RISC-V since version 19. Given Apache Common Crypto is a commonly used library in the Java ecosystem, it's important for it to support RISC-V as well.

This patch adds the linux-riscv64 target similarly to linux-aarch64.

I've verified it builds with docker compose -f src/docker/docker-compose.yaml run crypto src/docker/build.sh.

[sebbASF]

We also need to ensure that the GH Actions include some tests for the architecture if possible.

[luhenry]

@sebbASF I couldn't find how the test is run on linux-aarch64. Where could I find it?

[sebbASF]

I don't know, that's why I asked. I assume there is some GH documentation on which architectures are available.
If risc64 is not available as a test host, then this needs to be documented, as it means the combination has not been tested.

[luhenry]

I've added a docker container to run the test and ran it successfully:

$> docker compose -f src/docker/docker-compose.yaml run crypto-riscv64-test
WARN[0000] Found orphan containers ([docker-cross-compile-riscv64-build-run-397c7aebdada docker-cross-compile-riscv64-runtime-setup-1 docker-cross-compile-riscv64-build-run-93a982a05cdc docker-cross-compile-riscv64-build-run-7dc6a41f1d5b docker-cross-compile-riscv64-build-run-644e853cbf55 docker-cross-compile-riscv64-build-run-3b09454dc280]) for this project. If you removed or renamed this service in your compose file, you can run this command with the --remove-orphans flag to clean it up.
[+] Running 1/1
 ! crypto-riscv64-test Warning                                                                                 2.0s
[+] Building 303.7s (9/9) FINISHED                                                docker-container:festive_mccarthy
 => [crypto-riscv64-test internal] load build definition from Dockerfile.riscv64                               0.0s
 => => transferring dockerfile: 1.97kB                                                                         0.0s
 => [crypto-riscv64-test internal] load metadata for docker.io/riscv64/ubuntu:20.04                            0.7s
 => [crypto-riscv64-test internal] load .dockerignore                                                          0.0s
 => => transferring context: 2B                                                                                0.0s
 => CACHED [crypto-riscv64-test 1/4] FROM docker.io/riscv64/ubuntu:20.04@sha256:c0e96f609e5f128bdb6d24d21d3bf  0.0s
 => => resolve docker.io/riscv64/ubuntu:20.04@sha256:c0e96f609e5f128bdb6d24d21d3bf0a9af6f17234976fa574dad7146  0.0s
 => [crypto-riscv64-test 2/4] RUN apt-get update && apt-get --assume-yes install software-properties-common  274.2s
 => [crypto-riscv64-test 3/4] RUN curl -L https://dlcdn.apache.org/maven/maven-3/3.8.8/binaries/apache-maven-  1.0s
 => [crypto-riscv64-test 4/4] WORKDIR /home/crypto                                                             0.0s
 => [crypto-riscv64-test] exporting to docker image format                                                    27.7s
 => => exporting layers                                                                                       17.4s
 => => exporting manifest sha256:9068927ccfbc46788ec15578541ebc06302b538d4ed75e308b694e1132dd55fd              0.0s
 => => exporting config sha256:27adda314855e324670f04d2cc656b48c19879ccf16b9e4f32f32cd273bb3319                0.0s
 => => sending tarball                                                                                        10.2s
 => [crypto-riscv64-test crypto-riscv64-test] importing to docker                                              9.1s
Apache Maven 3.8.8 (4c87b05d9aedce574290d1acc98575ed5eb6cd39)
Maven home: /opt/maven
Java version: 21-ea, vendor: Private Build, runtime: /usr/lib/jvm/java-21-openjdk-riscv64
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "6.2.0-35-generic", arch: "riscv64", family: "unix"
[INFO] Scanning for projects...
[INFO]
[INFO] -----------------< org.apache.commons:commons-crypto >------------------
[INFO] Building Apache Commons Crypto 1.2.1-SNAPSHOT
[INFO]   from pom.xml
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- maven-surefire-plugin:3.1.2:test (default-cli) @ commons-crypto ---
[INFO] Using auto detected provider org.apache.maven.surefire.junitplatform.JUnitPlatformProvider
[INFO]
[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running org.apache.commons.crypto.CryptoTest
OpenJDK 64-Bit Server VM warning: You have loaded library /tmp/commons-crypto-1e590661-aa26-4da1-b5c2-8d721ca08d18-libcommons-crypto.so which might have disabled stack guard. The VM will try to fix the stack guard now.
It's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'.
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.095 s -- in org.apache.commons.crypto.CryptoTest
[INFO] Running org.apache.commons.crypto.OsInfoTest
Linux/riscv64
Linux
riscv64
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.663 s -- in org.apache.commons.crypto.OsInfoTest
[INFO] Running org.apache.commons.crypto.utils.UtilsTest
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.159 s -- in org.apache.commons.crypto.utils.UtilsTest
[INFO] Running org.apache.commons.crypto.utils.EnumTest
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.033 s -- in org.apache.commons.crypto.utils.EnumTest
[INFO] Running org.apache.commons.crypto.jna.CtrCryptoJnaStreamTest
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 12.60 s -- in org.apache.commons.crypto.jna.CtrCryptoJnaStreamTest
[INFO] Running org.apache.commons.crypto.jna.CtrNoPaddingCipherJnaStreamTest
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.532 s -- in org.apache.commons.crypto.jna.CtrNoPaddingCipherJnaStreamTest
[INFO] Running org.apache.commons.crypto.jna.OpenSslJnaTest
Apache Commons Crypto OpenSslJna: enabled = true, version = 0x1010106F
OpenSSLVersion(0): OpenSSL 1.1.1f  31 Mar 2020
OpenSSLVersion(1): compiler: gcc -fPIC -pthread -Wa,--noexecstack -Wall -Wa,--noexecstack -g -O2 -fdebug-prefix-map=/build/openssl-zF9Nk8/openssl-1.1.1f=. -fstack-protector-strong -Wformat -Werror=format-security -DOPENSSL_TLS_SECURITY_LEVEL=2 -DOPENSSL_USE_NODELETE -DOPENSSL_PIC -DNDEBUG -Wdate-time -D_FORTIFY_SOURCE=2
OpenSSLVersion(2): built on: Tue Oct 10 09:03:48 2023 UTC
OpenSSLVersion(3): platform: debian-riscv64
OpenSSLVersion(4): OPENSSLDIR: "/usr/lib/ssl"
OpenSSLVersion(5): ENGINESDIR: "/usr/lib/riscv64-linux-gnu/engines-1.1"
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.394 s -- in org.apache.commons.crypto.jna.OpenSslJnaTest
[INFO] Running org.apache.commons.crypto.jna.OpenSslJnaCipherTest
[INFO] Tests run: 11, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.447 s -- in org.apache.commons.crypto.jna.OpenSslJnaCipherTest
[INFO] Running org.apache.commons.crypto.jna.CbcNoPaddingCipherJnaStreamTest
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.448 s -- in org.apache.commons.crypto.jna.CbcNoPaddingCipherJnaStreamTest
[INFO] Running org.apache.commons.crypto.jna.PositionedCryptoInputStreamJnaTest
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.210 s -- in org.apache.commons.crypto.jna.PositionedCryptoInputStreamJnaTest
[INFO] Running org.apache.commons.crypto.jna.OpenSslNativeJnaTest
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.300 s -- in org.apache.commons.crypto.jna.OpenSslNativeJnaTest
[INFO] Running org.apache.commons.crypto.jna.OpenSslJnaCryptoRandomTest
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 16.24 s -- in org.apache.commons.crypto.jna.OpenSslJnaCryptoRandomTest
[INFO] Running org.apache.commons.crypto.jna.CbcPkcs5PaddingCipherJnaStreamTest
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.609 s -- in org.apache.commons.crypto.jna.CbcPkcs5PaddingCipherJnaStreamTest
[INFO] Running org.apache.commons.crypto.cipher.CryptoCipherTest
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.328 s -- in org.apache.commons.crypto.cipher.CryptoCipherTest
[INFO] Running org.apache.commons.crypto.cipher.OpenSslCipherTest
[INFO] Tests run: 16, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.684 s -- in org.apache.commons.crypto.cipher.OpenSslCipherTest
[INFO] Running org.apache.commons.crypto.cipher.CryptoCipherFactoryTest
[INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.266 s -- in org.apache.commons.crypto.cipher.CryptoCipherFactoryTest
[INFO] Running org.apache.commons.crypto.cipher.OpenSslCommonModeTest
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.014 s -- in org.apache.commons.crypto.cipher.OpenSslCommonModeTest
[INFO] Running org.apache.commons.crypto.cipher.GcmCipherTest
[INFO] Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.431 s -- in org.apache.commons.crypto.cipher.GcmCipherTest
[INFO] Running org.apache.commons.crypto.cipher.JceCipherTest
[INFO] Tests run: 11, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.126 s -- in org.apache.commons.crypto.cipher.JceCipherTest
[INFO] Running org.apache.commons.crypto.stream.CbcPkcs5PaddingCipherStreamTest
[INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.579 s -- in org.apache.commons.crypto.stream.CbcPkcs5PaddingCipherStreamTest
[INFO] Running org.apache.commons.crypto.stream.output.StreamOutputTest
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.037 s -- in org.apache.commons.crypto.stream.output.StreamOutputTest
[INFO] Running org.apache.commons.crypto.stream.CbcNoPaddingCipherStreamTest
[INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.528 s -- in org.apache.commons.crypto.stream.CbcNoPaddingCipherStreamTest
[INFO] Running org.apache.commons.crypto.stream.input.ChannelInputTest
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.015 s -- in org.apache.commons.crypto.stream.input.ChannelInputTest
[INFO] Running org.apache.commons.crypto.stream.PositionedCryptoInputStreamTest
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.747 s -- in org.apache.commons.crypto.stream.PositionedCryptoInputStreamTest
[INFO] Running org.apache.commons.crypto.stream.CtrNoPaddingCipherStreamTest
[INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.471 s -- in org.apache.commons.crypto.stream.CtrNoPaddingCipherStreamTest
[INFO] Running org.apache.commons.crypto.stream.CtrCryptoStreamTest
[INFO] Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.703 s -- in org.apache.commons.crypto.stream.CtrCryptoStreamTest
[INFO] Running org.apache.commons.crypto.random.OpenSslCryptoRandomTest
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 9.651 s -- in org.apache.commons.crypto.random.OpenSslCryptoRandomTest
[INFO] Running org.apache.commons.crypto.random.CryptoRandomFactoryTest
[INFO] Tests run: 13, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.530 s -- in org.apache.commons.crypto.random.CryptoRandomFactoryTest
[INFO] Running org.apache.commons.crypto.random.JavaCryptoRandomTest
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 19.79 s -- in org.apache.commons.crypto.random.JavaCryptoRandomTest
[INFO] Running org.apache.commons.crypto.random.OsCryptoRandomTest
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.594 s -- in org.apache.commons.crypto.random.OsCryptoRandomTest
[INFO] Running org.apache.commons.crypto.NativeCodeLoaderTest
[WARNING] Tests run: 5, Failures: 0, Errors: 0, Skipped: 3, Time elapsed: 1.206 s -- in org.apache.commons.crypto.NativeCodeLoaderTest
[INFO]
[INFO] Results:
[INFO]
[WARNING] Tests run: 141, Failures: 0, Errors: 0, Skipped: 3
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  02:18 min
[INFO] Finished at: 2023-10-31T13:04:26Z
[INFO] ------------------------------------------------------------------------

[luhenry]

@sebbASF I've added testing on GitHub Actions for linux-aarch64 and linux-riscv64. You'll note I had to downgrade JNA to 5.12 because of java-native-access/jna#1557.

[garydgregory]

I would not rollback JNA for ALL configurations just because one has special requirements, just downgrade in that case.

[luhenry]

I would not rollback JNA for ALL configurations just because one has special requirements, just downgrade in that case.

It's a build-time dependency with the commons-crypto package being the same across all platforms. I'm afraid there is not an easy way to specify per-arch dependency in that case. I'll keep monitoring the JNA linux-riscv64 issue and update anything here whenever there is an update (that we merge this PR with the downgrade first, or that we wait to merge that PR)

[sebbASF]

Please see individual review comments

[sebbASF]

@sebbASF I've added testing on GitHub Actions for linux-aarch64 and linux-riscv64. You'll note I had to downgrade JNA to 5.12 because of java-native-access/jna#1557.

BTW: well done for finding out how to do that!

[codecov-commenter]

Codecov Report

Merging #264 (4c69119) into master (c2865b7) will not change coverage.
Report is 9 commits behind head on master.
The diff coverage is n/a.

@@            Coverage Diff            @@
##             master     #264   +/-   ##
=========================================
  Coverage     75.27%   75.27%           
  Complexity      441      441           
=========================================
  Files            38       38           
  Lines          1820     1820           
  Branches        178      178           
=========================================
  Hits           1370     1370           
  Misses          340      340           
  Partials        110      110           

📣 Codecov offers a browser extension for seamless coverage viewing on GitHub. Try it in Chrome or Firefox today!

[sebbASF]

See comments

[sebbASF]

See review comment

[sebbASF]

The cross-builds now seem to be working, though they do take a long time.
Are they doing more than they need?

Also it looks like they don't only build the native files for their own architectures.
They probably need their own build.sh files, or build.sh needs a parameter to control which native files are built

[sebbASF]

Looks like the docker run commands no longer need the -test services

[luhenry]

The cross-builds now seem to be working, though they do take a long time.
Are they doing more than they need?

The step that takes the longest is setting up the java and openssl packages on the aarch64/riscv64 docker images. (ex: https://github.com/apache/commons-crypto/actions/runs/6733654561/job/18304390181#step:4:6004, 6 out of 10 minutes just on that step). This docker image could be cached by pushing it to the repo's docker repository. I've seen that done as part of https://github.com/uraimo/run-on-arch-action.

[sebbASF]

The cross-builds now seem to be working, though they do take a long time.
Are they doing more than they need?

The step that takes the longest is setting up the java and openssl packages on the aarch64/riscv64 docker images. (ex: https://github.com/apache/commons-crypto/actions/runs/6733654561/job/18304390181#step:4:6004, 6 out of 10 minutes just on that step). This docker image could be cached by pushing it to the repo's docker repository. I've seen that done as part of https://github.com/uraimo/run-on-arch-action.

Do we need the full JDK and the dev version of libssl?
Installing only the JVM and standard libssl might speed things up.

[luhenry]

Do we need the full JDK and the dev version of libssl?

I've trimmed it down to openjdk-17-jre-headless coming from ubuntu packages and not a specific PPA, but I needed to keep libssl-dev as libssl1.1 only installs libcrypto.so.1.1 while the native commons-crypto library links against libcrypto.so, leading to a linker issue.

Overall, it goes from ~360s to ~110s locally. And on CI, it goes down to ~150s.

[sebbASF]

Thanks for your patience dedication getting this together!
Any further tweaks can be made in the main repo

[luhenry]

@sebbASF Thank you for the review and continuous feedback as well! :)