For platforms that implement runtime fallback logic, all paths should be tested

[briansmith]

Consider the case of Linux. We hope to use the getrandom syscall and then if that fails, we fall back to doing file I/O. We should test at least the following scenerios:

• Linux returns ENOSYS, which should trigger the fallback to file I/O.
• Linux returns EPERM, which is presently expected to do the same fallback, though there's an issue on file to change this.
• The syscall succeeds.

This type of exhaustive testing might not be practical on all targets. However, BoringSSL has shown how to do it on Linux; see https://boringssl.googlesource.com/boringssl/+/refs/heads/master/crypto/fipsmodule/rand/urandom_test.cc.

Backends with runtime fallback logic:

• linux_android_with_fallback (fallack to /dev/urandom)
• netbsd (fallback to KERN_ARND)
• wasm_js (runtime selection between Web and Node.js APIs, there is also an IE 11 workaround)

[josephlr]

I agree that this testing will be challenging for Platforms that aren't Tier 1, but I think we should look into it for Linux and MacOS. Maybe using LD_PRELOAD shenanigans with an older libc could work?

For the syscall vs file fallback, it would be interesting if we could get a ptrace-based test to work.

[newpavlov]

Maybe using LD_PRELOAD shenanigans with an older libc could work?

Using just older libc will not help since we use libc::syscall. It may be worth to change code of getrandom_test_linux_fallback to use dlopen instead.