Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

test: improve test infrastructure #554

Open
wants to merge 7 commits into
base: main
Choose a base branch
from

Commits on Nov 21, 2024

  1. test: improve test infrastructure

    This change represents a rather large re-design in how `wasi-libc`
    builds and runs its tests. Initially, WebAssembly#346 retrieved the `libc-test`
    repository and built a subset of those tests to give us some amount of
    test coverage. Later, because there was no way to add custom C tests,
     WebAssembly#522 added a `smoke` directory which allowed this. But (a) each of
    these test suites was built and run separately and (b) it was unclear
    how to add more tests flexibly--some tests should only run on `*p2`
    targets or `*-threads` targets, e.g.
    
    This change reworks all of this so that all tests are built the same
    way, in the same place. For downloaded tests like those from
    `libc-test`, I chose to add "stub tests" that `#include` the original
    version. This not only keeps all enabled tests in one place, it also
    allows us to add "directives," C comments that the `Makefile` uses to
    filter out tests for certain targets or add special compile, link or run
    flags. These rudimentary scripts, along with other Bash logic I moved
    out of the Makefile now live in the `scripts` directory.
    
    Finally, all of this is explained more clearly in an updated
    `README.md`. The hope with documenting this a bit better is that it
    would be easier for drive-by contributors to be able to either dump in
    new C tests for regressions they may find or enable more libc-tests. As
    of my current count, we only enable 40/75 of libc-test's functional
    tests, 0/228 math tests, 0/69 regression tests, and 0/79 API tests.
    Though many of these may not apply to WASI programs, it would be nice to
    explore how many more of these tests can be enabled to increase
    wasi-libc's test coverage. This change should explain how to do that
    and, with directives, make it possible to condition how the tests
    compile and run.
    abrown committed Nov 21, 2024
    Configuration menu
    Copy the full SHA
    dff5f29 View commit details
    Browse the repository at this point in the history

Commits on Nov 22, 2024

  1. ci: tweak test step

    To avoid a `Makefile` dependency issue, a previous commit changed the
    location of the download directory to live inside the build directory;
    this change propagates that to the CI configuration. Also, it is no
    longer necessary to clean up anything between runs: both the `build` and
    `run` directory are subdivided by target triple so repeated builds will
    not interfere.
    abrown committed Nov 22, 2024
    Configuration menu
    Copy the full SHA
    5795c3c View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    355683d View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    22815af View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    86bb7a1 View commit details
    Browse the repository at this point in the history

Commits on Nov 23, 2024

  1. Configuration menu
    Copy the full SHA
    0626461 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    bcbd60f View commit details
    Browse the repository at this point in the history