-
Notifications
You must be signed in to change notification settings - Fork 203
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
test: improve test infrastructure #554
Open
abrown
wants to merge
7
commits into
WebAssembly:main
Choose a base branch
from
abrown:improve-test-infrastructure
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Commits on Nov 21, 2024
-
test: improve test infrastructure
This change represents a rather large re-design in how `wasi-libc` builds and runs its tests. Initially, WebAssembly#346 retrieved the `libc-test` repository and built a subset of those tests to give us some amount of test coverage. Later, because there was no way to add custom C tests, WebAssembly#522 added a `smoke` directory which allowed this. But (a) each of these test suites was built and run separately and (b) it was unclear how to add more tests flexibly--some tests should only run on `*p2` targets or `*-threads` targets, e.g. This change reworks all of this so that all tests are built the same way, in the same place. For downloaded tests like those from `libc-test`, I chose to add "stub tests" that `#include` the original version. This not only keeps all enabled tests in one place, it also allows us to add "directives," C comments that the `Makefile` uses to filter out tests for certain targets or add special compile, link or run flags. These rudimentary scripts, along with other Bash logic I moved out of the Makefile now live in the `scripts` directory. Finally, all of this is explained more clearly in an updated `README.md`. The hope with documenting this a bit better is that it would be easier for drive-by contributors to be able to either dump in new C tests for regressions they may find or enable more libc-tests. As of my current count, we only enable 40/75 of libc-test's functional tests, 0/228 math tests, 0/69 regression tests, and 0/79 API tests. Though many of these may not apply to WASI programs, it would be nice to explore how many more of these tests can be enabled to increase wasi-libc's test coverage. This change should explain how to do that and, with directives, make it possible to condition how the tests compile and run.
Configuration menu - View commit details
-
Copy full SHA for dff5f29 - Browse repository at this point
Copy the full SHA dff5f29View commit details
Commits on Nov 22, 2024
-
To avoid a `Makefile` dependency issue, a previous commit changed the location of the download directory to live inside the build directory; this change propagates that to the CI configuration. Also, it is no longer necessary to clean up anything between runs: both the `build` and `run` directory are subdivided by target triple so repeated builds will not interfere.
Configuration menu - View commit details
-
Copy full SHA for 5795c3c - Browse repository at this point
Copy the full SHA 5795c3cView commit details -
Configuration menu - View commit details
-
Copy full SHA for 355683d - Browse repository at this point
Copy the full SHA 355683dView commit details -
Configuration menu - View commit details
-
Copy full SHA for 22815af - Browse repository at this point
Copy the full SHA 22815afView commit details -
Configuration menu - View commit details
-
Copy full SHA for 86bb7a1 - Browse repository at this point
Copy the full SHA 86bb7a1View commit details
Commits on Nov 23, 2024
-
Configuration menu - View commit details
-
Copy full SHA for 0626461 - Browse repository at this point
Copy the full SHA 0626461View commit details -
Configuration menu - View commit details
-
Copy full SHA for bcbd60f - Browse repository at this point
Copy the full SHA bcbd60fView commit details
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.