Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

scarb test -f should output tests in a consistent order #1805

Open
feltroidprime opened this issue Dec 3, 2024 · 1 comment
Open

scarb test -f should output tests in a consistent order #1805

feltroidprime opened this issue Dec 3, 2024 · 1 comment

Comments

@feltroidprime
Copy link

feltroidprime commented Dec 3, 2024

Problem

doing multiple scarb test -f pattern outputs the tests in different order each time
this is annoying especially when comparing resources between multiple runs, i want the first test to be the same at each run etc.

Proposed Solution

just sort alphabetically

Notes

thank you

@github-project-automation github-project-automation bot moved this to Triage in Scarb Dec 3, 2024
@feltroidprime feltroidprime changed the title scarb test -f should output test in a consistent order scarb test -f should output tests in a consistent order Dec 3, 2024
@maciektr
Copy link
Contributor

maciektr commented Dec 5, 2024

Hi!

This inconsistency stems from the fact, that tests are both executed and results are reported in parallel. (https://github.com/starkware-libs/cairo/blob/96ba347a283aaf197daa918105b5fb4fde8cc403/crates/cairo-lang-test-runner/src/lib.rs#L387)
I don't think we want to wait for all tests to complete before reporting the results.

Quick work-around for now would be to disable parallelism by setting RAYON_NUM_THREADS=1 env var when you really need the ordering of results to be deterministic.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: Triage
Development

No branches or pull requests

2 participants