You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardexpand all lines: README.md
+27-13
Original file line number
Diff line number
Diff line change
@@ -8,21 +8,37 @@ That's a lofty goal. As a start, we'll keep things simple.
8
8
9
9
## What can QA help me do today?
10
10
11
-
Nothing yet, because we haven't release any source code or binaries yet. :(
11
+
1. Run your tests faster. Run `qa run <type>:<glob>` in your project directory and watch your test results scream by as they run in parallel. QA provides a beautiful, easy to understand report. No Rakefile necessary!
12
+
13
+
2. See which tests are slowing down your testrun. QA highlights tests that are dramatically slower than the average test duration. Look for the 🐌 at the end of successful testrun!
14
+
15
+
3. See which test-specific stderr and stdout, even when tests are run in parallel! generated it.
16
+
17
+
4. Generate a flamegraph (or icicle graph) for the entire testrun, using `-save-flamegraph`, `-save-icegraph`. See also `-save-palette`.
18
+
19
+
5. Run your tests in parallel. QA does this for you automatically, for test types `rspec`, `rspec-pendantic`, `minitest`, `minitest-pendantic`, `test-unit`, and `test-unit-pendantic`. The `-pendantic` suffix runs each test method in its own worker process. (The default is to run each test case in its own worker fork.)
20
+
21
+
6. Track threads, GC, require, SQL queries, and other noteworthy operations in a tracing format that can be used with the `chrome://tracing` tool, using `-save-trace` option.
22
+
23
+
7. If a test fails, see source code snippets and, in some cases, actual values of local variables (use option `-errors-capture-locals`, experimental and Mac OS X only) for for each entry in the stack trace.
24
+
25
+
8. Record test output as TAP-J, using `-save-tapj` option.
26
+
27
+
9. Special ActiveRecord integration means QA will automatically partition tests across multiple databases, one per worker. If the required test databases do not exist, they will be setup automatically before tests begin. NOTE This functionality is highly experimental.
12
28
13
29
## What languages and test frameworks does QA support?
14
30
15
-
None yet. Please open an issue to request one!
31
+
Ruby's RSpec, MiniTest, test-unit. Be sure to use `bundle exec` when you run qa, if you're managing dependencies with Bundler.
16
32
17
-
## What will QA help me do tomorrow?
33
+
Please open an issue to request other languages and frameworks!
18
34
19
-
1. Run your tests faster. Just run `qa` (in your project directory) and watch your test results scream by as they run in parallel. QA auto-detects your framework and provides a beautiful, easy to understand report. No Rakefile or other boilerplate necessary.
35
+
## What will QA help me do tomorrow?
20
36
21
-
2. Analyze and eliminate [test flakiness](#whatis_flaky). By recording test outcomes across different runs, QA can integrate with existing tools (along with some new ones) to identify and diagnose flaky, slow, and otherwise problematic tests.
37
+
1. Analyze and eliminate [test flakiness](#whatis_flaky). By recording test outcomes across different runs, QA can integrate with existing tools (along with some new ones) to identify and diagnose flaky, slow, and otherwise problematic tests.
22
38
23
-
3. A faster and more focused development cycle. QA will prioritize test ordering based on what's failed recently and what files you're changing.
39
+
2. A faster and more focused development cycle. QA will prioritize test ordering based on what's failed recently and what files you're changing.
24
40
25
-
4. Run your tests *even faster*. QA will package your test execution environment, run it on a massive fleet of remote machines and stream the results back to your terminal in real time.
41
+
3. Run your tests *even faster*. QA will package your test execution environment, run it on a massive fleet of remote machines and stream the results back to your terminal in real time.
26
42
27
43
28
44
## Getting started with QA
@@ -43,7 +59,7 @@ test/
43
59
Example usage and output:
44
60
```
45
61
> cd $project
46
-
> qa
62
+
> qa run minitest:test/test**.rb
47
63
???
48
64
```
49
65
@@ -71,10 +87,8 @@ Now the good news: with QA, we've set out to address the shortcomings we see wit
71
87
## QA Roadmap
72
88
73
89
### Basic functionality
74
-
-[x] Write initial binary to detect simple Ruby minitest tests
75
-
-[x] Expand ruby test detection to rspec.
76
90
-[x] Attaching a custom reporter to test runner
77
-
-[] Use process forking to amortize cost of starting test runner
91
+
-[x] Use process forking to amortize cost of starting test runner
78
92
-[ ] Support for Go, Java, JavaScript, Python, PHP
79
93
80
94
### Parallelization
@@ -87,9 +101,9 @@ Now the good news: with QA, we've set out to address the shortcomings we see wit
87
101
-[ ] Support for additional external execution environments, like Kubernetes, Mesos
88
102
89
103
### Tracking stats / artifacts
90
-
-[] Support for working with a local audit folder
104
+
-[x] Support for working with a local audit folder
91
105
-[ ] Support for working with a remote audit folder (e.g. S3)
92
-
-[] Generate trace file
106
+
-[x] Generate trace file
93
107
-[ ] Generate single html file report with results, audits, flakiness statistics
94
108
-[ ] Add integration with system monitoring agents (e.g. performance copilot)
0 commit comments