Introducing the -watch option, and waaaay more!

So this commit is massive. Sorry about that. I got carried
away adding -watch support and ended up improving a lot of
other things along the way.

Current qa -watch features include runtime dependency analysis
of any require, load, and require_relative. Editing a test
file will re-run all test cases in that file, but editing
a dependency of a single test will only re-run test that
use that functionality. Future work may introduce finer-
grained logic.

Debut (and rename) "qa auto" as "qa watch". This functionality is
available on any qa test run command. For example:
  qa minitest -watch

Globs used by the -watch option support {foo,bar} style
expressions (in addition to * and **), so you can do obscene things
like:
  qa test-unit -watch 'test/{unit,functional}/**/test*.rb'
Note the single quotes so the shell doesn't expand the pattern
and prevent the filesystem watcher from discovering new tests
that fit the pattern.

Split -warmup into -warmup and -eager-load. This way we can avoid
-eager-load when capturing runtime dependencies, which would
introduce noise. Make -warmup default to true again, so basic
rails tests can pass out of the box with a -jobs value > 1.

Make test filter for rspec deterministic across runs (other
frameworks already had this property). This was required for
-watch functionality.

Add -memprofile and -heapdump command line options to help
diagnose memory usage problems within qa itself. This was
important for getting heap usage low enough for -watch to
be useful on large rails projects like discourse.

Improve support for syntax and load errors during test runs. This
was also important for -watch to be useful in the real word.

Also add automated demo asciicast creation. A decent amount
of work in this behemoth of a change was thanks to iterating
on qa so the demo asciicasts would look reasonable. For example:
- Strip minitest paths from backtraces.
- Refine normal qa run output to be tighter and easier to scan.
- Tweak colors and ordering of snail report.
- Fix test seeding to actually generate different values each
  qa run, discovered while trying to generate interesting data for
  a `qa flaky` demo.
- `qa flaky` output has color and no more cruft.
- `qa flaky` shows error messages in overview.
- `qa flaky` shows relative frequency of each type of error.

Other changes that got sucked into this change and couldn't
escape its gravity well:
- Rename tapjio event types to have more regular, sane names.
- Fix a bug in minitest spec support that caused problems
  for tests that had different but identically named describe
  blocks. Thanks to @halostatue for reporting it!
- Add framework-specific commands, so tab completion works. So
  instead of:
    qa run minitest:test/**/test*.rb
      use
    qa minitest test/**/test*.rb
- Fix hang after ctrl-c of qa run command.
- Enable local variable capture by default on macOS, leave disabled by
  default on Linux (for stability reasons).

Finally, update README.md and include the most basic asciicast.

Author: ajbouh

Gimmefile

@@ -18,7 +18,7 @@
       "commands": [
         "env GOPATH=$PWD GOGENPATH=$GIMME_SCRATCH/src go generate -v -x qa/main qa/runner qa/tapjio qa/analysis",
         "env GOPATH=$PWD:$GIMME_SCRATCH go build -o $GIMME_OUTPUT/bin/qa qa/main",
-        "env GOPATH=$PWD:$GIMME_SCRATCH go test -v -race qa_test",
+        "env GOPATH=$PWD:$GIMME_SCRATCH go test -v -race $(env GOPATH=$PWD go list ./... | grep -v /vendor/)",
         "true"
       ],
       "prepend-platform-env": {

README.md

@@ -2,35 +2,39 @@
 
 QA is a lightweight tool for running your tests *fast*.
 
-For years, the software testing ecosystem has lagged behind other parts of the software development pipeline. Advances in type systems, compiler technology, and prototyping environments (to name a few) have helped make software engineers much more productive. QA is an effort to make similar strides for automated testing tools.
+[![qa minitest asciicast](https://asciinema.org/a/d94fig03bbkmnzdy4mtk87ns4.png)](https://asciinema.org/a/d94fig03bbkmnzdy4mtk87ns4)
+
+Advances in type systems, compiler technology, and prototyping environments (to name a few) have helped make many software engineering activities more productive. QA is an effort to make similar strides for automated testing tools.
 
 ## What can QA help me do today?
 
-1. Run your tests faster. Run `qa run <type>` in your project directory and watch your test results scream by as they run in parallel. QA provides a beautiful, easy to understand report. No Rakefile necessary!
+1. Run your tests faster. Run `qa rspec`, `qa minitest`, or `qa test-unit` in your project directory and watch your test results scream by as they run in parallel. QA provides a beautiful, easy to understand report. No Rakefile necessary!
 
-2. See which tests are slowing down your testrun. QA highlights tests that are dramatically slower than the average test duration. Look for the 🐌 at the end of successful testrun!
+2. See which tests are slowing you down. QA highlights tests that are dramatically slower than average. Look for the 🐌 at the end of successful testrun!
 
 3. See per-test stderr and stdout. Even when running tests in parallel!
 
 4. Investigate test performance by generating flamegraphs (or icicle graph) for the entire testrun. See `-save-flamegraph`, `-save-icegraph`, and `-save-palette`.
 
-5. Run your tests in parallel. QA does this for you automatically, for test types `rspec`, `rspec-pendantic`, `minitest`, `minitest-pendantic`, `test-unit`, and `test-unit-pendantic`. The `-pendantic` suffix runs each *method* in a separate child process. (Versus each *case* in its own worker process.)
+5. Run your tests in parallel. QA does this for you automatically. Use `-squash=none` to run each *method* in a separate child process. The default is `-squash=file`, which runs each *file* in its own process.
 
-6. Analyze and eliminate [test flakiness](#whatis_flaky). The `-archive-base-dir` option for `qa run` records test outcomes across different runs. Use the `qa flaky` command with the same `-archive-base-dir` option to identify and diagnose flaky tests. This is new functionality, so please [open an issue](https://github.com/ajbouh/qa/issues/new) with questions and feedback!
+6. Analyze and eliminate [test flakiness](#whatis_flaky). The `-archive-base-dir` option records test outcomes across different runs. Use the `qa flaky` command with the same `-archive-base-dir` option to identify and diagnose flaky tests. This is new functionality, so please [open an issue](https://github.com/ajbouh/qa/issues/new) with questions and feedback!
 
 7. Track threads, GC, require, SQL queries, and other noteworthy operations in a tracing format that can be used with the `chrome://tracing` tool, using `-save-trace` option.
 
-8. See source code snippets and (with the experimental `-errors-capture-locals`) actual values of local variables for each from of an error's stack trace.
+8. See source code snippets and actual values of local variables for each frame of an error's stack trace.
 
 9. Record test output as TAP-J, using `-save-tapj` option.
 
 10. Automatically partition Rails tests across multiple databases, one per worker (Using custom ActiveRecord integration logic). If the required test databases do not exist, they will be setup automatically before tests begin. NOTE This functionality is highly experimental. Disable it with `-warmup=false`. Please [open an issue](https://github.com/ajbouh/qa/issues/new) if you have trouble.
 
 ## What languages and test frameworks does QA support?
 
-Ruby's RSpec, MiniTest, test-unit. Be sure to use `bundle exec` when you run qa, if you're managing dependencies with Bundler. For example, if you're using rspec:
+Ruby 2.3+, and any of: RSpec, MiniTest, test-unit.
+
+Be sure to use `bundle exec` when you run qa, if you're managing dependencies with Bundler. For example, if you're using Rspec:
 ```
-bundle exec qa run rspec
+bundle exec qa rspec
 ```
 
 ## What will QA help me do tomorrow?
@@ -66,15 +70,15 @@ test/
 Example usage and output:
 ```
 > cd $project
-> qa run minitest
+> qa minitest
 ...
 ```
 
 ## Troubleshooting QA
 
 Since QA is still in alpha, there are a number of rough edges.
 
-If `qa run` seems to be acting strangely and isn't providing a reasonable error message, you may be experiencing a bug relating to swallowed error output. This is tied to QA's stdout and stderr capture logic. Adding the `-capture-standard-fds=false` option will disable the capture logic and should allow the original error to bubble up. Please [open an issue](https://github.com/ajbouh/qa/issues/new) with the error output.
+If `qa` seems to be acting strangely and isn't providing a reasonable error message, you may be experiencing a bug relating to swallowed error output. This is tied to QA's stdout and stderr capture logic. Adding the `-capture-standard-fds=false` option will disable the capture logic and should allow the original error to bubble up. Please [open an issue](https://github.com/ajbouh/qa/issues/new) with the error output.
 
 ## What are flaky tests?<a name="whatis_flaky"></a>
 
@@ -87,21 +91,21 @@ So that's the bad news: by their very nature, flaky tests are hard to avoid. In
 ## How do I use QA to detect flaky tests?
 An example session
 ```
-$ qa run -archive-base-dir ~/.qa/archive
+$ qa minitest -archive-base-dir ~/.qa/archive
   # ... unexpected test failure
-$ qa run -archive-base-dir ~/.qa/archive
+$ qa minitest -archive-base-dir ~/.qa/archive
   # ... that same test now passes
 ```
 
-To analyze the last few days worth of test results, you can use the `qa flaky` command. It's important to use the same value for `-archive-base-dir` as given to `qa run`. For example, continuing the session from above:
+To analyze the last few days worth of test results, you can use the `qa flaky` command. It's important to use the same value for `-archive-base-dir` as given to other `qa` commands. For example, continuing the session from above:
 
 ```
 $ qa flaky -archive-base-dir ~/.qa/archive
 ```
 
 ## How does QA detect flaky tests?
 
-At a high level, QA considers a test to be flaky if, for a particular code revision, that test has both passed and failed. That's why you should provide a `-suite-coderef` value to `qa run`.
+At a high level, QA considers a test to be flaky if, for a particular code revision, that test has both passed and failed. That's why you should provide a `-suite-coderef` value to `qa` commands.
 
 At a low level, QA uses a few tricks to find as many examples of a flaky failure as it can. The actual algorithm for discovering flaky tests is:
 - Fingerprint all failures using:
@@ -111,8 +115,8 @@ At a low level, QA uses a few tricks to find as many examples of a flaky failure
 - Find all tests that, for a single revision, have both passed and failed.
 - Put test failures from different revisions in the same bucket if their fingerprint matches a known flaky test
 
-## How will QA help me with test flakiness?
-Now the good news: with QA, we've set out to address the shortcomings we see with today's testing tools. We want a toolset that's *fast* and gives us more firepower for dealing with the reality of flaky tests.
+## How will future versions of QA help me with test flakiness?
+With QA, we've set out to address the shortcomings we see with today's testing tools. We want a toolset that's *fast* and gives us more firepower for dealing with the reality of flaky tests.
 
 - **Testing code that includes dependencies you didn't write?** QA will isolate tests from network services using an OS-specific sandbox.
 
@@ -153,7 +157,7 @@ Now the good news: with QA, we've set out to address the shortcomings we see wit
 - [X] Add TAP-J analysis tools, to detect rates of flakiness in tests
 - [ ] Add support for marking some tests as (implicitly?) new, forcing them to be run many times and pass every time
 - [ ] Add support for marking tests as flaky, separating their results from the results of other tests
-- [ ] For tests that are failing flakily, show distribution of which line failed, test duration, version of code
+- [x] For tests that are failing flakily, show distribution of which line failed, test duration, version of code
 
 ### Continuous integration
 - [ ] Add support for auto-filing issues (or updating existing issues) when a merged test fails that should not be flaky
@@ -162,7 +166,8 @@ Now the good news: with QA, we've set out to address the shortcomings we see wit
 ### Local development
 - [ ] Order test run during local development based on what's failed recently
 - [ ] Line-level code coverage report
-- [ ] Rerunning tests during local development affected by what code you just modified (test code or AUT, using code coverage analysis)
+- [x] Rerunning tests during local development affected by what code you just modified (test code or AUT, using code coverage analysis)
+- [ ] Line-level test rerunning, using code coverage
 - [ ] Limit tests to files that are open in editor (open test files, open AUT files, etc)
 - [ ] Can run with git-bisect to search for commit that introduced a bug
 - [ ] Suggest which failing tests to debug first (based on heuristics)

demos/flaky/flaky.script

@@ -0,0 +1,34 @@
+# qa flaky
+
+# We'll be using the ruby-mime-types test suite to
+# demonstrate finding flaky tests with qa.
+#
+git remote get-url origin; git reset --hard aa499d1; tree ./test
+
+#
+# First fix a parallelism bug in the cache tests.
+# Test processes must use separate scratch files.
+#
+git apply ../patches/fix-cache-test-parallelism.patch
+git diff -U2; git add --update
+
+#
+# Break one test so it will never pass, and another
+# test to pass ~25% of the time, fail one way ~50%
+# of the time, and otherwise fail another way.
+#
+git apply ../patches/introduce-faults.patch
+git diff -U2; git add --update
+
+#
+# Record enough data for qa to analyze.
+#
+for x in $(seq 12)
+do
+  bundle exec qa run -archive-base-dir=.qa-archive -quiet minitest
+done
+
+#
+# Now use qa to find that flaky test!
+#
+qa flaky -archive-base-dir=.qa-archive

demos/flaky/patches/fix-cache-test-parallelism.patch

@@ -0,0 +1,13 @@
+diff --git a/test/test_mime_types_cache.rb b/test/test_mime_types_cache.rb
+index 3b5859b..dd359d6 100644
+--- a/test/test_mime_types_cache.rb
++++ b/test/test_mime_types_cache.rb
+@@ -12,7 +12,7 @@ describe MIME::Types::Cache do
+     require 'fileutils'
+ 
+     MUTEX.synchronize do
+-      @cache_file = File.expand_path('../cache.tst', __FILE__)
++      @cache_file = File.expand_path("../cache.tst#{ENV['QA_WORKER']}", __FILE__)
+       ENV['RUBY_MIME_TYPES_CACHE'] = @cache_file
+       clear_cache_file
+ 

demos/flaky/patches/introduce-faults.patch

@@ -0,0 +1,25 @@
+diff --git a/test/test_mime_types.rb b/test/test_mime_types.rb
+index caadc37..1ded080 100644
+--- a/test/test_mime_types.rb
++++ b/test/test_mime_types.rb
+@@ -117,9 +117,9 @@ describe MIME::Types do
+     end
+ 
+     it 'successfully adds from another MIME::Types' do
+-      mt = MIME::Types.new
++      mt = rand(2) == 0 ? nil : MIME::Types.new
+       mt.add(mime_types)
+-      assert_equal mime_types.count, mt.count
++      assert_equal mime_types.count, rand(2) == 0 ? mt.count : -1
+ 
+       mime_types.each do |type|
+         assert_equal mt[type.content_type], [ type ]
+@@ -155,7 +155,7 @@ describe MIME::Types do
+ 
+   describe '#count' do
+     it 'can count the number of types inside' do
+-      assert_equal 6, mime_types.count
++      assert_equal 4, mime_types.count
+     end
+   end
+ end

demos/record-2script.sh

@@ -0,0 +1,181 @@
+#!/bin/bash
+
+# usage:
+# record-2script.sh <script_file> <output_path>
+set -e
+
+SCRIPT=$1
+OUTPUT_PATH=$2
+
+# Title is first line of script, with leading "# " removed.
+TITLE=$(head -n 1 $SCRIPT | tail -c +3)
+DEMO_SEMAPHORE=$PWD/.tmux-semaphore
+DEMO_RCFILE=$PWD/.bashrc
+MAX_WAIT=2
+HEIGHT=30
+WIDTH=200
+COMMENT_KEY_DELAY=0.02
+COMMENT_SPACE_DELAY=0.18
+COMMAND_KEY_DELAY=0.06
+LINE_DELAY=1.8
+
+trap '(test -e $DEMO_SEMAPHORE && rm $DEMO_SEMAPHORE); (test -e $DEMO_RCFILE && rm $DEMO_RCFILE)' EXIT
+
+SESSION=$USER
+NESTED_SESSION=${SESSION}_nested
+
+function update_semaphore_token() {
+  head -c 20 /dev/urandom | xxd -p > "${DEMO_SEMAPHORE}$1"
+}
+
+function await_semaphore_token() {
+  tmux wait-for "$(cat ${DEMO_SEMAPHORE}$1)"
+}
+
+function start_tmux_session() {
+  export DEMO_SEMAPHORE
+  cat > $DEMO_RCFILE <<'EOF'
+PS1='\e[92m»\e[m $(tmux wait-for -S $(cat $DEMO_SEMAPHORE))'
+PS2='  \e[92m…\e[m $(tmux wait-for -S $(cat $DEMO_SEMAPHORE))'
+EOF
+
+  update_semaphore_token
+  tmux -2 \
+      new-session \
+      -x $WIDTH \
+      -y $HEIGHT \
+      -d \
+      -s $SESSION \
+      asciinema rec -y \
+          --title="$TITLE" \
+          --max-wait="$MAX_WAIT" \
+          --command="/bin/bash --noprofile --rcfile $DEMO_RCFILE" \
+          $OUTPUT_PATH
+}
+
+function type_tmux_keys() {
+  tmux_target="$1"
+  keys="$2"
+
+  tmux select-pane -t "$tmux_target"
+  tmux send-keys -t "$tmux_target" "$keys"
+}
+
+function type_tmux_line() {
+  tmux_target="$1"
+  line="$2"
+
+  tmux select-pane -t $tmux_target
+
+  eol_key=C-m
+  if [ "$line" != "#" ]; then
+    word_delay=$COMMAND_KEY_DELAY
+    char_delay=$COMMAND_KEY_DELAY
+    if [ "${line:0:1}" = "#" ]; then
+      word_delay=$COMMENT_SPACE_DELAY
+      char_delay=$COMMENT_KEY_DELAY
+    fi
+
+    # Comment out to keep the leading "# " for comments and use
+    # if [ "${line:0:2}" = "# " ]; then
+    #   line=$(echo -n "$line" | tail -c +3)
+    #   eol_key=C-c
+    # fi
+
+    while IFS='' read -n 1 char; do
+      if [ "$char" = ' ' ]; then
+        key_delay=$word_delay
+      else
+        key_delay=$char_delay
+      fi
+
+      # For some reason, we need to escape semicolons
+      if [ "$char" = ';' ]; then
+        char='\;'
+      fi
+      tmux send-keys -t "$tmux_target" -l "$char"
+      sleep $key_delay
+    done < <(echo -n "$line")
+  fi
+
+  tmux send-keys -t "$tmux_target" $eol_key
+}
+
+function drive_tmux_session() {
+  tmux_session=$1
+  tmux_script=$2
+
+  has_split=
+  while IFS= read line; do
+    if [ "$line" = "" ]; then
+      sleep $LINE_DELAY
+      continue
+    fi
+
+    # Figure out which session...
+    session_index=$(echo "$line" | cut -d' ' -f1)
+    line="$(echo "$line" | cut -d' ' -f2-)"
+
+    if [ "${session_index:0:1}" = "1" ]; then
+      if [ -z "$has_split" ]; then
+        has_split=1
+        update_semaphore_token .1
+        tmux split-window -t $NESTED_SESSION -h -p 55 env DEMO_SEMAPHORE=${DEMO_SEMAPHORE}.1 /bin/bash --noprofile --rcfile $DEMO_RCFILE
+        await_semaphore_token .1
+      fi
+    fi
+
+    # Is this an asynchronous line?
+    if echo "$session_index" | grep -q -E '^\d+&$'; then
+      session_index=${session_index:0:${#session_index} - 1}
+      type_tmux_line $tmux_session.$session_index "$line"
+      # Or an key line
+    elif echo "$session_index" | grep -q -E '^\d+E$'; then
+      session_index=${session_index:0:${#session_index} - 1}
+      type_tmux_keys $tmux_session.$session_index "$line"
+      # Is this a well-formed synchronous line?
+    elif echo "$session_index" | grep -q -E '^\d+$'; then
+      update_semaphore_token .$session_index
+      type_tmux_line $tmux_session.$session_index "$line"
+      await_semaphore_token .$session_index
+
+      heredoc_token="$(echo "$line" | grep -E '<<([^ ]+)' | sed -E "s/^.*<<'?([^ ']+).*\$/\\1/")"
+      if [ -n "$heredoc_token" ]; then
+        while IFS= read heredoc_line; do
+          tmux send-keys -t "$tmux_session.$session_index" -l "$heredoc_line"
+          update_semaphore_token .$session_index
+          tmux send-keys -t "$tmux_session.$session_index" C-m
+          await_semaphore_token .$session_index
+          if [ "$heredoc_line" == "$heredoc_token" ]; then
+            break
+          fi
+        done
+      fi
+    else
+      echo "Malformed line: $line" >&2
+      exit 1
+    fi
+  done < <(tail -n +2 $tmux_script)
+
+  sleep $LINE_DELAY
+  tmux send-keys -t $tmux_session.1 C-d
+  tmux send-keys -t $tmux_session.0 C-d
+}
+
+start_tmux_session
+await_semaphore_token
+
+update_semaphore_token .0
+tmux send-keys -l "exec tmux new-session -s $NESTED_SESSION env DEMO_SEMAPHORE=${DEMO_SEMAPHORE}.0 /bin/bash --noprofile --rcfile $DEMO_RCFILE ';' set status off"
+tmux send-keys C-m
+
+await_semaphore_token .0
+
+drive_tmux_session $NESTED_SESSION $SCRIPT &
+
+tmux set-window-option -t $SESSION force-width $WIDTH
+tmux set-window-option -t $SESSION force-height $HEIGHT
+tmux set-window-option -t $SESSION aggressive-resize off
+
+# exec tmux attach-session -r -t $SESSION
+exec tmux attach-session -t $SESSION