Merge pull request #2482 from buildkite/add-test-analytics-issues

Add Run Issues to the Test Analytics docs

data/tiles.yml

@@ -2,12 +2,13 @@
 test_analytics_features:
   - title: "Deep performance analysis"
     url: "/docs/test-analytics/test-suites#trends-and-analysis"
-    image_path: "images/test_analytics/icon-performance-analysis.svg"
     desc: "Automatic tracing across your test suite, deeply integrated with your programming language and test framework."
   - title: "Find and fix flaky tests"
     url: "/docs/test-analytics/test-suites#detecting-flaky-tests"
-    image_path: "images/test_analytics/icon-find-flaky-tests.svg"
     desc: "Quickly identify which tests are the most disruptive for your team, and get a head-start on fixing them."
+  - title: "See urgent issues in your run"
+    url: "/docs/test-analytics/test-suites#run-issues"
+    desc: "Find and resolve issues with problematic tests in every run."
 test_analytics_guides:
   - title: "Languages"
     links:

pages/test_analytics/test_suites.md

@@ -35,6 +35,26 @@ Alternatively, you can create [scheduled builds](/docs/pipelines/scheduled-build
 
 Test Analytics reviews the test results to detect flaky tests after every test run.
 
+## Run issues
+
+>🛠 Currently in private beta
+> This feature is currently in private beta. If you would like to request access to Run issues, [contact us here](mailto:[email protected])
+
+<%= image "run-issues.png", alt: "Screenshot of a run with issues displaying in a list, including flaky, slow and failures." %>
+
+Test Analytics will automatically detect issues per run. For each test, we currently detect three issues: flaky, slow and failure. And display total number of issues, along with the following, ordered by the most problematic to least problematic:
+
+- **Flaky:** [See section on detecting flaky tests](#detecting-flaky-tests).
+
+- **Slow:** Slowness is measured by the comparative performance of tests within the current run. The system automatically flags slow tests when the slowest 1% of tests take more than 15% of the overall run time. This threshold can be manually amended within the suite settings.
+
+- **Failure:** A failed test will impact the overall test performance and efficiency.
+
+These detected issues will also show on a test execution page:
+
+<%= image "execution-issues.png", alt: "Screenshot of an execution with issues with their descriptions, displaying a dropdown, including flaky, slow and failures." %>
+
+
 ## Tracking reliability
 
 Test Analytics calculates reliability of both your entire test suite and individual tests as a measure of flakiness over time.