Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Run Issues to the Test Analytics docs #2482

Merged
merged 11 commits into from
Sep 25, 2023
5 changes: 3 additions & 2 deletions data/tiles.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,12 +2,13 @@
test_analytics_features:
- title: "Deep performance analysis"
url: "/docs/test-analytics/test-suites#trends-and-analysis"
image_path: "images/test_analytics/icon-performance-analysis.svg"
meghan-kradolfer marked this conversation as resolved.
Show resolved Hide resolved
desc: "Automatic tracing across your test suite, deeply integrated with your programming language and test framework."
- title: "Find and fix flaky tests"
url: "/docs/test-analytics/test-suites#detecting-flaky-tests"
image_path: "images/test_analytics/icon-find-flaky-tests.svg"
desc: "Quickly identify which tests are the most disruptive for your team, and get a head-start on fixing them."
- title: "See urgent issues in your run"
url: "/docs/test-analytics/test-suites#run-issues"
desc: "Find and resolve issues with problematic tests in every run."
test_analytics_guides:
- title: "Languages"
links:
Expand Down
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
20 changes: 20 additions & 0 deletions pages/test_analytics/test_suites.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,26 @@ Alternatively, you can create [scheduled builds](/docs/pipelines/scheduled-build

Test Analytics reviews the test results to detect flaky tests after every test run.

## Run issues

>🛠 Currently in private beta
> This feature is currently in private beta. If you would like to request access to Run issues, [contact us here]([email protected])

<%= image "run-issues.png", alt: "Screenshot of a run with issues displaying in a list, including flaky, slow and failures." %>

Test Analytics will automatically detect issues per run. For each test, we currently detect three issues: flaky, slow and failure. And display total number of issues, along with the following, ordered by the most problematic to least problematic:

- **Flaky:** [See section on detecting flaky tests](#detecting-flaky-tests).

- **Slow:** Slowness is measured by the comparative performance of tests within the current run. The system automatically flags slow tests when the slowest 1% of tests take more than 15% of the overall run time. This threshold can be manually amended within the suite settings.

meghan-kradolfer marked this conversation as resolved.
Show resolved Hide resolved
- **Failure:** A failed test will impact the overall test performance and efficiency.

These detected issues will also show on a test execution page:

<%= image "execution-issues.png", alt: "Screenshot of an execution with issues with their descriptions, displaying a dropdown, including flaky, slow and failures." %>


## Tracking reliability

Test Analytics calculates reliability of both your entire test suite and individual tests as a measure of flakiness over time.
Expand Down