Skip to content

Cleanup tests by using checked in baseline files #589

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

dpaoliello
Copy link
Contributor

The current test system in ClangSharp is... awkward:

  • Each test is "implemented" in multiple derived types:
    • Each of which has a, usually, exact same copy of the test input (but it's not obvious where they may differ - either on purpose or due to a typo).
    • It's easy to accidentally call the incorrect function to run the test, which results in a coverage cap (testing the wrong OS or output version).
  • Since the test output is encoded as a C# string, it needs to be escaped and there is no syntax highlighting.
  • When tests fail, it's difficult to update the expected output:
    • By default, you only get a snippet around the first difference, requiring the test to be run multiple times.
    • The tests can be hacked to show the full output, but it gets formatted as a normal C# string literal (not a raw string) which doesn't match the string literal style currently used.

This change switches to a model where the test output is checked in as baseline files and multiple variants of the same test are run using the TestFixtureSource attribute of NUnit:

  • There are no longer a bunch of derived types, but rather a single test class that modifies its input string as required.
  • Baselines can be updated automatically by setting the AUTO_UPDATE_BASELINE env var and running tests.
  • There is only one function to run the tests, which will pick up the correct variant to use based on the test class's constructor (which NUnit automatically calls with all test combinations via the TestFixtureSource attribute).
  • Added a new osxTransform parameter to modify the actual test output in cases where MacOS differs from Linux.
  • In case of test failure, the actual output is written to the test results directory and "attached" to the test. This should allow retrieving the output from CI and manually updating the baseline files by copying the actual output (either from CI or locally) to the baseline directory.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant