A better way to deal with tabular test cases

[dabrahams]

Description

Here's an example I'm working with:

let regularCases: [String: [(input: String, expected: Bool)]] = [
  // Basic cases
  "": [("", true), ("x", false), ("xy", false)],
  "x": [("", false), ("x", true), ("xy", false)],
  "x+": [("", false), ("x", true), ("xy", false), ("xx", true)],
  "x*": [("", true), ("x", true), ("xy", false), ("xx", true)],
  "x?": [("", true), ("x", true), ("xy", false), ("xx", false)],
  "x|y": [("", false), ("x", true), ("y", true), ("xx", false)],
  
  // Nested groups
  "(xy)+": [("", false), ("xy", true), ("xyxy", true), ("x", false)],
  "(x|y)*": [("", true), ("x", true), ("y", true), ("xy", true), ("yx", true), ("xyxy", true)],
  
  // Complex combinations
  "x(y|z)+": [("", false), ("x", false), ("xy", true), ("xz", true), ("xyz", true), ("xyzyz", true)],
  "(ab|cd)*": [("", true), ("ab", true), ("cd", true), ("abcd", true), ("cdab", true), ("abc", false)],
  
  // Multiple alternatives
  "a|b|c": [("", false), ("a", true), ("b", true), ("c", true), ("d", false), ("ab", false)],
  "(x|y)(a|b)": [("xa", true), ("xb", true), ("ya", true), ("yb", true), ("xx", false), ("ab", false)]
]


@Test func nfaToDfa() async throws {

  for (pattern, expectations) in regularCases {
    let n = TestNFA(pattern)
    let d = EquivalentDFA(n)
    for (input, expectedMatch) in expectations {
      #expect(n.recognizes(input) == expectedMatch, "pattern: \(pattern), input: \(input), nfa:\n\(n)")
      #expect(d.recognizes(input) == expectedMatch, "pattern: \(pattern), input: \(input), dfa:\n\(d)")
    }
  }
}

When one of these fails, it of course points to one of the two expect lines; that's not really what I want at all. Plus, I have to add these comments to get useful information out. Maybe there's a different idiom that would work better, in which case document it? Or maybe different library features are needed.

Expected behavior

No response

Actual behavior

No response

Steps to reproduce

No response

swift-testing version/commit hash

No response

Swift & OS version (output of swift --version && uname -a)

No response

[grynspan]

Sorry, but it's not clear what you're asking for here. Please clarify—if there's something actionable here, we can reopen this issue.

[grynspan]

Might #840 help?

[stmontgomery]

I think the feature you're looking for is what we call Parameterized Tests. See https://swiftpackageindex.com/swiftlang/swift-testing/6.0.3/documentation/testing

[grynspan]

Oh hey yeah. I think my mind filled in arguments: [...] automatically.

[dabrahams]

OK, I tried that, but it's not really satisfactory. There are two problems:

1. (minor) I get output like ◇ Passing 1 argument patternAndExpectations → (key: "a|b|c", value: [(input: "", expected: false), (input: "a", expected: true), (input: "b", expected: true), (input: "c", expected: true), (input: "d", expected: false), (input: "ab", expected: false)]) to nfaToDfa(patternAndExpectations:) for each of the cases; that's very noisy!
2. (major) I still have to loop over the expectations for each pattern. The arguments: parameter gives me cross-products of elements, which isn't what I need.

[grynspan]

If you're using swift test, you can pass --quiet to reduce output. In Xcode, look at the test report.

If you don't want all combinations of inputs, you can use zip() or just pass tuples of inputs you want to test—or pass most any other collection, if you prefer. Please review the documentation for parameterized testing for more info.

[grynspan]

(.flatMap may be your friend here.)

[dabrahams]

Exactly how could that help me?

[dabrahams]

BTW, I read the documentation. I don't think you are understanding the problem here. The looping is unavoidable in this case. Clone the project yourself and see if it isn't obvious from GitHub. Once you start looping over things in a test case, the feedback you get about what failed and why is poor, so you end up writing large "comment" parameters to the #expect to try to extract information. That would seem to be the very reason you implemented parameterized testing. But it's only structured to work on cross products of collections, not a collection with a dependent collection for each element (the data structure is a tree).

[grynspan]

If I'm understanding you correctly, you have this:

let regularCases: [String: [(input: String, expected: Bool)]] = [
  "": [("", true), ("x", false), ("xy", false)],
  "x": [("", false), ("x", true), ("xy", false)],
  "x+": [("", false), ("x", true), ("xy", false), ("xx", true)],
  "x*": [("", true), ("x", true), ("xy", false), ("xx", true)],
  "x?": [("", true), ("x", true), ("xy", false), ("xx", false)],
  "x|y": [("", false), ("x", true), ("y", true), ("xx", false)],
]

And you want to transform it into something like [(pattern: String, input: String, expected: Bool)] so that you can process each set of inputs one-at-a-time. That's just flatMap:

let regularCases: [(pattern: String, input: String, expected: Bool)] = [
  "": [("", true), ("x", false), ("xy", false)],
  ...
].flatMap { pattern, inputAndExpected in
  inputAndExpected.map { input, expected in
    (pattern, input, expected)
  }
}

@Test(arguments: regularCases) func myTest(pattern: String, input: String, expected: Bool) {
  ...
}

Feel free to tweak the exact spelling to your liking, of course.

[dabrahams]

I understand what you're suggesting here. The problem is that for each pattern, there's a bunch of preliminary work (and potentially, tests) that shouldn't be repeated for each input/expectation combination.

I can imagine it might be possible to solve this by nesting test functions. Is that supported?

[grynspan]

That pattern would be covered by @stmontgomery's work on scoped traits here.

[dabrahams]

Reading that proposal, I don't see how to apply it. I'd assume the general idea is:

• Establish a scope that does all the preliminary computation for a pattern and stores the expectations (the values in the regularCases dictionary), and within that scope:
  • do some individual tests on the result of that computation, and
  • do a parameterized test over the expectations

But I don't see how to express it. Can you show how that would be done.
It seems to me that to do this at the very least you'd need to parameterize the scope somehow, but I don't see any facility for that. Did I miss it?

This is how I imagine expressing the problem if nested tests were supported:

@Test(arguments: regularCases) func nfaToDfa(pattern: String, expectations: [(input: String, expected: Bool)]) async throws {
  let n = TestNFA(pattern)
  let d = SmallDFA(EquivalentDFA(n)) // small makes it easier to read.
  let m = MinimizedDFA(d)
  #expect(m.states.count <= d.states.count)
  
  @Test(arguments: expectations) func matching(x: (input: String, expectedMatch: Bool)) async throws {
    #expect(n.recognizes(x.input) == x.expectedMatch)
    #expect(d.recognizes(x.input) == x.expectedMatch)
    #expect(m.recognizes(x.input) == x.expectedMatch)
  }
}