Implement a minimal coverage strategy and make it the default

[swankjesse]

Today if you use a lot of Burst, you end up running a lot of tests!

Cartesian Product

A test with N,M parameters runs NxM tests. For example, a test with 3 sizes & 4 sodas runs 12 tests:

	Small	Medium	Large
Pepsi	✅	✅	✅
Coke	✅	✅	✅
Dr Pepper	✅	✅	✅
Root Beer	✅	✅	✅

Adding a choice of 3 distributions triples the number of tests to 36. With Cartesian product strategy, the number of tests multiply.

Minimal Coverage

I’d like to build a new coverage strategy that runs each option at least once.

A test with N,M parameters would run N+M-1 tests. The test above would run 6 tests:

	Small	Medium	Large
Pepsi	✅	✅	✅
Coke		✅
Dr Pepper		✅
Root Beer		✅

Adding a choice of 3 distributions adds only 2 more tests. (All existing tests get distribution=Can, plus Pepsi_Medium_Bottle, Pepsi_Medium_Fountain).

API

I think we express it like this:

@Burst(strategy = CartesianProduct)
class DrinkSodaTest { ... }

@Burst(strategy = MinimalCoverage)
class DrinkSodaTest { ... }

Once implemented I’d also like to explore making this option the default. I’d like to encourage people to Use More Burst and the Cartesian product strategy makes it expensive to do so.

[JakeWharton]

Thinking of all the places I've used parameterization over the years, I have a hard time believing the common case is when the axes of variation do not interact with each other. There definitely are cases where I've only needed minimal coverage as you've defined, but in most cases that I can think of, for me at least, I'm actively seeking the full cartesian product.

I would be pretty upset if manufacturing changed the paper composition of our large cups to save money, but then we only discover (in the field) that Dr Pepper dissolves it in seconds giving customers wet laps and our tests didn't cover large cups with Dr Pepper.

[swankjesse]

Yeah, maybe changing the default is bad.

In our Cash App tests we vary on accessibility text size & dark mode. These are independent!

[JakeWharton]

When I was thinking about my reply above, I actually came up with that as an example of a pair which I thought were independent. But then I convinced myself that sometimes they're not. A larger text size might change the layout such that different subcomponents are selected which may or may not react to theme correctly. I will admit that this is an extremely unlikely scenario, and in this case you could switch the strategy back.

An alternate proposal that I have thought about for no more than a few seconds would be to declare a particular parameter as being @Independent or something.

[oldergod]

So... here's what I think would not be too bad.

/**
 * For parameters with the same [matrixName], Instead of executing all permutations for these
 * parameters, Burst will only, for each parameter of the matrix, execute all permutations of that
 * parameter against the default values of the other parameters of the matrix. If the default value
 * of a parameter is not set, Burst will use the default value of its type (null if nullable, first
 * constant if enum, first element if a list, etc.)
 * */
annotation class MutuallyExclusiveParameter(val matrixName: String = "default")

@Burst
class LocalBrandProfileViewV2Test(
  accessibilityTextSize: AccessibilityTextSize,
  private val modelProvider: ModelProvider,
  @MutuallyExclusiveParameter("design") private val design: DesignTheme = DesignTheme.Light,
  @MutuallyExclusiveParameter("design") private val showMoreOptionsSheet: Boolean,
  @MutuallyExclusiveParameter("layouts") private val showMapDecisionSheet: Boolean,
  @MutuallyExclusiveParameter("layouts") private val hideShareProfileOption: Boolean = false,
  @MutuallyExclusiveParameter("layouts") private val hideNavigationIcon: Boolean,
  @MutuallyExclusiveParameter("layouts") private val withLocationsSheet: Boolean,
) {

[swankjesse]

@oldergod I love your idea of tagging parameters by how they interact.

[swankjesse]

Let me do a proper example to explore the API with. Suppose we’re testing an OkHttp call...

  @Test
  fun callServer( 
    method: String = burstValues("GET", "POST", "HEAD", "DELETE", "PUT"),
    url: String = burstValues("/", "/index.html", "/search?q=burst%20party"),
    protocol: Protocol = burstValues(HTTP_1_1, HTTP_2),
    connectionSpec: ConnectionSpec = burstValues(
      CLEARTEXT, 
      COMPATIBLE_TLS, 
      MODERN_TLS, 
      RESTRICTED_TLS,
    ),
    proxy: Proxy = burstValues(Proxy.NO_PROXY, fakeHttpProxy()),
    body: RequestBody? = burstValues(
      null,
      fakeRequestBody(size=0),
      fakeRequestBody(size=1),
      fakeRequestBody(size=1024 * 1024 * 16),
    ),
  ) { ... }

Firstly, I like our starting point. We get to express a bunch of different features, and it isn’t much work at all to introduce new parameters. For example, if we wanted to add cache set-up here, it’s pretty straightforward to do so.

But this test is basically broken because of how many parameters it specifies. I count 960 different permutations. That’s too many!

And finally there’s tons of ways these parameters interact with each other.

Can we use this sample to workshop an API?

[JakeWharton]

Is a test written like that useful? You probably aren't validating the result all of those axes after each call. So is it just a smoke test?

I'm not denying that having independent variables is useful in some scenarios, but once the cardinality gets that high I question the test a bit. Varying things about the request construction seems independent enough from the transport part that I would think it should be two tests, for example.