button on GitHub screen for filing issue for test failure

[davepacheco]

It would be handy if the GitHub summary for a failed run provided a button to automatically file an issue on the repo. I hope this would make it much easier for people to file issues for flaky tests in CI.

More concretely, on this screen:
https://github.com/oxidecomputer/omicron/runs/20443797017

It'd be neat to have a button that would file an issue for this failure. Here's an example issue:
oxidecomputer/omicron#4590

The issue has:

• a link to the GitHub screen summarizing the result. This lets someone quickly find the commit and PR, if applicable. (This could happen on a commit on "main" or a commit on a PR branch.)
• a link to the full buildomat log (probably not as critical, since you can get it from the link above, but maybe useful to save people a click)
• ideally: a pasted excerpt from the log that summarizes the failure. I find this extremely valuable but I'm not sure how best to do it. If stderr is pretty concise on these failures, great. Otherwise we could maybe use some heuristics from cargo nextest output? Or maybe there's already a parseable form of output?

We might also consider a link to search for the same issue already having been filed.

[sunshowers]

Otherwise we could maybe use some heuristics from cargo nextest output? Or maybe there's already a parseable form of output?

For parseable output, there are two current options:

1. Use the JUnit output. JUnit is generally quite widely supported (though support in Rust is a bit iffy -- for nextest I wrote https://crates.io/crates/quick-junit which doesn't support deserialization at the moment).
2. There's an experimental feature for libtest JSON output. This is still in flux somewhat -- it's relatively untested.

[davepacheco]

👍

Also a lot of CI failures aren't Rust test failures (e.g., oxidecomputer/omicron#4085, oxidecomputer/omicron#2689, oxidecomputer/omicron#4165) and many of our projects don't use nextest so I imagine we'll want some kind of heuristic fallback anyway.

[jclulow]

I suspect producing a repository-specific excerpt will ultimately involve some programmable logic specified in the repository, so I might defer that part initially. And it might end up being something like: you specify a rhai script that we'll run in a safe context, and for a limited duration, and give you access to data from the job so you can produce your own summary.