Ensure accuracy of compat data for 3 engines from 2020 onwards

[Elchi3]

What would you like to see added to BCD?

I'd like to propose this issue to be a tracking issue similar to #3555.

With every release of BCD, I'd like a table to be generated which is added as a comment to this issue.

It would look something like this:

Status as of BCD version 5.2.43 (released on 2023-03-17) for web all platform features. Powered by BCD collector v8.0.0 changes (Feb. 21, 2023).

BCD and the collector agreement for browsers releases 2020 onwards:

browser	agreement	disagreement
All	xx.xx%	xx.xx%
Chrome (Chromium)	xx.xx%	xx.xx%
Firefox (Gecko)	xx.xx%	xx.xx%
Safari (WebKit)	xx.xx%	xx.xx%

Link to pull request showing differences between BCD and the collector.

Additionally, I propose we have a draft PR that stays open. You can always look at the latest state of BCD and collector differences. It would basically be #18977 but always open and whenever there is a BCD release you force push to it. (Maybe the statistics table can be generated from that PR, or there other simpler ways of doing it.)

cc @foolip, @queengooborg

How impactful do you think this enhancement will be?

It would help Open Web Docs to track our goal: openwebdocs/project#85

Note that we're not interested in pre-2020 data. If you want to track those, I would propose to record that in a different tracking issue.

Do you have anything more you want to share?

No response

[foolip]

Regular tracking like this was incredibly effective when driving down the null and true values, so I think this is a great idea.

@queengooborg do you think these stats could be produced from existing scripts, or what would it take?

[gsnedders]

I'd like to also see an "unknown" column, showing the percentage of items the collector doesn't have data for.

[queengooborg]

The easiest way is probably to track the git diff, but that only gives us a way to track the changes that the collector wants to make, and doesn't really give us insight for things like features the collector doesn't track.

(I'd imagine that the code we develop will also aid in #6863 and the improved changelog idea.)

[Elchi3]

I'd like to also see an "unknown" column, showing the percentage of items the collector doesn't have data for.

Does "doesn't have data for" mean the same as "collector has no test code for a feature"?

Maybe we should somehow include that in the report format (foolip/mdn-bcd-collector#2759) so that we could have stats about it?

[queengooborg]

I don't think we need to include that data in the reports because that'd only make the files bigger than they need to be, especially if the tests are intentionally limited to a specific scope. We already have a file generated with all our tests and test code, so we should probably just use that.

[gsnedders]

I'd like to also see an "unknown" column, showing the percentage of items the collector doesn't have data for.

Does "doesn't have data for" mean the same as "collector has no test code for a feature"?

Maybe we should somehow include that in the report format (foolip/mdn-bcd-collector#2759) so that we could have stats about it?

Conservatively, yes.

We could be more aggressive and say that null values also count against that, but that's much harder to achieve (for many reasons, especially when there are dependencies).

[queengooborg]

The OWD goal has been completed, and the similar goals have been low priority in comparison to our other goals. Is this issue still relevant?

[Elchi3]

I think we can close