Internal feature request: extend fuzzer ruff_fix_validity to non-default linter settings

[dylwil3]

The fuzzer looks very cool and I'd love to use it!

When adding new preview rules, it would help catch edge cases faster if we were able to fuzz test against them. My understanding of the fuzzer documentation (which could be wrong!) is that it is not currently possible to do this.

Related to #4972.

[dhruvmanila]

Yes, that's correct. The fuzzer uses the default linter settings:

ruff/fuzz/fuzz_targets/ruff_fix_validity.rs

Lines 18 to 19 in 1f51048

	// the settings are immutable to test_snippet, so we avoid re-initialising here
	let settings = SETTINGS.get_or_init(LinterSettings::default);

Is the feature request to manually create a config which the fuzzer uses? I think this would be really useful. While I was working on the parser, I manually updated the default settings in the code to reflect the config I wanted.

We could either use a command-line flag (if it's even possible) or an environment variable (RUFF_FUZZ_CONFIG).

[dylwil3]

Is the feature request to manually create a config which the fuzzer uses?

Yes, exactly!

The tasks it would be nice to easily perform would be:

1. Fuzz with all (compatible) rules, including preview rules, turned on.
2. Fuzz with just a single new rule turned on.
3. (1) and (2) but with different target version.
4. (1) and (2) but iterated through all possible (or maybe randomly generated if the set is prohibitively large) auxiliary settings.

I'm not super familiar with cargo fuzz, but it seems like (1) would be the simplest to implement because it could just be a separate fuzzer (alternatively, we could replace the default linter settings in ruff_fix_validity with this setting).

The remaining ones seem more difficult... could this section of the docs be relevant, or am I not understanding it very well?

I'm agnostic about whether this is callable via a command-line flag, environment variable, or even a reference to a ruff.toml file.

[dhruvmanila]

I'm agnostic about whether this is callable via a command-line flag, environment variable, or even a reference to a ruff.toml file.

The reason I suggested a command-line flag or an environment variable is because we don't run any fuzzer as part of the CI. It needs to be run locally. This means it might be more useful (and easier?) to just read a local config file which would be git-ignored but can be used to run the fuzzer for a limited time locally to catch any bugs if the contributor wants to.