feat(pytest/plugins): Fork covariant parametrize

[marioevz]

🗒️ Description

This PR introduces a new decorator called fork_covariant_parametrize.

The goal of the decorator is to be able to dynamically parametrize a test depending on which fork the tests are being filled.

This is done using a custom function that is passed to the decorator, which will then be executed once per fork that the test is being filled with.

It allows to dynamically create a different amount of test cases per fork (Useful for EIP-4844 tests which are being modified by fork now).

Example

import pytest

from pytest_plugins import fork_covariant_parametrize

def covariant_function(fork):
    return [[1, 2], [3, 4]] if fork.name() == "Paris" else [[4, 5], [5, 6], [6, 7]]

@fork_covariant_parametrize(parameter_names=[
    "test_parameter", "test_parameter_2"
], fn=covariant_function)
@pytest.mark.valid_from("Paris")
@pytest.mark.valid_until("Shanghai")
def test_case(state_test_only, test_parameter, test_parameter_2):
    pass

In this example, the test will be parametrized with the values [1, 2] and [3, 4] for the Paris fork, with values 1 and 3 being assigned to test_parameter and 2 and 4 being assigned to test_parameter_2.

For the Shanghai fork, the test will be parametrized with the values [4, 5], [5, 6], and [6, 7].

Therefore, more test cases will be generated for the Shanghai fork.

🔗 Related Issues

✅ Checklist

• All: Set appropriate labels for the changes.
• All: Considered squashing commits to improve commit history.
• All: Added an entry to CHANGELOG.md.
• All: Considered updating the online docs in the ./docs/ directory.
• Tests: All converted JSON/YML tests from ethereum/tests have been added to converted-ethereum-tests.txt.
• Tests: A PR with removal of converted JSON/YML tests from ethereum/tests have been opened.
• Tests: Included the type and version of evm t8n tool used to locally execute test cases: e.g., ref with commit hash or geth 1.13.1-stable-3f40e65.
• Tests: Ran mkdocs serve locally and verified the auto-generated docs for new tests in the Test Case Reference are correctly formatted.

[marioevz]

To give an example of the usefulness of this, when parametrizing the valid blob tests using this feature, on Cancun with vanilla 4844 we get the 87 test cases (not showing complete list here):

...
    <Function test_valid_blob_tx_combinations[fork_Cancun-blobs_per_tx_(6,)-blockchain_test]>
    <Function test_valid_blob_tx_combinations[fork_Cancun-blobs_per_tx_(6,)-blockchain_test_engine]>
    <Function test_valid_blob_tx_combinations[fork_Cancun-blobs_per_tx_(2, 1, 1, 1, 1)-blockchain_test]>
    <Function test_valid_blob_tx_combinations[fork_Cancun-blobs_per_tx_(2, 1, 1, 1, 1)-blockchain_test_engine]>
    <Function test_valid_blob_tx_combinations[fork_Cancun-blobs_per_tx_(3, 1, 1, 1)-blockchain_test]>
    <Function test_valid_blob_tx_combinations[fork_Cancun-blobs_per_tx_(3, 1, 1, 1)-blockchain_test_engine]>
    <Function test_valid_blob_tx_combinations[fork_Cancun-blobs_per_tx_(2, 2, 1, 1)-blockchain_test]>
    <Function test_valid_blob_tx_combinations[fork_Cancun-blobs_per_tx_(2, 2, 1, 1)-blockchain_test_engine]>
    <Function test_valid_blob_tx_combinations[fork_Cancun-blobs_per_tx_(4, 1, 1)-blockchain_test]>
    <Function test_valid_blob_tx_combinations[fork_Cancun-blobs_per_tx_(4, 1, 1)-blockchain_test_engine]>
    <Function test_valid_blob_tx_combinations[fork_Cancun-blobs_per_tx_(3, 2, 1)-blockchain_test]>
    <Function test_valid_blob_tx_combinations[fork_Cancun-blobs_per_tx_(3, 2, 1)-blockchain_test_engine]>
    <Function test_valid_blob_tx_combinations[fork_Cancun-blobs_per_tx_(5, 1)-blockchain_test]>
    <Function test_valid_blob_tx_combinations[fork_Cancun-blobs_per_tx_(5, 1)-blockchain_test_engine]>
    <Function test_valid_blob_tx_combinations[fork_Cancun-blobs_per_tx_(4, 2)-blockchain_test]>
    <Function test_valid_blob_tx_combinations[fork_Cancun-blobs_per_tx_(4, 2)-blockchain_test_engine]>
...

On prague with EIP-7691 and EIP-7762 enabled we get 338 test cases (again not showing complete list here):

...
    <Function test_valid_blob_tx_combinations[fork_Prague-blobs_per_tx_(9,)-blockchain_test]>
    <Function test_valid_blob_tx_combinations[fork_Prague-blobs_per_tx_(9,)-blockchain_test_engine]>
    <Function test_valid_blob_tx_combinations[fork_Prague-blobs_per_tx_(2, 1, 1, 1, 1, 1, 1, 1)-blockchain_test]>
    <Function test_valid_blob_tx_combinations[fork_Prague-blobs_per_tx_(2, 1, 1, 1, 1, 1, 1, 1)-blockchain_test_engine]>
    <Function test_valid_blob_tx_combinations[fork_Prague-blobs_per_tx_(3, 1, 1, 1, 1, 1, 1)-blockchain_test]>
    <Function test_valid_blob_tx_combinations[fork_Prague-blobs_per_tx_(3, 1, 1, 1, 1, 1, 1)-blockchain_test_engine]>
    <Function test_valid_blob_tx_combinations[fork_Prague-blobs_per_tx_(2, 2, 1, 1, 1, 1, 1)-blockchain_test]>
    <Function test_valid_blob_tx_combinations[fork_Prague-blobs_per_tx_(2, 2, 1, 1, 1, 1, 1)-blockchain_test_engine]>
    <Function test_valid_blob_tx_combinations[fork_Prague-blobs_per_tx_(4, 1, 1, 1, 1, 1)-blockchain_test]>
    <Function test_valid_blob_tx_combinations[fork_Prague-blobs_per_tx_(4, 1, 1, 1, 1, 1)-blockchain_test_engine]>
    <Function test_valid_blob_tx_combinations[fork_Prague-blobs_per_tx_(3, 2, 1, 1, 1, 1)-blockchain_test]>
    <Function test_valid_blob_tx_combinations[fork_Prague-blobs_per_tx_(3, 2, 1, 1, 1, 1)-blockchain_test_engine]>
    <Function test_valid_blob_tx_combinations[fork_Prague-blobs_per_tx_(2, 2, 2, 1, 1, 1)-blockchain_test]>
    <Function test_valid_blob_tx_combinations[fork_Prague-blobs_per_tx_(2, 2, 2, 1, 1, 1)-blockchain_test_engine]>
    <Function test_valid_blob_tx_combinations[fork_Prague-blobs_per_tx_(5, 1, 1, 1, 1)-blockchain_test]>
    <Function test_valid_blob_tx_combinations[fork_Prague-blobs_per_tx_(5, 1, 1, 1, 1)-blockchain_test_engine]>
    <Function test_valid_blob_tx_combinations[fork_Prague-blobs_per_tx_(4, 2, 1, 1, 1)-blockchain_test]>
    <Function test_valid_blob_tx_combinations[fork_Prague-blobs_per_tx_(4, 2, 1, 1, 1)-blockchain_test_engine]>
    <Function test_valid_blob_tx_combinations[fork_Prague-blobs_per_tx_(3, 3, 1, 1, 1)-blockchain_test]>
    <Function test_valid_blob_tx_combinations[fork_Prague-blobs_per_tx_(3, 3, 1, 1, 1)-blockchain_test_engine]>
    <Function test_valid_blob_tx_combinations[fork_Prague-blobs_per_tx_(3, 2, 2, 1, 1)-blockchain_test]>
    <Function test_valid_blob_tx_combinations[fork_Prague-blobs_per_tx_(3, 2, 2, 1, 1)-blockchain_test_engine]>
    <Function test_valid_blob_tx_combinations[fork_Prague-blobs_per_tx_(2, 2, 2, 2, 1)-blockchain_test]>
...

And both are generated using the same test function and same fork-covariant test case generator, but the different values returned from fork.max_blobs_per_block() is what causes the test case increment.