CI: Seperate out tests that fail due to HTTP calls

[KennethEnevoldsen]

Currently, the test suite sometimes fails due to an error outside the scope of mteb (e.g. on HF). This makes it hard to trust when the test fails. A solution to this would be to simply factor out those tests in a separate CI (such that if the main tests fail it is clear easy to trust the error)

@Samoed and @isaac-chung WDYT about this?

thread on how to do this in pytest for reference

[Samoed]

Yes, we can separete integration tests

[sam-hey]

This might also be a solution: https://github.com/pytest-dev/pytest-rerunfailures

One argument in favor of this fix is that cases not covered otherwise might be missed and it is less work.

pytest --reruns 5 --only-rerun AssertionError --only-rerun ValueError --reruns-delay 1

[KennethEnevoldsen]

pytest --reruns 5 --only-rerun AssertionError --only-rerun ValueError --reruns-delay 1

Hmm, but this could also hide an error that happens by chance.

[sam-hey]

I don't think that AssertionError and ValueError are merely placeholders for a ConnectionError or any other temporary issue. Since this error should disappear after a short time, rerunning the test with --reruns-delay xxxx should resolve the issue. However, if the problem persists, it might indicate a deeper issue that should be addressed by a developer as it will fail also after the reruns.