Incompatibile dill version (0.3.9) in datasets 2.18.0 - 3.5.0

[JGrel]

Describe the bug

Datasets 2.18.0 - 3.5.0 has a dependency on dill < 0.3.9. This causes errors with dill >= 0.3.9.

Could you please take a look into it and make it compatible?

Steps to reproduce the bug

1. Install setuptools >= 2.18.0
2. Install dill >=0.3.9
3. Run pip check
4. Output:
   ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
   datasets 2.18.0 requires dill<0.3.9,>=0.3.0, but you have dill 0.3.9 which is incompatible.

Expected behavior

Pip install both libraries without any errors

Environment info

Datasets version: 2.18 - 3.5
Python: 3.11

[lhoestq]

Hi ! We can bump dill to 0.3.9 if we make sure it's deterministic and doesn't break the caching mechanism in datasets.

Would you be interested in opening a PR ? Then we can run the CI to see if it works

[JGrel]

Hi!. Yeah I can do it. Should I make any changes besides dill versions?

[lhoestq]

There are probably some usage of internal functions from dill that we'll need to update in datasets

If you run pytest tests/test_fingerprint.py you should already have a good idea of what works and what doesn't.
But feel free to open a PR anyway, this way we can run the full CI and see the results

[JGrel]

Hi, sorry for no response from my side. I will try to do it today.

[JGrel]

Created pull request: LINK
Tried to run tests by using command you have send and got few errors: