feat: add new HanLP integration with ChineseDocumentSplitter

[julian-risch]

Related Issues

• Addresses parts of New HanLP integration with ChineseDocumentSplitter #1944

Proposed Changes:

• This PR introduces a ChineseDocumentSplitter that supports accurate sentence and paragraph splitting for Chinese documents. It leverages the HanLP library for Chinese linguistic analysis, including sentence segmentation and tokenization.
• It keeps the commit history from drafts created in the haystack repository: feature: Chinese DocumentSplitter haystack#9453 and feat: Add ChineseDocumentSplitter haystack#9494
• In addition, there is a warm_up method loading the models, support for English language is removed, and ChineseDocumentSplitter is no longer inheriting from the DocumentSplitter
• Similar to the findings and changes of the following PR, we skip tests for the combination of Windows with python 3.13 because of an incompatibility with the sentence-piece dependency chore: stop testing instructor-embedders on windows + python 3.13 #1941
• There is a Github workflow running the tests for the integration nightly and at every PR
• The labeler.yml file has been updated

How did you test it?

We should test with this notebook. It shows how to use the new component in the forked repository: https://github.com/mc112611/haystack/blob/307f8340b2e1a9104efe4e33d8c1885d17143c36/examples/chinese_RAG_test_haystack_chinese.ipynb

Notes for the reviewer

Before this can be reviewed we need to work on:

• Current implementation of ChineseDocumentSplitter is inheriting from the DocumentSplitter. That's probably not needed.
• We can remove some parts that are only needed to handle English, for example self.language == english
• We need to define warm_up in ChineseDocumentSplitter, which should load external data
• We need to add a usage example to the component docstring
• py.typed needs to be added
• Tests. Their should be proper unit tests and only a limited number of integration tests. Similar to NLTK (needs to download extra data) right now all tests are integrations tests. we should change that.

I had a look at the other tokenizers that HanLP supports. All of them seem to be worse than the two tokenizers that we support in this integration. Therefore, I'd limit the user's options to just the two. https://hanlp.hankcs.com/docs/api/hanlp/pretrained/tok.html

Checklist

• I have read the contributors guidelines and the code of conduct
• I have updated the related issue with new insights and changes
• I added unit tests and updated the docstrings
• I've used one of the conventional commit types for my PR title: fix:, feat:, build:, chore:, ci:, docs:, style:, refactor:, perf:, test:.