add Sequence Parallelism (reopened #6506)

[HaoshengZou]

What does this PR do?

add Sequence Parallelism (reopened #6506)

Before submitting

• [√] Did you read the contributor guideline?
• [√] Did you write any new necessary tests?

[hiyouga]

Nice work, we expect to add this feature to LlamaFactory v1.0

[xiaosu-zhu]

#7335 Has a new implementation with Deepspeed Ulysses, please consider it as an alternative solution.

[HaoshengZou]

@xiaosu-zhu Thanks for pointing it out!

Deepspeed Ulysses is always on our ToDo list and we have already integrated it recently and now pushed here, supporting --sequence_parallel_mode zigzag-ring / ulysses.
Correctness is thoroughly verified and training speed tested similarly here.
Our Ulysses integration is based on yunchang for now but could be stripped with core functions adapted and source-acknowledged.

This PR reopened #6506, which was opened on Jan.2 and accidentally closed last week #6506 (comment).

#7335 may represent alternative nice ways to split SP data and monkey patching. Still, to get SP 100% correct requires much more detailed work in dealing with padding, labels and loss computation etc. (as we've discussed Swift here), followed by thorough testing.

We are confident that our implementation is 100% correct up to numerical differences inherent to SP, and our implementation bears close-to-minimal modular code change to original LLaMA-Factory.

[zhijie-reallm]

does it support DPO?

[HaoshengZou]

@zhijie-reallm yeah