New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Qwen2.5-Code-Instruct Performance without DPO #357

Open

DarrenRuan opened this issue Jan 8, 2025 · 0 comments

DarrenRuan commented Jan 8, 2025

Hi Qwen team, Thanks for your great work, which is really helpful!

However, I have some questions:

Have you tested the model performance only with SFT, i.e., without DPO? E.g., Qwen2.5-Code-SFT (SFT only) vs. Qwen2.5-Code-Instruct (SFT + DPO)?
Decontamination. Ideally speaking, the test repos or the test benchmark should be excluded from the training set. In your paper, it is mentioned that:

10-gram overlap method, where any training data with a 10-gram word-level overlap with the test data was removed.

Is there any chance 10-gram approach still has some leakage?

DarrenRuan changed the title ~~Performance with only SFT without DPO~~ Qwen2.5-Code-Instruct Performance without DPO

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment