Could you please release the processed pretraining data?

[phellonchen]

No description provided.

[StevenTang1998]

You can download them at the link: https://huggingface.co/RUCAIBox. Since some datasets have license limitations, we cannot merge them into one dataset. You can merge them by your own.

[phellonchen]

Thanks. One more question, where can I find the code about a temperature-scaled mixing strategy (Raffel et al., 2020) with a rate of T = 2 to mitigate the disparity in tasks and datasets ? I have not found it in https://github.com/RUCAIBox/TextBox.

[StevenTang1998]

The general code of pre-training is still under developping. For pre-training MVP, we just conducted the temperature-scaled mixing strategy by copying instances. You can also use it as a simple alternative.
For example, A dataset has 2 instances and B dataset has 8 instances. We merge them into a unified datasest with the temperature-scaled mixing strategy by doubling the instances in A dataset.