transfer.md

Transferability of LLMs

Here're some resources about transferability of LLMs, including (De)composition, Routing, Fusion / Stacking / Ensembling, and even Unlearning

Ensembling

LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Composition

paper link: here

github link: here

hfhub link: here

citation:

@misc{huang2024lorahub,
      title={LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Composition}, 
      author={Chengsong Huang and Qian Liu and Bill Yuchen Lin and Tianyu Pang and Chao Du and Min Lin},
      year={2024},
      eprint={2307.13269},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}

Routing

RouteLLM: Learning to Route LLMs with Preference Data

paper link: here

github link: here

citation:

@misc{ong2024routellmlearningroutellms,
      title={RouteLLM: Learning to Route LLMs with Preference Data}, 
      author={Isaac Ong and Amjad Almahairi and Vincent Wu and Wei-Lin Chiang and Tianhao Wu and Joseph E. Gonzalez and M Waleed Kadous and Ion Stoica},
      year={2024},
      eprint={2406.18665},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2406.18665}, 
}

Unlearning

Large Language Model Unlearning

paper link: here

citation:

@misc{yao2023large,
      title={Large Language Model Unlearning}, 
      author={Yuanshun Yao and Xiaojun Xu and Yang Liu},
      year={2023},
      eprint={2310.10683},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}