llms-learning 📚 🦙

A repository sharing the literatures and resources about Large Language Models (LLMs) and beyond.

Hope you find this repository useful! 😊

News 🔥

• 2024.10.24
  • Welcome to watch our new online free LLMs intro course on bilibili!
  • We also open-source and keep updating the course assignments for you to take a deep dive into LLMs.
  • If you like this course or this repository, you can subscribe to the teacher's bilibili account and maybe ⭐ this GitHub repo 😜.
• 2024.03.07
  • We offer a comprehensive notebook tutorial on efficient GPU kernel coding using Triton, building upon the official tutorials and extending them with additional hands-on examples, such as the Flash Attention 2 forward/backward kernel.
  • In addition, we also provide a step-by-step math derivation of Flash Attention 2, enabling a deeper understanding of its underlying mechanics.

Table of Contents

• Tutorials
  • Notebooks
  • LLMs Intro Course Assignments
• Development Stages
  • Modeling
    • Language Models
      • Transformer
        • Attention
          • Full Attention
          • Sparse Attention
          • Linear Attention
        • Dense
        • Mixture of Experts (MoE)
        • Normalization
        • Position Embedding
        • Activation Functions
      • State Space Models (SSM)
      • Kolmogorov-Arnold Networks (KAN)
      • Miscellaneous
    • Multi-Modal Language Models
      • Text <-> Image
      • Text <-> Audio
      • Text <-> Video
      • Text <-> Omni
  • Serving
    • Inference
    • Quantization
    • Pruning
    • Deployment
    • Evaluation
  • Training
    • Pre-Training
      • General Training Recipes
      • Mixed-Precision Training
      • Heterogenous Training
      • Parallelism
        • Integration of Parallelism (IP)
        • Data Parallelism (DP)
        • Tensor Parallelism (TP)
        • Pipeline Parallelism (PP)
        • Context Parallelism (CP)
        • Expert Parallelism (EP)
        • Automatic Parallelism (AP)
        • Distributed Communication
      • Memory Management
        • Recomputation / Activation Checkpointing
        • Offloading
        • Checkpointing
        • Device Placement
      • Optimizer
      • Objectives / Loss Functions
    • Fine-Tuning
      • Efficient Fine-Tuning (EFT)
      • Instruction Fine-Tuning (IFT)
    • Alignment
      • Alignment
• Applications
  • Retrieval-Augmented Generation (RAG)
  • LLMs as Agents
  • LLMs for Autonomous Driving
  • LLMs for Code
  • LLMs for Math
• Study
  • Emergence Abilities
  • Embodied Intelligence
  • Robustness
  • Transferability
  • In-Context Learning (ICL)
  • Long-Context Capabilities
  • General Empirical Study
  • General Surveys
• Basics
• Assets

---

Note:

• Each markdown file contains collected papers roughly sorted by published year in descending order; in other words, newer papers are generally placed at the top. However, this arrangement is not guaranteed to be completely accurate, as the published year may not always be clear.

• The taxonomy is complex and not strictly orthogonal, so don't be surprised if the same paper appears multiple times under different tracks.