Production ready LLM model compression/quantization toolkit with accelerated inference support for both cpu/gpu via HF, vLLM, and SGLang.

☸️ Easy, advanced inference platform for large language models on Kubernetes. 🌟 Star to support our work!

AI-based search engine done right

A guide to structured generation using constrained decoding

Controllable Language Model Interactions in TypeScript

Kernel Library Wheel for SGLang

SgLang vs vLLM Comparison

Examples of serving LLM on Modal.

llmd is a LLMs daemonset, it provide model manager and get up and running large language models, it can use llama.cpp or vllm or sglang to running large language models.