FlashInfer: Kernel Library for LLM Serving
🖥️ 本地推理与部署
Ollama、llama.cpp、vLLM：本地与生产环境的模型推理
全部 1047🧠 大语言模型 25🤖 AI 智能体 164🧩 应用开发平台 23📚 RAG 与知识库 46💬 聊天界面与客户端 42🖥️ 本地推理与部署 87👨💻 AI 编程 97🛠️ 开发框架与 SDK 48🔌 MCP 与工具生态 36🎨 图像生成 26🎬 视频生成 18🎙️ 语音与音频 44👁️ 计算机视觉 26📄 文档解析与抓取 24🗄️ 向量数据库 25🏋️ 训练微调与 MLOps 76🧪 评测·观测·安全 80🚪 API 网关与代理 12🔎 搜索与深度研究 13⚙️ 深度学习基础库 54🏷️ 数据与标注 31🔬 行业与科研 AI 36⚡ 效率工具 14
⚡ Pure-Rust WebGPU inference engine — OpenAI-API compatible, GGUF native, runs on any GPU. No Python. No llama.cpp. Single binary.
Reliable model swapping for any local OpenAI/Anthropic compatible server - llama.cpp, vllm, etc
A GPU cluster manager for high-performance AI model serving (vLLM, SGLang) and on-demand SSH-accessible GPU instances.
A unified library of SOTA model optimization techniques like quantization, distillation, pruning, neural architecture search, speculative decoding, etc. It compresses deep learning models for downstream deployment framew
A compact implementation of SGLang, designed to demystify the complexities of modern LLM serving systems.
Cost-efficient and pluggable Infrastructure components for GenAI inference
On-device AI across mobile, embedded and edge for PyTorch
Achieve state of the art inference performance with modern accelerators on Kubernetes
Fast inference engine for Transformer models
TNN: developed by Tencent Youtu Lab and Guangying Lab, a uniform deep learning inference framework for mobile、desktop and server. TNN is distinguished by several outstanding features, including its cross-platform capabil
LightLLM is a Python-based LLM (Large Language Model) inference and serving framework, notable for its lightweight design, easy scalability, and high-speed performance.
Optimizing inference proxy for LLMs
Rapid-MLX is an open-source (Apache 2.0) OpenAI- and Anthropic-compatible LLM inference server for Apple Silicon, built on MLX, focused on reliable tool calling for coding agents. Up to 4× faster than Apple's MLX (mlx-lm
A minimal Python framework for building custom AI inference servers with full control over logic, batching, and scaling.
State-of-the-art LLM compression, built for production inference with vLLM
🚀 Accelerate inference and training of 🤗 Transformers, Diffusers, TIMM and Sentence Transformers with easy to use hardware optimization tools
Switchyard lets LLM applications route traffic across models and providers while preserving native OpenAI and Anthropic API compatibility - enabling flexible model selection, benchmarking, and cost/performance optimizati
Run Claude Code 100% on-device with local AI on Apple Silicon. MLX-native Anthropic-API server. 6 fighters incl. Muse-Glimmer 30B (now multimodal — reads images, abliterated), Gemma 4 31B, Qwen 3.5 122B (65 tok/s), DeepS
Stop configuring your AI stack. Start using it. One command brings a complete pre-wired LLM stack with hundreds of services to explore.
Distributed LLM inference. Connect home devices into a powerful cluster to accelerate LLM inference. More devices means faster inference.
RamaLama is an open-source developer tool that simplifies the local serving of AI models from any source and facilitates their use for inference in production, all through the familiar language of containers.
Infinity is a high-throughput, low-latency serving engine for text-embeddings, reranking models, clip, clap and colpali
Community maintained hardware plugin for vLLM on Huawei Ascend