NVIDIA 官方大模型推理优化库，在 GPU 上提供极致吞吐与低延迟。
🖥️ Local Inference & Deployment
Ollama, llama.cpp, vLLM: model inference locally and in production
All 1045🧠 Large Language Models 24🤖 AI Agents 162🧩 App Development Platforms 22📚 RAG & Knowledge Bases 48💬 Chat UIs & Clients 40🖥️ Local Inference & Deployment 88👨💻 AI Coding 97🛠️ Frameworks & SDKs 47🔌 MCP & Tool Ecosystem 36🎨 Image Generation 26🎬 Video Generation 17🎙️ Speech & Audio 45👁️ Computer Vision 26📄 Document Parsing & Scraping 25🗄️ Vector Databases 25🏋️ Training, Fine-Tuning & MLOps 71🧪 Evaluation, Observability & Safety 79🚪 API Gateways & Proxies 12🔎 Search & Deep Research 15⚙️ Deep Learning Libraries 58🏷️ Data & Annotation 32🔬 Industry & Science AI 34⚡ Productivity Tools 16
Automation foundation model for tiny devices: 2-bit, 8-29 MB, tool calls, ASR, structured extraction and embeddings on phones, wearables, smart homes, robots, cars and microcontrollers.
Open Machine Learning Compiler Framework
以 OpenAI 兼容接口运行任意开源大模型，并可一键部署到云端。
LMCache: Supercharge Your LLM with the Fastest KV Cache Layer
单文件即可运行的 GGUF 模型工具，自带 Web 界面，适合角色扮演与写作。
NVIDIA 的通用模型推理服务器，支持多框架、多模型并发部署。
OpenVINO™ is an open source toolkit for optimizing and deploying AI inference
Hugging Face 的生产级大模型推理服务，支持张量并行与连续批处理。
llama.cpp 的 Python 绑定，提供 OpenAI 兼容的 Web 服务。
High-speed Large Language Model Serving for Local Deployment
统一部署与服务各类开源大模型、嵌入与多模态模型，一行代码替换 OpenAI。
🤖 AI Gateway | AI Native API Gateway
Open-source, high-performance AI gateway that works as your LLM gateway, MCP gateway and agent gateway. One OpenAI-compatible API for 10,000+ models, with failover, load balancing, semantic caching, virtual keys, budgets
在 Intel CPU / GPU / NPU 上加速运行与微调大模型。
Accessible large language models via k-bit quantization for PyTorch.
A Datacenter Scale Distributed Inference Serving Framework
LMDeploy is a toolkit for compressing, deploying, and serving LLMs.
Fast, flexible LLM inference
在 Mac 上用 MLX 运行与微调大模型的工具包。
A framework for efficient model inference with omni-modality models
Open source inference engine that tunes its kernels on your hardware to find the fastest way to run each model. 2x faster than Ollama. Works on Apple Silicon, NVIDIA, AMD, or just a CPU.
Mooncake is the serving platform for Kimi, a leading LLM service provided by Moonshot AI.
LiteRT-LM is Google's production-ready, high-performance, open-source inference framework for deploying Large Language Models on edge devices.