NVIDIA 官方大模型推理优化库，在 GPU 上提供极致吞吐与低延迟。
🖥️ Inferência e implantação local
Ollama, llama.cpp, vLLM: inferência de modelos local e em produção
Todos 1045🧠 Grandes modelos de linguagem 24🤖 Agentes de IA 162🧩 Plataformas de desenvolvimento de apps 22📚 RAG e bases de conhecimento 48💬 Interfaces de chat e clientes 40🖥️ Inferência e implantação local 88👨💻 Programação com IA 97🛠️ Frameworks e SDKs 47🔌 MCP e ecossistema de ferramentas 36🎨 Geração de imagens 26🎬 Geração de vídeo 17🎙️ Voz e áudio 45👁️ Visão computacional 26📄 Análise de documentos e scraping 25🗄️ Bancos de dados vetoriais 25🏋️ Treinamento, ajuste fino e MLOps 71🧪 Avaliação, observabilidade e segurança 79🚪 Gateways e proxies de API 12🔎 Busca e pesquisa profunda 15⚙️ Bibliotecas de deep learning 58🏷️ Dados e anotação 32🔬 IA para indústria e ciência 34⚡ Ferramentas de produtividade 16
Automation foundation model for tiny devices: 2-bit, 8-29 MB, tool calls, ASR, structured extraction and embeddings on phones, wearables, smart homes, robots, cars and microcontrollers.
Open Machine Learning Compiler Framework
以 OpenAI 兼容接口运行任意开源大模型，并可一键部署到云端。
LMCache: Supercharge Your LLM with the Fastest KV Cache Layer
单文件即可运行的 GGUF 模型工具，自带 Web 界面，适合角色扮演与写作。
NVIDIA 的通用模型推理服务器，支持多框架、多模型并发部署。
OpenVINO™ is an open source toolkit for optimizing and deploying AI inference
Hugging Face 的生产级大模型推理服务，支持张量并行与连续批处理。
llama.cpp 的 Python 绑定，提供 OpenAI 兼容的 Web 服务。
High-speed Large Language Model Serving for Local Deployment
统一部署与服务各类开源大模型、嵌入与多模态模型，一行代码替换 OpenAI。
🤖 AI Gateway | AI Native API Gateway
Open-source, high-performance AI gateway that works as your LLM gateway, MCP gateway and agent gateway. One OpenAI-compatible API for 10,000+ models, with failover, load balancing, semantic caching, virtual keys, budgets
在 Intel CPU / GPU / NPU 上加速运行与微调大模型。
Accessible large language models via k-bit quantization for PyTorch.
A Datacenter Scale Distributed Inference Serving Framework
LMDeploy is a toolkit for compressing, deploying, and serving LLMs.
Fast, flexible LLM inference
在 Mac 上用 MLX 运行与微调大模型的工具包。
A framework for efficient model inference with omni-modality models
Open source inference engine that tunes its kernels on your hardware to find the fastest way to run each model. 2x faster than Ollama. Works on Apple Silicon, NVIDIA, AMD, or just a CPU.
Mooncake is the serving platform for Kimi, a leading LLM service provided by Moonshot AI.
LiteRT-LM is Google's production-ready, high-performance, open-source inference framework for deploying Large Language Models on edge devices.