FlashInfer: Kernel Library for LLM Serving
🖥️ Inferencia y despliegue local
Ollama, llama.cpp, vLLM: inferencia de modelos en local y en producción
Todo 1045🧠 Modelos de lenguaje 24🤖 Agentes de IA 162🧩 Plataformas de desarrollo de apps 22📚 RAG y bases de conocimiento 48💬 Interfaces de chat y clientes 40🖥️ Inferencia y despliegue local 88👨💻 Programación con IA 97🛠️ Frameworks y SDK 47🔌 MCP y ecosistema de herramientas 36🎨 Generación de imágenes 26🎬 Generación de vídeo 17🎙️ Voz y audio 45👁️ Visión por computador 26📄 Análisis de documentos y scraping 25🗄️ Bases de datos vectoriales 25🏋️ Entrenamiento, ajuste fino y MLOps 71🧪 Evaluación, observabilidad y seguridad 79🚪 Pasarelas y proxies de API 12🔎 Búsqueda e investigación profunda 15⚙️ Bibliotecas de aprendizaje profundo 58🏷️ Datos y anotación 32🔬 IA para industria y ciencia 34⚡ Herramientas de productividad 16
Reliable model swapping for any local OpenAI/Anthropic compatible server - llama.cpp, vllm, etc
⚡ Pure-Rust WebGPU inference engine — OpenAI-API compatible, GGUF native, runs on any GPU. No Python. No llama.cpp. Single binary.
LLocalSearch is a completely locally running search aggregator using LLM Agents. The user can ask a question and the system will use a chain of LLMs to find the answer. The user can see the progress of the agents and the
A GPU cluster manager for high-performance AI model serving (vLLM, SGLang) and on-demand SSH-accessible GPU instances.
A unified library of SOTA model optimization techniques like quantization, distillation, pruning, neural architecture search, speculative decoding, etc. It compresses deep learning models for downstream deployment framew
A compact implementation of SGLang, designed to demystify the complexities of modern LLM serving systems.
Cost-efficient and pluggable Infrastructure components for GenAI inference
On-device AI across mobile, embedded and edge for PyTorch
Achieve state of the art inference performance with modern accelerators on Kubernetes
Fast inference engine for Transformer models
TNN: developed by Tencent Youtu Lab and Guangying Lab, a uniform deep learning inference framework for mobile、desktop and server. TNN is distinguished by several outstanding features, including its cross-platform capabil
LightLLM is a Python-based LLM (Large Language Model) inference and serving framework, notable for its lightweight design, easy scalability, and high-speed performance.
Optimizing inference proxy for LLMs
Rapid-MLX is an open-source (Apache 2.0) OpenAI- and Anthropic-compatible LLM inference server for Apple Silicon, built on MLX, focused on reliable tool calling for coding agents. Up to 4× faster than Apple's MLX (mlx-lm
A minimal Python framework for building custom AI inference servers with full control over logic, batching, and scaling.
State-of-the-art LLM compression, built for production inference with vLLM
🚀 Accelerate inference and training of 🤗 Transformers, Diffusers, TIMM and Sentence Transformers with easy to use hardware optimization tools
Switchyard lets LLM applications route traffic across models and providers while preserving native OpenAI and Anthropic API compatibility - enabling flexible model selection, benchmarking, and cost/performance optimizati
Run Claude Code 100% on-device with local AI on Apple Silicon. MLX-native Anthropic-API server. 6 fighters incl. Muse-Glimmer 30B (now multimodal — reads images, abliterated), Gemma 4 31B, Qwen 3.5 122B (65 tok/s), DeepS
Stop configuring your AI stack. Start using it. One command brings a complete pre-wired LLM stack with hundreds of services to explore.
Distributed LLM inference. Connect home devices into a powerful cluster to accelerate LLM inference. More devices means faster inference.
RamaLama is an open-source developer tool that simplifies the local serving of AI models from any source and facilitates their use for inference in production, all through the familiar language of containers.
Community maintained hardware plugin for vLLM on Huawei Ascend