FlashInfer: Kernel Library for LLM Serving
🖥️ Local Inference & Deployment
Ollama, llama.cpp, vLLM: model inference locally and in production
All 1045🧠 Large Language Models 24🤖 AI Agents 162🧩 App Development Platforms 22📚 RAG & Knowledge Bases 48💬 Chat UIs & Clients 40🖥️ Local Inference & Deployment 88👨💻 AI Coding 97🛠️ Frameworks & SDKs 47🔌 MCP & Tool Ecosystem 36🎨 Image Generation 26🎬 Video Generation 17🎙️ Speech & Audio 45👁️ Computer Vision 26📄 Document Parsing & Scraping 25🗄️ Vector Databases 25🏋️ Training, Fine-Tuning & MLOps 71🧪 Evaluation, Observability & Safety 79🚪 API Gateways & Proxies 12🔎 Search & Deep Research 15⚙️ Deep Learning Libraries 58🏷️ Data & Annotation 32🔬 Industry & Science AI 34⚡ Productivity Tools 16
Reliable model swapping for any local OpenAI/Anthropic compatible server - llama.cpp, vllm, etc
⚡ Pure-Rust WebGPU inference engine — OpenAI-API compatible, GGUF native, runs on any GPU. No Python. No llama.cpp. Single binary.
LLocalSearch is a completely locally running search aggregator using LLM Agents. The user can ask a question and the system will use a chain of LLMs to find the answer. The user can see the progress of the agents and the
A GPU cluster manager for high-performance AI model serving (vLLM, SGLang) and on-demand SSH-accessible GPU instances.
A unified library of SOTA model optimization techniques like quantization, distillation, pruning, neural architecture search, speculative decoding, etc. It compresses deep learning models for downstream deployment framew
A compact implementation of SGLang, designed to demystify the complexities of modern LLM serving systems.
Cost-efficient and pluggable Infrastructure components for GenAI inference
On-device AI across mobile, embedded and edge for PyTorch
Achieve state of the art inference performance with modern accelerators on Kubernetes
Fast inference engine for Transformer models
TNN: developed by Tencent Youtu Lab and Guangying Lab, a uniform deep learning inference framework for mobile、desktop and server. TNN is distinguished by several outstanding features, including its cross-platform capabil
LightLLM is a Python-based LLM (Large Language Model) inference and serving framework, notable for its lightweight design, easy scalability, and high-speed performance.
Optimizing inference proxy for LLMs
Rapid-MLX is an open-source (Apache 2.0) OpenAI- and Anthropic-compatible LLM inference server for Apple Silicon, built on MLX, focused on reliable tool calling for coding agents. Up to 4× faster than Apple's MLX (mlx-lm
A minimal Python framework for building custom AI inference servers with full control over logic, batching, and scaling.
State-of-the-art LLM compression, built for production inference with vLLM
🚀 Accelerate inference and training of 🤗 Transformers, Diffusers, TIMM and Sentence Transformers with easy to use hardware optimization tools
Switchyard lets LLM applications route traffic across models and providers while preserving native OpenAI and Anthropic API compatibility - enabling flexible model selection, benchmarking, and cost/performance optimizati
Run Claude Code 100% on-device with local AI on Apple Silicon. MLX-native Anthropic-API server. 6 fighters incl. Muse-Glimmer 30B (now multimodal — reads images, abliterated), Gemma 4 31B, Qwen 3.5 122B (65 tok/s), DeepS
Stop configuring your AI stack. Start using it. One command brings a complete pre-wired LLM stack with hundreds of services to explore.
Distributed LLM inference. Connect home devices into a powerful cluster to accelerate LLM inference. More devices means faster inference.
RamaLama is an open-source developer tool that simplifies the local serving of AI models from any source and facilitates their use for inference in production, all through the familiar language of containers.
Community maintained hardware plugin for vLLM on Huawei Ascend