Infinity is a high-throughput, low-latency serving engine for text-embeddings, reranking models, clip, clap and colpali
🖥️ Local Inference & Deployment
Ollama, llama.cpp, vLLM: model inference locally and in production
All 1045🧠 Large Language Models 24🤖 AI Agents 161🧩 App Development Platforms 22📚 RAG & Knowledge Bases 48💬 Chat UIs & Clients 40🖥️ Local Inference & Deployment 88👨💻 AI Coding 98🛠️ Frameworks & SDKs 47🔌 MCP & Tool Ecosystem 36🎨 Image Generation 26🎬 Video Generation 17🎙️ Speech & Audio 45👁️ Computer Vision 26📄 Document Parsing & Scraping 25🗄️ Vector Databases 25🏋️ Training, Fine-Tuning & MLOps 71🧪 Evaluation, Observability & Safety 79🚪 API Gateways & Proxies 12🔎 Search & Deep Research 15⚙️ Deep Learning Libraries 58🏷️ Data & Annotation 32🔬 Industry & Science AI 34⚡ Productivity Tools 16
Mirage Persistent Kernel: Compiling LLMs into a MegaKernel
MII makes low-latency and high-throughput inference possible, powered by DeepSpeed.
Large-scale LLM inference engine
TT-NN operator library, and TT-Metalium low level kernel programming model.
Fast Multimodal LLM on Mobile Devices
A high-performance inference engine for LLM, VLM, DiT and REC models, optimized for diverse AI accelerators. It is hosted in OpenAtom Foundation.
Self-hosted LLM router with a managed safety net. OpenAI-compatible. BYOK. Single-workspace. Streaming. For more advanced routing choose hosted OrcaRouter
RTP-LLM: Alibaba's high-performance LLM inference engine for diverse applications.
WebAssembly binding for llama.cpp - Enabling on-browser LLM inference
[NeurIPS'24 Spotlight, ICLR'25, ICML'25] To speed up Long-context LLMs' inference, approximate and dynamic sparse calculate the attention, which reduces inference latency by up to 10x for pre-filling on an A100 while mai
A FastAPI service for semantic text search using precomputed embeddings and advanced similarity measures, with built-in support for various file types through textract.
RAFT contains fundamental widely-used algorithms and primitives for machine learning and information retrieval. The algorithms are CUDA-accelerated and form building blocks for more easily writing high performance applic
Kubernetes AI Toolchain Operator
Open Model Engine (OME) — Kubernetes operator for LLM serving, GPU scheduling, and model lifecycle management. Works with SGLang, vLLM, TensorRT-LLM, and Triton
QA-Pilot is an interactive chat project that leverages online/local LLM for rapid understanding and navigation of GitHub code repository.