Infinity is a high-throughput, low-latency serving engine for text-embeddings, reranking models, clip, clap and colpali
🖥️ Inferencia y despliegue local
Ollama, llama.cpp, vLLM: inferencia de modelos en local y en producción
Todo 1045🧠 Modelos de lenguaje 24🤖 Agentes de IA 161🧩 Plataformas de desarrollo de apps 22📚 RAG y bases de conocimiento 48💬 Interfaces de chat y clientes 40🖥️ Inferencia y despliegue local 88👨💻 Programación con IA 98🛠️ Frameworks y SDK 47🔌 MCP y ecosistema de herramientas 36🎨 Generación de imágenes 26🎬 Generación de vídeo 17🎙️ Voz y audio 45👁️ Visión por computador 26📄 Análisis de documentos y scraping 25🗄️ Bases de datos vectoriales 25🏋️ Entrenamiento, ajuste fino y MLOps 71🧪 Evaluación, observabilidad y seguridad 79🚪 Pasarelas y proxies de API 12🔎 Búsqueda e investigación profunda 15⚙️ Bibliotecas de aprendizaje profundo 58🏷️ Datos y anotación 32🔬 IA para industria y ciencia 34⚡ Herramientas de productividad 16
Mirage Persistent Kernel: Compiling LLMs into a MegaKernel
MII makes low-latency and high-throughput inference possible, powered by DeepSpeed.
Large-scale LLM inference engine
TT-NN operator library, and TT-Metalium low level kernel programming model.
Fast Multimodal LLM on Mobile Devices
A high-performance inference engine for LLM, VLM, DiT and REC models, optimized for diverse AI accelerators. It is hosted in OpenAtom Foundation.
Self-hosted LLM router with a managed safety net. OpenAI-compatible. BYOK. Single-workspace. Streaming. For more advanced routing choose hosted OrcaRouter
RTP-LLM: Alibaba's high-performance LLM inference engine for diverse applications.
WebAssembly binding for llama.cpp - Enabling on-browser LLM inference
[NeurIPS'24 Spotlight, ICLR'25, ICML'25] To speed up Long-context LLMs' inference, approximate and dynamic sparse calculate the attention, which reduces inference latency by up to 10x for pre-filling on an A100 while mai
A FastAPI service for semantic text search using precomputed embeddings and advanced similarity measures, with built-in support for various file types through textract.
RAFT contains fundamental widely-used algorithms and primitives for machine learning and information retrieval. The algorithms are CUDA-accelerated and form building blocks for more easily writing high performance applic
Kubernetes AI Toolchain Operator
Open Model Engine (OME) — Kubernetes operator for LLM serving, GPU scheduling, and model lifecycle management. Works with SGLang, vLLM, TensorRT-LLM, and Triton
QA-Pilot is an interactive chat project that leverages online/local LLM for rapid understanding and navigation of GitHub code repository.