FlashInfer: Kernel Library for LLM Serving
🖥️ ローカル推論とデプロイ
Ollama、llama.cpp、vLLM：ローカルから本番環境までのモデル推論
すべて 1045🧠 大規模言語モデル 24🤖 AI エージェント 162🧩 アプリ開発プラットフォーム 22📚 RAG とナレッジベース 48💬 チャット UI とクライアント 40🖥️ ローカル推論とデプロイ 88👨💻 AI コーディング 97🛠️ 開発フレームワークと SDK 47🔌 MCP とツールエコシステム 36🎨 画像生成 26🎬 動画生成 17🎙️ 音声とオーディオ 45👁️ コンピュータビジョン 26📄 ドキュメント解析とスクレイピング 25🗄️ ベクトルデータベース 25🏋️ 学習・ファインチューニングと MLOps 71🧪 評価・可観測性・安全性 79🚪 API ゲートウェイとプロキシ 12🔎 検索とディープリサーチ 15⚙️ ディープラーニング基盤ライブラリ 58🏷️ データとアノテーション 32🔬 業界・研究向け AI 34⚡ 効率化ツール 16
Reliable model swapping for any local OpenAI/Anthropic compatible server - llama.cpp, vllm, etc
⚡ Pure-Rust WebGPU inference engine — OpenAI-API compatible, GGUF native, runs on any GPU. No Python. No llama.cpp. Single binary.
LLocalSearch is a completely locally running search aggregator using LLM Agents. The user can ask a question and the system will use a chain of LLMs to find the answer. The user can see the progress of the agents and the
A GPU cluster manager for high-performance AI model serving (vLLM, SGLang) and on-demand SSH-accessible GPU instances.
A unified library of SOTA model optimization techniques like quantization, distillation, pruning, neural architecture search, speculative decoding, etc. It compresses deep learning models for downstream deployment framew
A compact implementation of SGLang, designed to demystify the complexities of modern LLM serving systems.
Cost-efficient and pluggable Infrastructure components for GenAI inference
On-device AI across mobile, embedded and edge for PyTorch
Achieve state of the art inference performance with modern accelerators on Kubernetes
Fast inference engine for Transformer models
TNN: developed by Tencent Youtu Lab and Guangying Lab, a uniform deep learning inference framework for mobile、desktop and server. TNN is distinguished by several outstanding features, including its cross-platform capabil
LightLLM is a Python-based LLM (Large Language Model) inference and serving framework, notable for its lightweight design, easy scalability, and high-speed performance.
Optimizing inference proxy for LLMs
Rapid-MLX is an open-source (Apache 2.0) OpenAI- and Anthropic-compatible LLM inference server for Apple Silicon, built on MLX, focused on reliable tool calling for coding agents. Up to 4× faster than Apple's MLX (mlx-lm
A minimal Python framework for building custom AI inference servers with full control over logic, batching, and scaling.
State-of-the-art LLM compression, built for production inference with vLLM
🚀 Accelerate inference and training of 🤗 Transformers, Diffusers, TIMM and Sentence Transformers with easy to use hardware optimization tools
Switchyard lets LLM applications route traffic across models and providers while preserving native OpenAI and Anthropic API compatibility - enabling flexible model selection, benchmarking, and cost/performance optimizati
Run Claude Code 100% on-device with local AI on Apple Silicon. MLX-native Anthropic-API server. 6 fighters incl. Muse-Glimmer 30B (now multimodal — reads images, abliterated), Gemma 4 31B, Qwen 3.5 122B (65 tok/s), DeepS
Stop configuring your AI stack. Start using it. One command brings a complete pre-wired LLM stack with hundreds of services to explore.
Distributed LLM inference. Connect home devices into a powerful cluster to accelerate LLM inference. More devices means faster inference.
RamaLama is an open-source developer tool that simplifies the local serving of AI models from any source and facilitates their use for inference in production, all through the familiar language of containers.
Community maintained hardware plugin for vLLM on Huawei Ascend