Google's end-to-end machine learning platform with a complete ecosystem.
Tokenizers
极速的现代分词器实现。
- Category
- ⚙️ Deep Learning Libraries
- Repository
- huggingface/tokenizers
- Stars
- ★ 11,181
- Forks
- 1,227
- Language
- Rust
- License
- Apache-2.0
- Last updated
- 2026-10-15
- Visits
- 0
Tokenization
Similar projects
A unified interface to tens of thousands of pre-trained models covering text, images, audio and multimodal tasks.
A deep-learning framework with dynamic computation graphs, the mainstream choice in both research and industry.
The classic Python machine-learning library: simple and efficient.
多后端（JAX / TF / PyTorch）的深度学习 API。
最简单快速的中型 GPT 训练与微调仓库。