GigaToken: SIMD + Cache Hierarchies for 1000x Faster Tokenization

Codex 校核（2026-10-09） GigaToken 官方基准与 API 说明中约 989×（标题取整为 1000×）来自双路 AMD EPYC 9565、11.9GB OpenWebText、GPT-2 tokenizer 的特定测量；其他 tokenizer 的倍率不同。原生批量 encode_files 避开 Python 数据搬运，而 as_hf() 兼容模式会增加开销，不能把 1000× 当作任意模型、普通电脑或逐条 Hugging Face 调用的保证。本页 stars/速度数字保留采集时点。

Core Idea

Rust tokenizer achieving ~1000x speedup over HuggingFace tokenizers via three key optimizations: SIMD-based pretokenization (replacing regex engine), multi-level cache hierarchy for seen word→token mappings, and zero-copy data paths with mmap file reading.

Why It Matters

Batch data preprocessing (training corpus tokenization, RAG doc preprocessing) bottleneck eliminated. Proves that even already-optimized Rust libraries (HF tokenizers is multithreaded Rust) have 2-3 orders of magnitude optimization headroom when you attack the hot path.

Technical Details

• SIMD replaces regex engine for pretokenization (main speedup)
• L1/L2/L3-aware cache for pretoken mappings (second main speedup)
• Zero-copy mmap file reads bypass Python object creation and GIL
• Minimal thread interaction — no-lock/no-contention design

Benchmarks (GPT-2 tokenizer)

Utility for Hermes

• pip install gigatoken for any batch text preprocessing
• Architecture pattern: profiling→hot path→SIMD+cache+zero-copy can apply to other bottlenecks