QUARTZ.HOST
Generation VII.
Ensemble · Overture · Prelude — three tokenizers for a zero-sum world.
A tiered family of byte-level BPE tokenizers built for the fundamental realities of modern language-model training. Vocabulary capacity is a zero-sum macroeconomic space: every slot given to Amharic is a slot taken from Python. We refuse to ship one dishonest compromise — so we ship three deliberate budgets.
Supersedes the QT.VI.6.4-49k generation · legacy 64K conversational prose: qt-VI.3.5-64k
Vocabulary is a zero-sum economy.
Supporting 204 global languages, dense programming syntax and scientific mathematics at the same time pushes Byte-Pair Encoding into a hard boundary: a point where data quality can no longer compensate for spatial scarcity. A tokenizer’s vocabulary is a fixed parameter budget, and every slot within it is fiercely contested.
Once the training corpus is perfectly balanced and the merge algorithm fully optimised, universal tokenisation stops being a data problem and becomes pure arithmetic. The choice between Ensemble (129K), Overture (96K) and Prelude (64K) is a choice between minimising static embedding memory and maximising cross-domain sequence compression.
Allocate the 129K budget yourself.
Drag the diamond dividers — or focus them and use the arrow keys. The budget never grows.
| Domain | Allocation | Slots | Capacity | Status |
|---|
Three budgets. Three instruments.
Generation VII supersedes the QT.VI line. Each artifact lives in its own subfolder of the same repository and loads with a single AutoTokenizer.from_pretrained(..., subfolder=...) call — pick the budget that matches your parameter class.
Stratum thickness ∝ vocabulary size. Select a layer to inspect it.
Domain competency, honestly scored.
Scored from rigorous internal benchmarking against frontier commercial tokenizers.
The benchmark artifacts.
We ran the maths across three primary evaluation suites. Every figure ships with its raw data — open a table and check our work.
TokenizerBench Multi-Domain Summary
Higher Chr/Tok = better compression. Violet = category winner.
Python Code Compression — Parquet Suite
Lower total tokens = better compression. Violet = best-in-class.
FLORES-200 Regional Script Health
Lower token count = better compression (fewer byte-fallbacks). Violet = fewest tokens in row. Shaded columns = QT family.
Choose your tokenizer.
Because tokenizer capacity is a zero-sum space, no single artifact is perfect for every use case. “We believe in being intellectually honest with the developer community about where our artifacts win — and where you are better off using something else.”
The definitive choice for frontier models requiring universal language support. Use this if your model must support all 204 FLORES languages with zero script collapses while maintaining top-tier Python and programming compression. It leads the QT family in human-language compression and statistically ties DeepSeek V4 on Python code.
The Goldilocks zone for mid-tier small language models. If code is required, we recommend Overture for 90% of SLM use-cases: it slashes embedding VRAM requirements by 25.7% compared to 129K models, keeps sequence expansion for code under 1.5%, and maintains complete multilingual health.
The strict requirement for micro-models. Do the maths: an untied 129K vocabulary eats over half the parameter budget of a 1B model — Prelude slashes that bloat while keeping code density remarkably tight for agentic and terminal use, improving pure code density by over 19% compared to our legacy 64K build.
Highly recommended for high-resource, main-language models. If your project strictly targets major global languages (English, Chinese, Arabic, Russian) and code — and does not need regional scripts like Amharic or Odia, which it drops — this tokenizer is exceptionally well-balanced.
The optimal choice for western, English-centric code projects. Llama 3 heavily optimises for Western code formats but sacrifices regional and non-Latin script health to do so, resulting in severe byte-fallbacks in Asian and African languages.
If your project is strictly a multilingual conversational assistant rather than a coding assistant for terminal use, we advise our legacy VI.3 build. It retains the highest human prose density at the 64K tier.
Load a tier in four lines.
A tokenizer-only repository — no model weights are included, and trust_remote_code=True is not required: the full pipeline, including the code- and math-oriented pre-tokenization rules, is serialized in each tokenizer.json.
JamesQuartz/QT-VII-Tokenizer-Family/ ├── qt_VII_Plus_129k/ # Ensemble — tokenizer.json · vocab.json · merges.txt · tokenizer_config.json · chat_template.jinja ├── qt_VII_Mini_96k/ # Overture — same artifact set ├── qt_VII_Micro_64k/ # Prelude — same artifact set └── benchmark_charts/ # the three figures rendered on this page
Cite this work.
If you use any of the QT Generation VII tokenizers in your research or project, please cite the release.
@software{quartz_qtvii_2026,
author = {Downey, James},
title = {QT Generation VII Tokenizer Family: Ensemble, Overture \& Prelude},
year = {2026},
url = {https://huggingface.co/JamesQuartz/QT-VII-Tokenizer-Family}
}
- KiteFishAI
- For the comprehensive TokenizerBench suite — the foundational multi-domain evaluations for Bash, programming languages, human languages and scientific formulas.
- Meta AI · NLLB Team
- For the FLORES-200 dataset — the gold standard for global multilingual script health, used to stress-test regional byte-fallback boundaries across 200+ languages.
- The open-source community
- Including OpenAI and Google, for the underlying Python coding datasets (HumanEval / MBPP formatted parquet materials) used to push our Python compression benchmarks to the limit.