A tokenizer study released on September 29 by the research group CLEAR (Coalition for Language Equity in AI Research) has been spreading over the past two weeks after coverage by Rest of World and other outlets. It measures one very specific thing: how many tokens a model has to cut the same content into when it is written in a different language. The researchers call the excess the "tokenizer tax."
CLEAR used the FLORES-200 benchmark, a set of professionally translated sentences in roughly 200 languages, which in practice covers 203 combinations of language and script. The example sentence in the study is "People should benefit from AI, regardless of the language they speak": 13 tokens in English, 26 in Malayalam and 78 in Amharic.
Same "frontier" label, gaps from 5x to 17x
The study's main metric is the max-to-min ratio: how many times more tokens a tokenizer uses on its most expensive language than on its cheapest one. Some of the results reported publicly:
- Google Gemini and Gemma family: 5.1x, the best among general-purpose frontier models
- Anthropic Claude: 9.3x
- Meta Llama 4 Scout family: 9.5x
- OpenAI GPT-5: 13.8x
- Falcon 3: 17x, the worst
Looking only at the 20 most widely used languages on the internet, most tokenizers fall between 1.5x and 3.1x, while Falcon 3 comes in at 4.5x. The two headline versions in the press, "1.5 to 5 times" and "1.5 to 3.1 times," differ only in whether Falcon 3 is counted.
The gap has more to do with writing systems than with the number of speakers. Asturian has few speakers, but it benefits from the Latin alphabet and its place in the Romance language family. Burmese, Tibetan, Khmer and Lao, along with the Tifinagh and Ol Chiki scripts, sit in the least favorable position on almost every model. Languages with hundreds of millions of speakers, such as Bengali, Urdu and Punjabi, were also singled out.
The ratio measures encoding length, which is not the same as actual price or task success rate. The baseline in the study is each tokenizer's own cheapest language, usually English or Simplified Chinese. CLEAR itself acknowledges that languages naturally differ in length, so a perfect one-to-one ratio is probably unrealistic.
A rough look at cost and context
More tokens means bigger bills, higher latency and less usable context. Extrapolating from the example sentence above (a single-sentence sample, so treat it as an order-of-magnitude guide only): an app that spends $1,000 a month on input in English would need roughly $6,000 for the same content in Amharic on the same model. A 128,000-token context window would hold only about one-sixth as much text, which hits long-document Q&A and agents first.
The counterargument comes from Dheemanth Reddy, CEO of Maya Research, who argues that falling token prices will ease the direct cost pressure:
"Token prices are falling very quickly, so the direct cost impact should become less severe over time"
He also cautions that adding more training data alone will not fix tokenization; labs need to design tokenizers specifically for these languages and scripts. Price cuts, he notes, do nothing about the squeeze on context.
What Chinese models and overseas teams should look at
Simplified Chinese is on the favorable side in this study, so users in China may not feel this tax in daily use. Overseas deployments are where to pay attention. Southeast Asian businesses often deal with Burmese, Khmer and Lao, which happen to be among the worst-affected languages.
Chinese models do not look good either: Qwen 3 has a ratio of 10.5, and the newer Qwen 3.6 rises to 13.1, so a model update did not automatically bring improvement. The numbers also show the gap is not a technical dead end. Meta's translation model NLLB-200 scores only 2.7x, since it was designed for multilingual use from the start.
CLEAR made several recommendations: document per-language token inequality in model cards; deliberately cover more languages in the next generation of models; since tokenizers can only change with a new model, bill by character or by language immediately; and have public procurement buyers require vendors to disclose costs across languages. Pratyush Choudhury, co-founder of Activate AI, asks for something simpler: every major lab should publish a per-language report card. In the public coverage so far, all of these remain recommendations.
Sources: Rest of World; CocoLoop; Trending Topics; the public summary of the CLEAR study. Per-model ratios follow CLEAR's figures as relayed in press coverage; the full report was not obtained directly.