When using a less popular language, the AI ​​consumes more tokens, putting users at a disadvantage compared to speakers of English or another popular language.

According to Pratyush Choudhury, co-founder of venture capital fund Activate AI (India), using a language other than English leaves the model with less space to store documents or conversation history, and takes more processing when giving feedback.

"Think of a tutor. Remembering the student's previous mistakes and explanations is important for an effective lesson. However, the disadvantage comes when that part of the history takes up the context window much faster if a certain language is used," Choudhury told Rest of World. The context window is the maximum amount of information that the AI ​​model can receive and process during a session, usually calculated in tokens - basic input data units.

Statistics released this month by the Coalition for Language Equality in AI Research (CLEAR) also show that, for some languages, the model requires significantly more tokens. The group calls this a "hidden tokeniz tax," which makes using AI in languages ​​less common on the Internet expensive. Tokenizer is a tool that splits raw text, images or audio into tokens.

Research shows that even among the 20 most popular languages ​​on the Internet, there are differences in token usage. The most expensive languages ​​require 1.5-5 times as many tokens as the cheapest. This gap widened when the researchers expanded the set to 200 languages, with the token fee difference in one AI model increasing 17-fold.

"Startups in Bengaluru (India) have to pay more for each command and response, while their competitors in Austin (USA) do not. The tokenizer tax also has consequences beyond the cost issue. A larger amount of tokens leads to higher latency and reduced actual context capacity," the CLEAR team said.

Language rarity is not the only factor that determines the amount of tokens an AI model consumes. According to CLEAR, the writing system also plays an important role. For example, Asturian has few speakers but benefits from the dominance of Latin, while languages ​​that use scripts where tokenizers are rare, such as Burmese, Tibetan, and Khmer, are at a significant disadvantage with nearly every AI model.

According to Choudhury, perfect equality may be unrealistic, but large and persistent disparities still need to be explained and planned for improvement. "I recommend that every major AI lab publish a language-by-language benchmark, clearly stating how much it costs to complete a comparable task, and commit to making measurable improvements in the most disadvantaged languages," he said.

Earlier this year, many companies encouraged employees to "maximize tokens," using token consumption as an indicator of work performance. In March, New York Times quoted sources saying that employees of some technology companies such as Meta and OpenAI even compete on internal rankings of token consumption. At Anthropic, an employee using the Claude Code tool spent more than $150,000 in just one month.

However, according to Business Insider, companies in the past few months have switched to a strategy of using AI economically and effectively. Founders, software engineers, UX designers, even vibe coders without a technical background are starting to apply the "paradigm shift" trick. They assign difficult, high-thinking tasks to advanced and expensive models, while assigning easy, repetitive tasks to older and cheaper models. This tip helps make the most of your budget and use tokens effectively in the context of cost-limited businesses.

Thu Thao summary