We Measured It: Japanese Uses 1.6–2.3× More Tokens Than English (50 Sentence Pairs)


 

"Non-English languages cost more tokens" is one of those claims everyone repeats and almost no one quantifies. So we measured it. We built 50 matched English–Japanese sentence pairs across five genres, ran them through two generations of a public tokenizer, and counted. Every number below is our own measurement, the method is fully described, and anyone can reproduce it in ten minutes.

Method

We wrote 50 natural parallel sentence pairs — 10 each in daily conversation, business email, technical writing, news style, and how-to instructions — and counted tokens with the open-source js-tiktoken implementation using two encodings: o200k_base (the GPT-4o generation) and cl100k_base (the GPT-4 generation). One honest caveat up front: every vendor's tokenizer differs, so these are exact numbers for these two public encodings, not universal constants.

Finding 1: Japanese costs 1.63× (newer) to 2.26× (older) more

EncodingEnglish totalJapanese totalJA / EN
o200k_base (newer)4948071.63×
cl100k_base (older)5001,1322.26×

Per-sentence medians match the totals (1.60× and 2.20×), and individual sentences ranged from 1.00× to 2.63× on the newer encoding. So "Japanese doubles your bill" is too crude — the premium varies a lot by what you are writing.

Finding 2: technical writing pays the biggest premium


By genre, everyday conversation shows the smallest gap (1.37× on o200k) and technical writing the largest (1.97×, and a striking 2.69× on the older encoding). Katakana loanwords like "tokenizer" and "fine-tuning" get shredded into many small pieces. If you generate long technical content in Japanese, this is why it costs more than intuition suggests.

Finding 3: the newer tokenizer quietly cut Japan's AI bill

The most interesting number in the experiment: on the older encoding, one Japanese character ≈ one token (0.98 chars/token measured). On the newer one, that improved to 1.37 characters per token — while English stayed flat at about 5.0 chars/token across both generations. The upgrade's benefit went almost entirely to non-English text, effectively cutting Japanese token costs by roughly 40%. Tokenizer generations are an unglamorous changelog item with real money attached.

Bonus: a five-language mini comparison

The same 10 sentences translated into five languages, counted on the newer encoding (English = 1.00):

LanguageTotal tokensvs English
English971.00
Chinese1101.13
Spanish1181.22
Korean1461.51
Japanese1591.64

A 10-sentence set is indicative rather than definitive, but within it Japanese was the most token-expensive language tested.

Three practical takeaways

  • Budgeting: if you priced an API workload in English, multiply by 1.5–2× before promising the same in Japanese or Korean.
  • Effective context shrinks too: a "100K-token" context window holds meaningfully less Japanese prose than English prose — same arithmetic, same reason.
  • Formal boilerplate is the cheapest cut: part of the business-email premium is long honorific formulas. Trimming ceremonial phrasing from prompts saves real tokens without losing meaning.

Limitations, stated plainly

We measured two public encodings, not every vendor's current tokenizer; translation choices shift counts a little; and 50 pairs shows a robust pattern, not a peer-reviewed sample. Still, the 1.5–2.3× range held consistently across genres, medians, and the multilingual set — and it matches what the pricing pages won't tell you directly.

Frequently asked questions

Can I reproduce this?

Yes — install js-tiktoken from npm, encode any parallel sentences with o200k_base and cl100k_base, and compare lengths. The underlying tokenizer definitions come from OpenAI's open-source tiktoken project.

Does this apply to every AI service?

The exact ratios, no — tokenizers differ per model. The direction — Japanese, Korean, and other non-Latin scripts costing more than English — is broadly shared. Check your vendor's token counter before committing to a budget.

What's a token again?

Our explainer What Is a Token in AI? covers the basics; this article is its measurement sequel.

Related on AI Learning Lab: What Is a Token in AI? · What Is a Context Window? · Free vs Paid AI Tools

Comments

Post a Comment

Popular posts from this blog

Free vs Paid AI Tools: When Is Upgrading Actually Worth It?

Image
Every AI assistant now has a free tier that's genuinely capable, and a paid tier that costs roughly the price of a couple of lunches per month. So the question isn't "is AI worth paying for" in the abstract — it's whether your specific usage crosses the line where paying changes anything . This article gives you that line. What you actually get when you pay Pricing and features shift constantly, so ignore the marketing lists and look at the three differences that persist across providers: Priority access. Free tiers get throttled or queued at busy hours. If your work happens during peak times, this is the difference between a tool you rely on and a tool that sometimes isn't there. The current top model. Free tiers usually route you to smaller or older models, with limited access to the best one. For simple tasks this doesn't matter. For reasoning-heavy work, it's the whole ballgame. Data handling. This is the one most people overlook. Paid and busin...
Read more

AI Search vs Traditional Search: How to Use Each in 2026

Image
Something changed in how people find things, and the numbers are unusually clear about it. According to a SparkToro analysis of Similarweb clickstream data published in June 2026, about 68% of U.S. Google searches between January and April 2026 ended without a click — up from roughly 60% in 2024. Answers are arriving without anyone visiting a page. That makes "which kind of search should I use?" a question worth having a real answer to. What the data actually says Two figures from that analysis matter most. AI Overviews now appear on more than 20% of Google searches, and click-through rates fall by roughly 60% when they do . The behavior shift is not that people stopped searching — it is that a search increasingly ends where it used to begin. For anyone doing research, the tradeoff is speed against provenance. You get an answer faster, and you can see less about where it came from. When AI search is the right tool You do not know the vocabulary yet — "the clause that l...
Read more

Are AI Certifications Worth It in 2026? A Practical ROI Test

Image
AI certifications have become a crowded market. Some are free and take an afternoon; others run into the thousands of dollars. The honest answer to "is it worth it?" is: it depends entirely on what you expect it to do for you . This article gives you a framework for judging that before you pay. What a certification can and can't do A credential is evidence, not a qualification. In practice it does three useful things: Gets you past a filter. When a job posting lists a certification, having it keeps you in the pile. Signals seriousness in a career change. If you're moving from a non-technical role into AI-adjacent work, a recognized credential shows you invested real effort. Forces structured learning. Many people need a syllabus and a deadline. That's a legitimate reason to pay. What it almost never does is produce a salary increase on its own. Employers pay for demonstrated capability. The certificate opens the conversation; your work wins it. A four-question ...
Read more