What are tokens in LLMs?

Token's the atomic unit: word chunk, subword, or byte sequence. BPE carves text into 'em. English efficient; others, not so much.

Why does Portuguese use more tokens than English?

Training data's English-heavy. Rare chars like accents fragment words. Expect 30-90% premium.

Shared space for input + output, up to 1M tokens in top models. But "lost in the middle" kills recall.

🚀 New Releases

Snap a full 'hello' brick in English. Now hack 'olá' into shards. That's your LLM reality—and it costs you.

theAIcatchup Apr 09, 2026 3 min read

Portuguese prompts consume 30-94% more tokens than English due to tokenizer bias. 𝕏
1M token contexts sound huge but suffer 'lost in the middle' attention loss. 𝕏
Confident hallucinations stem from autoregressive generation—mimics human overconfidence. 𝕏

Published by

Ship faster. Build smarter.

#BPE #Claude AI #Claude models #LLM tokens #context window #tokenization bias

Get the best Developer Tools stories of the week in your inbox — no noise, no spam.

Originally reported by dev.to