HN

Comments (42)

• Menu

  kouteiheika
  > Opus 4.7 tokenizer used 1.46x the number of tokens as Opus 4.6Interesting. Unfortunately Anthropic doesn't actually share their tokenizer, but my educated guess is that they might have made the tokenizer more semantically aware to make the model perform better. What do I mean by that? Let me give you an example. (This isn't necessarily what they did exactly; just illustrating the idea.)Let's take the gpt-oss-120b tokenizer as an example. Here's how a few pieces of text tokenize (I use "|" here to separate tokens): Kill -> [70074] Killed -> [192794] kill -> [25752] k|illed -> [74, 7905] <space>kill -> [15874] <space>killed -> [17372] You have 3 different tokens which encode the same word (Kill, kill, <space>kill) depending on its capitalization and whether there's a space before it or not, you have separate tokens if it's the past tense, etc.This is not necessarily an ideal way of encoding text, because the model must learn by brute force that these tokens are, indeed, related. Now, imagine if you'd encode these like this: <capitalize>|kill <capitalize>|kill|ed kill| kill|ed <space>|kill <space>|kill|ed Notice that this makes much more sense now - the model now only has to learn what "<capitalize>" is, what "kill" is, what "<space>" is, and what "ed" (the past tense suffix) is, and it can compose those together. The downside is that it increases the token usage.So I wouldn't be surprised if this is what they did. Or, my guess number #2, they removed the tokenizer altogether and replaced them with a small trained model (something like the Byte Latent Transformer) and simply "emulate" the token counts.
• Menu

  aliljet
  This is the rugpull that is starting to push me to reconsider my use of Claude subscriptions. The "free ride" part of this being funded as a loss leader is coming to a close. While we break away from Claude, my hope is that I can continue to send simple problems to very smart local llms (qwen 3.6, I see you) and reserve Claude for purely extreme problems appropriate for it's extreme price.
• Menu

  lifis
  I'm really surprised that:1. Anthropic has not published anything about why they made the change and how exactly they changed it2. Nobody has reverse engineered it. It seems easy to do so using the free token counting APIs (the Google Vertex AI token count endpoint seems to support 2000 req/min = ~3million req/day, seems enough to reverse engineer it)
• sergiopreira
  An interesting question is whether the tokenizer is better at something measurable or just denser. A denser tokenizer with worse alignment to semantic boundaries costs you twice, higher bill and worse reasoning. A denser tokenizer that actually carves at the joints of the model's latent space pays for itself in quality. Nobody outside Anthropic can answer which it is without their eval suite, so the rugpull read is fair but premature. Perhaps the real tell will be whether 4.7 beats 4.6 on the same dollar budget on the benchmarks you care about, not on the per-token ones Anthropic publishes.
• hugodan
  Aren't these increases offset by the quality of the responses and reducing the iterations needed to fine-tune the responses?
• Menu

  great_psy
  Is there any provided reason from anthropic why they changed the tokenizer ?Is there a quality increase from this change or is it a money grab ?
• onchainintel
  Many comparisons between 4.6 & 4.7 at https://tokens.billchambers.me/leaderboard My prompt was 40% more tokens using Opus 4.7.
• tpowell
  I just asked Claude about defaulting to 4.6 and there are several options. I might go back to that as default and use --model claude-opus-4-7 as needed. The token inflation is very real.
• Menu

  tomglynch
  Interesting findings. Might need a way to downsample images on upload to keep costs down.
• Menu

  mudkipdev
  Why do you need an API key to tokenize the text? Isn't it supposed to be a cheap step that everything else in the model relies on?
• chattermate
  [dead]
• yogigan
  [dead]