HN

Comments (3)

• Menu

  augment_me
  Its cool, but it goes against the whole optimization that NLP has gone through with BPE and redundancy/repetition in large data in general.Put this on a coding task or English and you will underperform against tokenizer that exploit multi-char patterns and can decode much faster.Given where things are going with spec decode (I've seen 32 token spec-decode pop up in the frontier models), you will need a very domain-specific use for this to be useful imo like massive tabular data, but at that point you already have everything structured, and might as well use an LLM with polars/sql/arrow.
• vatsachak
  Oh man, that haskell quicksort tokenization has got me feeling great
• aburayhan
  [flagged]