HN

Comments (40)

• Menu

  kierangill
  Instead of compaction, has anyone seen a successful implementation of pruning? That is, the agent looks at the conversation history and removes any low-value messages.For example, sometimes context will be taken up by a side tangent, tool call outputs, or low-value codebase exploration.Much of the time, I prefer to preserve the history of my conversation instead of summarizing it. I find summarized conversations lead to more frustrating future chats because the LLM misses intent and or context. (Or, the presence of paragraphs and paragraphs of LLM output makes the next token predictor dumber? Unsure.)
• Menu

  novaRom
  Compaction is painful if you run just one local LLM, the best way to avoid it is to keep context as small as possible.One trick I find useful is to have one model with two KV caches running and while first cache has produced tokens, second cache immediately summarizes them during input tokens are being generated (tools time), then harness switches to the second KV cache which takes newly produced input tokens while KV in first cache is getting replaced with compacted summary tokens. This is a kind of ping pong, so we trade more space for less time. Still experimenting but it looks it works, and nice bonus it improves GPU utilization. Btw I have my own harness and model serving code, but it can be easily implemented in any other harness and model server.
• skeledrew
  I think the way prompt caching works really discourages more creative compaction techniques. Like perhaps some kind of heuristic progressive compaction that replaces tool results and thinking traces after use with pointers could potentially keep the model smart for much longer, but that'd mean breaking cache every turn, and possibly even within a turn, seriously driving up cost.
• Menu

  damsta
  I don't like any of current solutions when it comes to compaction. I'd love to have a way to say what exactly should be summarized, because most of the time I just need to compact some noisy MCP tool calls, test runs and things like that. Just let me pick what should be summarized and keep the rest as is.
• Menu

  jakswa
  OMP changed the default compaction to images! Kinda nuts to read about. Saves the generation cost of the traditional compaction step and writes the context as tiny text to an image, if I was following correctly.
• Menu

  pornel
  I don't like that it throws away the whole KV cache when compacting. It costs a cache miss of the whole conversation length, and that's a waste of time and money.LLMs are perfectly capable of summarising the conversation without a new system prompt.
• storus
  The advantage of running local stack is that you can do the compaction at the time of inference, i.e. some tool call runs out of context, you can just pause inference, purge/replace old tool calls with their summaries or just logs by operating directly over tokens on a GPU, rebuilding KV cache (one time prefill hit) and resuming the inference, easily being able to e.g. read 1000 markdowns, each 50k long, in a single LLM call. That's not possible with current agentic harnesses using LLM calls.
• Menu

  zahrevsky
  Was expecting the article to go more in-depth.Say, what happens when chain of summaries grows so long, that it still overflows context window. Is summarization runned over the summaries in the context window?
• Menu

  kennywinker
  Compaction has been a pretty painful part of local llm usage. Scrapping the current context and parsing almosy 128k of context then generating something like 5-10k tokens - that can take quite a while when you’re working with 10t/s-45t/s (depending on the model).I pretty much just start a new session whenever i fill the context.
• anon
  undefined
• Menu

  brcmthrowaway
  Is pi a drop in replacement for OpenCode / Claude Code? I cannot be bothered installing 50 million plugins.
• Menu

  Gecko4072
  Can someone recommend a Hermes alternative that is less token hungry? Pi did not work well for my use case.
• Menu

  searealist
  I expect Pi is mostly used with OpenAI plans, and OpenAI has a dedicated compaction endpoint you should probably be using with their models instead of a compaction prompt.
• Menu

  randomblock1
  TLDR: It keeps ~20k tokens of recent conversations, then hands the rest of the conversation to another model with a special system & user prompt. This then fills out a template with relevant information.See: https://github.com/earendil-works/pi/blob/main/packages/codi...
• Menu

  cyanydeez
  Opencodes dynamic context pruning works by labeling tools and chat and the rest and the agent can collapse and expand summaries.I get it into 1M+ routinely on local models with operations between 50k-85k
• Menu

  juleiie
  Again someone stealing Tolkien work and using it for their corporate name.Evil cannot create only imitate.