<- Back
Comments (38)
- ricardobeatEveryone is doing this to emulate Jev, but...I took a random book excerpt with 23,000 words (±30k input tokens) and used it as context. Jev still responds in 800ms, sometimes 500ms. That's in the neighbourhood of 20-50,000 tok/s prefill, which is obviously not possible with normal LLMs, not even Cerebras is this fast.
- ArtRichardsI personally love Privatemode's approach. Having Jev-like speed for confidential ai use cases is a huge enabler.
- walrus01You can turn any sufficiently smart LLM into yes/no decision model or equivalent. I already have an existing workflow with a two paragraph detailed prompt, that sends pages of stuff to an LLM and asks it to return only 7 JSON objects. Several of those objects are binary "yes or no" choices of like, whether the content contains certain things.You can even do it with small not particularly hard to host local LLMs like a variant of Qwen 3.6 35B A3B or 3.8 27B.
- prjkthow is Jev cheaper if I can run locally. 0.5% prefill, 0.1% decode, 99.4% cached, latency is <20ms
- m4y0uMy question is why not use Jev instead? It's faster and cheaper.
- janalsncmIf you are using an autoregressive decoder (which glm is) it is not “jev-like”. You lose all of the speed advantages that Jev has.
- ttoinouIsnt this obvious ? I would have thought people would try such things before deciding they need something like Jev
- JabrovIs this a joke? “Jev-like” properties? People have been using LLMs as classifiers or rankers in a similar way for ages. I feel like we’re losing our minds
- yogthosRIP Jev
- anonundefined
- anonundefined
- tomek7667[dead]