Flide is extremely fast when one input carries many questions, as the shared input is computed only once. Even with a single question, it can still be up to 3x faster, depending on the model size and input length.

You really don&#x27;t need an entire new inference engine for this. I modified vllm to work just fine for my decision model experiments.It properly uses the single shared context and only recomputes the questions, or if the questions are longer, I let the questions go first and then the context that changes often can go after, and then the answer menu tokens... I got roughly the same speed increase with the right workloads that you&#x27;re saying though.

I wanted to try this - essentially for my dataset I have a number of decision questions for the same input. I haven&#x27;t figured out yet which local model to try first with it!

Hi HN! I&#x27;m one of the Flide team.We built Flide from scratch to make Jev-like decision model inference up to 9.5x faster than vLLM and SGLang, while keeping exactness.Here is the source code: <a href="https:&#x2F;&#x2F;github.com&#x2F;OmniJev&#x2F;Flide" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;OmniJev&#x2F;Flide</a>Look forward to hearing your feedback and ideas!

do i need to host the model to do this ?
if yes what model can i host in 3050 card

Decision models need their own inference engine.So we developed Flide, a high-performance and exact inference engine for Jev-like decision models.Flide achieves up to a 9.5x speedup over vLLM and SGLang.It now supports 30+ popular local Jev-like decision models.Start to make your decision immediately:pip install flide

HN

Show HN: Flide: Make Jev-like decision models up to 9.5x faster

Comments (23)