Need help?
<- Back

Comments (28)

  • itissid
    Someone mentioned that FPGAs though amazing to create bespoke chips that are fast, run into a fab capacity issue. Someone explained briefly to me that even if you could make an fpga that could do inference better than current GPUs it was economically possible to make them at scale. I can't recall whether that was for technology related issues or just because fab capacity is completely filled up.
  • jluxenberg
    A fun and interesting project, but to spend $10k+ in tokens on a curiosity seems excessive! Outside of my budget for sure.
  • cpldcpu
    I did something like this with Opus 4.5: https://github.com/cpldcpu/smollm.cIt seems implementing transformers is the new "hello world" of AI assisted coding.
  • hoppp
    This is exciting, I have been wishing for good ternary models for a while as it seems more effective and combining it with FPGA I think this is a super promising path forward.
  • datadrivenangel
    Does not say what type of model or size they were serving, but one of the traces mentions Maple, which is a 20b model. Still, getting ~70+ tok/s is quite good.
  • codekansas
    This is coolNext step is tuning your post training pipeline to optimize your model for your chosen FPGA architecture...
  • lxe
    Repetitive "make it faster" prompting is essentially what all these "RSI" claims are
  • chalkycrimp
    Good chance I'm not the only one thinking this: share the code
  • saejox
    i wonder if we can emulate an FPGA on a rtx 3090? then test it on an AWS EC2 F2 few hours a day.
  • spwa4
    I must say I do see one application for this, that you can't really match reasonably otherwise: you can bake weights into an FPGA and then run them for 5W at a speed that will be competitive with a big Nvidia card.(unfortunately for a decent model it will also be price competitive with a big nvidia card)
  • Archit3ch
    Now do silicon. ;)