HN

Need help?
<- Back
Running a 28.9M parameter LLM on an $8 microcontroller

Running a 28.9M parameter LLM on an $8 microcontroller

boveyking

Comments (24)

  • titzer
    It's crazy what $5 can buy you in a microcontroller these days. Have a look at these Milk-V boards:https://milkv.ioThe duo has up to 256MB of memory, and a 1TOPS@INT8 TPU. They run Linux and are $5. I bought 5!
  • rao-v
    This is a really neat use of the per-layer embedding trick. It's also worth noting that there viable TTS models that are ~20-30M param, so it might mean you can have a ESP32 with no network access read stuff out to you in near real time!
  • ReactiveJelly
    Finally, an LLM that can run on my i5
  • chrishynes
    Why can't this scale to run much larger models on CPU backed by flash with good access patterns?
  • spacedoutman
    >esp32-s3This microcontroller is a beast, currently using it to do dev work on a pi4.Having two usb ports with one otg lets you do some neat things that would cost $100+ otherwise
  • althea_tx
    This is a really cool project. Thanks for sharing!
  • cr125rider
    9.7 tokens/sec actually seems like a lot! That’s fun!
  • smy20011
    Super cool, thank you for sharing!
  • anon
    undefined