<- Back
Comments (65)
- dofmAmazing to me that something as wild as this is now basically a trope.I don't mean this negatively; open weights AI and open source tooling is about the only thing that gives me hope.The fact that there are dozens of these things hints at the beginning of the dull normalisation of openly available, non-monopoly AI tools, which is what is needed if we are all to survive.
- jdb9001Every few weeks, someone stumbles on what llama.cpp has had for years again, and reimplements it. I'm not sure why.
- suprjamiQwen 3? (Jumanji) What year is it?What a waste of time.Strata already does this with Qwen3.8-Flash-Next: https://github.com/Niko1221/Strata
- sieve> Using a GPU has, once, caused a driver reset that took the desktop with it.Only once? This has happened to me so many times with LLM-assisted development. I was writing an OCR engine and pathological shapes would cause OOM that would take the display out. On other occasions, the LLM would try to load 24GB of bf16 weights onto a 16GB GPU without understanding the consequences.Have to encode this in logic: keep some space reserved for the display, and check before allocation. But some errors are bound to slip through.---Inspired by DwarfStar4 and Strata, I am trying to build my own gemma4-specific inference engine designed to run on gfx1201.It is a lot of work. You cannot simply vibecode one in two days.
- 3eb7988a1663If anyone needs to see what 62 tokens/second looks like: https://token.qusai.pro/?speed=62Which seems sufficient for mortals who are not running gas town, but want a local model.
- LercI think this is still the tip of the iceberg in terms of this method.There is no particular requirement for this to be limited to a single SSD. What happens when someone makes a GPU Card with enough SSDs attached to feed every RAM chip individually.
- alexpotatoI'm waiting for this idea to get generalized out to networks.e.g. each expert lives on one node of the network and the model gets streamed.
- skeledrewThis could be a game changer for me. The best I've been able to do so far is ~8 tok/s with Qwen 35B A3B, 32GB RAM. But then again I just got an OpenCode Go plan yesterday and have yet to figure if I can even consume my quota on DeepSeek Flash as the need to plan things well is bottlenecking me. Might be a while before I try the local thing again.
- yaanfpvCouldnt engines already do this albeit super slow?
- LeoisNotAII think fits in a RAM is the first step, a real practical question is if in the mean time you can still use the machine for basic tasks like use browser watch YouTube/VS Code etc.
- kadobanWhy are the models being used so old (in llm years anyway). Why no Qwen 3.8 ?
- aboardRat4Zoomers discover swap?
- sheepscreekHonestly, I’m more worried about losing my SSD especially with where the prices are today. Pretty sure this is going to put a lot of strain on it.
- chernobyI'm always excited to see more open source apps like this.
- 0x073Let's make ssds more expensive.
- InsidiousWombatI wonder when we will start seeing small commercial accelerators integrated with NVMes; could potentially use far more bandwidth than typical NVMe drive can push thru the four lanes.Given that you'd have probably a lot more capacity just to get to required bandwidth it could be used to switch very quickly between models (as all stored there would essentially be "loaded")
- KerrAvonThere are many of these stream-experts-from-SDDD projects now. What's needed is a guide to all of them and maybe for someone to take the best features of each of them and merge them into one uber project.
- purplecatswould this work on a virtual disk, say w s3 or dropbox
- sheephess44[flagged]
- singularityisne[flagged]
- kbesplv[dead]
- zelconI think GitHub repos should have a new tag listing the model used to write it along with the token cost.I don’t want to bother reading anything sub-Fable, and this one looks sub-Fable.