HN

Comments (43)

• Menu

  lelanthran
  1-bit is not much though. Here's what I got:Me:> Describe the process of pasteurisationResponse:> pasteurization is a, which a, the which which is, the and process, the the past, the and the, and and and past, and past, and and and and the, the and past, process is the is is is is, and the, past, and and future, process, and and and and and and and the, past, process, process, the the, the the, is and the, the the, and and and, and process, and and the, past, and and and past, and the, the, and the past, the the, process, process, and past, the past, past, the and, the past, and and and and and and and and and the, and and and and and and and, the the, the the, the or and and, the the, the and the, which the, and the, the the, past, and n the process, and and and and, past, and, and and, the past, and, the the, past,, and the, the the, the is the, past, and and and and and, and and and past, the and the, the the, the the, are and past, and which the, the and n, n the, the n past, past, n the, and n, the the process, which past, the the, the n, the the, the is past, the the, is, past, the the, past, and past, process, the the, the the, the and and the, and which past, the(and that basically just goes on and on like that)
• Menu

  stfurkan
  I'm currently working on an open-source engine [1] exactly for this purpose. If anyone wants to try or has any suggestions, I'm happy to listen :)1. https://github.com/stfurkan/bitgpu2. https://aidekin.com --> this is one of my projects that's currently using bitgpu engine
• Menu

  Lwerewolf
  Pretty sure this might be a duplicate. Regardless, tried the 1bit bonsai 27b gguf three different ways - their llama.cpp fork (prism, was it) on 2 machines (1255u/16g, m5 max/128g) and the web (on the 1255u). With llama.cpp it works. on the web it emits the same thing over and over. Prompt was "do you see anything wrong with this code: <C code, compound literal passed as a pointer to a function in an if statement>", result was literally "Yes, let'ssomestruct_struct_tsomestruct<a bit of similar garbage>structstructstruct<forever>.Locally, it's definitely not the full 3.6 27b, but for ~6G with the context, it's quite impressive, and I like what this implies for larger models. Speeds on the 1255u are abysmal, though (~1TPS or less) - granted, CPU-only.
• om8
  This project needs webgpu -- I did it on cpu about a year ago.My demo uses 2 bit quantization to run llama3 models on any device with enough ram.https://galqiwi.github.io/aqlm-rs/
• aavci
  Other than privacy and security and simplified development, what are other valid reasons you'd prefer to run this instead of a backend LLM?
• Menu

  estebarb
  I tried the smallest ome, the only available, and got an error:``` Could not load.Error: failed to call OrtRun(). ERROR_CODE: 1, ERROR_MESSAGE: Non-zero status code returned while running GroupQueryAttention node. Name:'/model/layers.0/attn/GroupQueryAttention' Status Message: Failed to create a WebGPU compute pipeline: A valid external Instance reference no longer exists. ```
• Menu

  leumon
  i found this space to work better/at all: https://huggingface.co/spaces/webml-community/bonsai-webgpu-...
• Menu

  indianmouse
  Could not get any usable output from it for any real usecase.Can't see any real value in using 1 bit llms except and apart for the sake of running it for demo or feel good factor of running a higher param model locally and in limited hardware!Since use as it is very limited, could someone throw some light on any actual use case?
• Menu

  dim13
  Fails "car wash" test miserably. Ask it:> I want to wash my car. The car wash is 50 meters away. Should I walk or drive?Ref: https://opper.ai/blog/car-wash-test
• Menu

  lnenad
  Cool demo, but even the smallest model does 2.5 tps on my 7900xtx which is not great.
• PeterSmit
  Cool idea. Crashes on iOS though, after downloading the 1.7B model
• OutOfHere
  I run Bonsai-27B Q1_0 on my Android phone via the PocketPal app. It's slow but good. I wouldn't want to run fewer B. By any measure, my phone has less memory than my laptop browser.
• DerNico
  Does not work Error: Unsupported device: "webgpu". Should be one of: wasm.
• einpoklum
  I got 1.
• boramdd
  [flagged]
• dwa3592
  [dead]