HN

Comments (43)

• Propelloni
  Great work! I still think that [1] does a better job of helping us understand how GPT and LLM work, but yours is funnier.Then, some criticism. I probably don't get it, but I think the HN headline does your project a disservice. Your project does not demystify anything (see below) and it diverges from your project's claim, too. Furthermore, I think you claim too much on your github. "This project exists to show that training your own language model is not magic." and then just posts a few command line statements to execute. Yeah, running a mail server is not magic, just apt-get install exim4. So, code. Looking at train_guppylm.ipynb and, oh, it's PyTorch again. I'm better off reading [2] if I'm looking into that (I know, it is a published book, but I maintain my point).So, in short, it does not help the initiated or the uninitiated. For the initiated it needs more detail for it to be useful, the uninitiated more context for it to be understood. Still a fun project, even if oversold.[1] https://spreadsheets-are-all-you-need.ai/ [2] https://github.com/rasbt/LLMs-from-scratch
• totetsu
  https://bbycroft.net/llm has 3d Visualization of tiny example LLM layers that do a very good job at showing what is going on (https://news.ycombinator.com/item?id=38505211)
• Elengal
  Cool
• ordinarily
  It's genuinely a great introduction to LLMs. I built my own awhile ago based off Milton's Paradise Lost: https://www.wvrk.org/works/milton
• Menu

  mudkipdev
  This is probably a consequence of the training data being fully lowercase:You> hello Guppy> hi. did you bring micro pellets.You> HELLO Guppy> i don't know what it means but it's mine.
• hackerman70000
  Finally an LLM that's honest about its world model. "The meaning of life is food" is arguably less wrong than what you get from models 10,000x larger
• bblb
  Could it be possible to train LLM only through the chat messages without any other data or input?If Guppy doesn't know regular expressions yet, could I teach it to it just by conversation? It's a fish so it wouldn't probably understand much about my blabbing, but would be interesting to give it a try.Or is there some hard architectural limit in the current LLM's, that the training needs to be done offline and with fairly large training set.
• ben8bit
  This is really great! I've been wanting to do something similar for a while.
• Menu

  zwaps
  I like the idea, just that the examples are reproduced from the training data set.How does it handle unknown queries?
• Menu

  gdzie-jest-sol
  * How creating dataset? I download it but it is commpresed in binary format.* How training. In cloud or in my own dev* How creating a gguf
• Menu

  cbdevidal
  > you're my favorite big shape. my mouth are happy when you're here.Laughed loudly :-D
• ankitsanghi
  Love it! I think it's important to understand how the tools we use (and will only increasingly use) work under the hood.
• kaipereira
  This is so cool! I'd love to see a write-up on how made it, and what you referenced because designing neural networks always feel like a maze ;)
• kubrador
  how's it handle longer context or does it start hallucinating after like 2 sentences? curious what the ceiling is before the 9M params
• cpldcpu
  Love it! Great idea for the dataset.
• Menu

  gnarlouse
  I... wow, you made an LLM that can actually tell jokes?
• Menu

  brcmthrowaway
  Why are there so many dead comments from new accounts?
• NyxVox
  Hm, I can actually try the training on my GPU. One of the things I want to try next. Maybe a bit more complex than a fish :)
• SilentM68
  Would have been funny if it were called "DORY" due to memory recall issues of the fish vs LLMs similar recall issues :)
• Menu

  rclkrtrzckr
  I could fork it and create TrumpLM. Not a big leap, I suppose.
• monksy
  Is this a reference from the Bobiverse?
• Menu

  AndrewKemendo
  I love these kinds of educational implementations.I want to really praise the (unintentional?) nod to Nagel, by limiting capabilities to representation of a fish, the user is immediately able to understand the constraints. It can only talk like a fish cause it’s very simpleEspecially compared to public models, thats a really simple correspondence to grok intuitively (small LLM > only as verbose as a fish, larger LLM > more verbose) so kudos to the author for making that simple and fun.
• martmulx
  How much training data did you end up needing for the fish personality to feel coherent? Curious what the minimum viable dataset looks like for something like this.
• nullbyte808
  Adorable! Maybe a personality that speaks in emojis?
• oyebenny
  Neat!
• anon
  undefined
• peifeng07
  [dead]
• zephyrwhimsy
  [dead]
• Morpheus_Matrix
  [flagged]
• Morpheus_Matrix
  [dead]
• Alexzoofficial
  [flagged]
• ethanmacavoy
  [flagged]
• agenexus
  [flagged]
• techpulselab
  [dead]
• george_belsky
  [dead]
• aesopturtle
  [flagged]
• weiyong1024
  [flagged]
• aditya7303011
  [dead]
• aditya7303011
  [dead]
• Menu

  LeonTing1010
  [flagged]
• Menu

  jiusanzhou
  [flagged]
• dinkumthinkum
  I think this is a nice project because it is end to end and serves its goal well. Good job! It's a good example how someone might do something similar for a specific purpose. There are other visualizers that explain different aspects of LLMs but this is a good applied example.