HN

Comments (28)

• y42
  shameless plug:A series of Jupyter notebooks explaining the whole machine learning mechanism, from the beginninghttps://github.com/nickyreinert/DeepLearning-with-PyTorch-fr...and of course also how to build an llm from scratchhttps://github.com/nickyreinert/basic-llm-with-pytorch/blob/...
• fabian_shipamax
  If someone is interested, I am giving short courses with walkthrough on how to train you LLM from scratch via AI Study Camp.
• Menu

  jvican
  If you're interested in this resource, I highly recommend checking out Stanford's CS336 class. It covers all this curriculum in a lot more depth, introduces you into a lot of theoretical aspects (scaling laws, intuitions) and systems thinking (kernel optimization/profiling). For this, you have to do the assignments, of course... https://cs336.stanford.edu/
• Menu

  NSUserDefaults
  Been doing it since the day I was born. The beginnings were hard but I’m getting there.
• JoeDaDude
  Coincidentally, I just started on Build a Large Language Model (From Scratch), a repo/book/course by Sebastian Raschka [0][1][2]. Maybe it is a good problem to have to have to decide which learning resource to use.[0] https://github.com/rasbt/LLMs-from-scratch[1] https://www.manning.com/books/build-a-large-language-model-f...[2] https://magazine.sebastianraschka.com/p/coding-llms-from-the...
• kriro
  I did it back in the day when fast.ai was relatively new with ULMFiT. This must have been when Bert was sota. The architecture allows you to train a base and specialize with a head. I used the entire Wikipedia for the base and then some GBs of tweets I had collected through the firehouse. I had access to a lab with 20 game dev computers. Must have been roughly GTX 2080s. One training cycle took about half a day for the tokenized Wikipedia so I hyper parameter tuned by running one different setting on each computer and then moving on with the winner as the starting point for the next day. It was always fun to come to work the next morning and check the results.The engineering was horrible and very ad-hoc but I learned a lot. Results were ok-ish (I classified tweets) but it gave me a good perspective on the sheer GPU power (and engineering challenges) one would need to do this seriously. I didn't fully grasp the potential of generating output but spent quite some time chuckling at generated tweets (was just curious to try it).
• antirez
  Context: he is one of the MLX developers, a skilled ML researcher.
• Menu

  ofsen
  This looks like exact copy of this video of andrej karpathy ( https://youtu.be/kCc8FmEb1nY ) but in a writing format, am i wrong ?
• steveharing1
  The documentation is really helpful enough to get started
• Menu

  hiroakiaizawa
  Nice. What scale does this realistically reach on a single machine?
• DeathArrow
  I would start with linear algebra, some calculus and statistics and understand how a neural network - which really is just one type of ML - works, the learn the basics of CNN and RNN, then learn transformers and LLM.But that is just me. I think is more useful to understand the how and whys before training a LLM.
• iamnotarobotman
  This looks great for a first introduction to training LLMs, and it looks simple enough to try this locally. Great job!
• Menu

  baalimago
  Train your LM from scratch*I doubt you have a machine big enough to make it "Large".
• yjaspar
  That’s actually super interesting
• anon
  undefined
• rithdmc
  I know it's a bit of a joke, but "I Built a Neural Network from Scratch in SCRATCH" gave me, a complete outsider, a lot of insight into how neural networks work.https://www.youtube.com/watch?v=5COUxxTRcL0
• flowdesktech
  [flagged]