HN

Comments (37)

• ketothekingdom
  Hey author here! I have a short summary here if people find it helpful: https://x.com/heyyalexwang/status/2111158818730397932?s=46
• vatsachak
  Great article. My favorite fact about "transformers as a hopfield network" is that transformers are so good at memorizing that they only need one step to find a suitable memory as opposed to multiple steps for a vanilla hopfield network
• Menu

  yoz-y
  Would be nice if articles defined the requirements to be a “you”.
• Menu

  dullcrisp
  For anyone here who understands this stuff, help me out because the article dropped me at the end when drawing the connection from memory to LLMs.For an LLM, the keys are the tokens in your context, and the values are … what exactly?
• femto
  "interference-to-signal ratio", "capacity", ... These terms, and their context, are things a communications/electrical engineer would understand. I encourage comms engineers to start learning how AI works under the hood. Chances are, a lot of what you know about maximising channel capacity and throughput is applicable to AI (can be seen as a channel with memory?).
• hum3hum3
  The animated bird, the kea is absolutely fabulous. I appreciate the effort and detail that went into this.
• sailfast
  This is a really interesting article! While I can’t profess to understand all the math it is a great way to demonstrate it.I recall awhile back hearing someone ask whether intelligence was actually the same or similar to “compression” or storing all of this knowledge in weights - IE the ability to create something from reason or weights or relationships / connections. This article makes it even more interesting!
• kooi
  Goodness, someone posts a great article which IMO accessible math, and people get upset.If you think that is crazy, ohh boy, just wait til you see all the other papers underlying LLMs, AI, and optimization in general.I appreciate this post because starting from linear algebra is the correct way to form an intuition of the higher dimensional semantic spaces we're dealing with.
• andai
  See also:NNCP: Lossless Data Compression with Neural Networks (2019)https://news.ycombinator.com/item?id=27244004https://bellard.org/nncp/
• 361994752
  Who? Me?
• throwaway613746
  [dead]
• Menu

  craftkiller
  This article does two things which make it unhelpful for me:1. It uses math notation. This is a software development site, wouldn't this be much more clear to this audience written as psuedo code instead of a complex series of greek and latin letters?2. It doesn't step through a concrete example. What is hollow R raised to the (M * d_v) ? Is it a number, a set, a list, a matrix?To use an example I understand: RAG used to seem complicated to me when only math people were talking about it in math words. For example, let's take a line from wikipedia on embeddings:> the representation is a real-valued vector that encodes the meaning of the word in such a way that the words that are closer in the vector space are expected to be similar in meaningWhat made it clear was learning that generating an embedding is just a function(string) -> list[float], and RAG is just taking a list[float] for your prompt and finding other list[float] that are close to it in your dataset.So what would make this article more clear, is step through a simplified example of putting a value into a memory store and querying that memory store, showing the actual data structures.