<- Back
Comments (19)
- MCP123This is great, thank you. I have to teach this stuff on Friday so perfect timing. It's hard to explain the attention mechanism in a way that becomes intuitive because the weighting scheme does not help much with the intuition. Having a visualization like this helps a lot. Don't move that page please since I'll link to it!
- fuddleThis is great, I've read multiple books and watched videos about the attention mechanism. Now that I understand it, this is the clearest example I've seen on how attention works.
- wopakneat, combining info from two phrases is hard to see without such a tool.are you worried later-layer attention gets drowned out by earlier layers just because there are more of them contributing to the sum?
- itsnasmeI like the visualisation. Pretty cool
- sva_I highly question this simplistic idea of high vector magnitude = high influence.
- fermlon30000INSANE
- ex-aws-dudeI don't know much about LLMs but does that mean you have N^2 computation with the context size since every token needs to track how it relates to every other token?
- staredI am curious what's the actual formula.I mean, there so many headers and layers, it is tricky to make a choice that will resonate with our intuition . Is it some weighted average? Or maybe ablation test?
- colophontio[dead]
- Yyylov[dead]