I made a video! It introduces superposition in neural networks, and shows how to construct weight matrices for superposition in a simple autoencoder. (1/3)
Just updated my homepage for the first time since “What (if any) are the limits of large deep learning models?” was an interesting question
2
240
Training some tiny ML models on one-hot arithmetic. I hadn't realized all of these can be done with a single matrix multiplication!
1
183
ML engineers: Alright, our compute budget is 5e23 flops, so let’s just plug that into Kaplan et al. (2020), and—
Offscreen: STOP, IN THE NAME OF THE CHINCHILLA LAW.
Replying to @kanjun
I doubt Steve Jobs would have framed it this way, but “faith in connecting the dots” can be a sort of utilitarian attitude. youtu.be/Hd_ptbiPoXM?t=640
1
Replying to @_ryanbloom @MetaDevo
Even though it’s hard to step through and simulate what the program is doing, the output is often informative by itself. This image gives a pretty good hint as to what went wrong with my polar transform, no printf needed.
1
1
Replying to @BugRib @AmandaAskell
Sorry, that schematic presented an oversimplified view of cognition. Below is the complete version; it should only take a few more lines of code.
Replying to @AmandaAskell
This is why cognitive architectures are the only realistic path to AGI



























