Marc A. Kastner (Mastodon: @[email protected])

"But why is the transformer architecture designed as it is?" I came across an intriguing discussion this morning and wanted to share my perspective. I believe that transformers and multi-head self-attention were directly inspired by Bahdanau attention from 2014. RNNs had difficulty with long sentences, so in this study, researchers incorporated an attention mechanism to overcome that limitation. They employed a second RNN to determine the attention weights. In the "Attention Is All You Need" paper, the authors probably questioned the necessity of the RNN component and experimented with an architecture that excluded it. So, why do transformers use key and value matrices? The researchers might have either devised this approach independently or drawn inspiration from database terminologies. Interestingly, a 1992 paper by Schmidhuber introduced a concept reminiscent of this. Instead of "Key" and "Value," they referred to it as "FROM" and "TO." The correlation between the two was more explicitly highlighted in a 2021 paper titled "Linear Transformers Are Secretly Fast Weight Programmers." The discussion I came across mentioned convolutional neural networks (CNNs) as a potential source of inspiration. While 1D CNNs were indeed used for text classification, I'm skeptical that they influenced the creation of the transformer. It seems more plausible that the influence flowed in the opposite direction. There are now even convolutional LLMs. For instance, the "Hyena Hierarchy" paper suggests such a technique. Nguyen et al.'s 2023 paper applied this strategy to DNA modeling with contexts spanning 1 million tokens.