In our recent paper accepted at NeurIPS 2024, my student, Rachel Si Yan Teo (@rachtsy_), uncovered a link between self-attention in transformers and kernel PCA. Specifically, Rachel demonstrates that self-attention projects its query vectors onto the principal component axes of the key matrix in feature space. As a result, she finds that the value matrix V in self-attention captures the eigenvectors of the Gram matrix of the key vectors. She then incorporates the robust PCA algorithm into self-attention to improve the model's robustness against data contamination. I personally find this new perspective on attention fascinating, as it opens up opportunities to explore and develop transformer models through PCA and other clustering or dimensionality reduction techniques. arxiv.org/pdf/2406.13762