![Overview of the paper. Title: The Emergent Symbolic Structure of Artificial Neural Networks Authors: Tom McCoy, Paul Soulos, Tal Linzen, Paul Smolensky Left: Neural networks encode information in vectors (there is then an image of a vector), yet they excel at tasks long thought to require symbolic structure (there is then an image of a symbolic representation, specifically a syntax tree). How do LLMs do it? Right: We find that LLM representations can be closely approximated with symbolic structures. This approximation lets us edit the structure of an LLM’s output by editing the structure of its internal representations, as shown. There is then an image of two edits to LLMs. In the first one, the original input is 3 + 6 * 8, with an answer of 51. But if we swap the positions of the 3 and the 6, the output becomes 30. In the second one, the original input is a Python command repeating the list [Z, U] three times, producing [Z, U, Z, U, Z, U]. But if we edit the input in a way that adds](https://pbs.twimg.com/media/HRJ5Jr5W0AAQ-sN.jpg?name=orig)
ALT Overview of the paper. Title: The Emergent Symbolic Structure of Artificial Neural Networks Authors: Tom McCoy, Paul Soulos, Tal Linzen, Paul Smolensky Left: Neural networks encode information in vectors (there is then an image of a vector), yet they excel at tasks long thought to require symbolic structure (there is then an image of a symbolic representation, specifically a syntax tree). How do LLMs do it? Right: We find that LLM representations can be closely approximated with symbolic structures. This approximation lets us edit the structure of an LLM’s output by editing the structure of its internal representations, as shown. There is then an image of two edits to LLMs. In the first one, the original input is 3 + 6 * 8, with an answer of 51. But if we swap the positions of the 3 and the 6, the output becomes 30. In the second one, the original input is a Python command repeating the list [Z, U] three times, producing [Z, U, Z, U, Z, U]. But if we edit the input in a way that adds
ALT Example tasks in LongProc.
ALT Top: A syntax tree for the sentence "the doctor by the lawyer saw the artist". Bottom: A continuous vector.
ALT Overhead view of Cornell Tech campus on Roosevelt Island
ALT Boston University's CDS building. (Official photo from BU CDS—not mine!)
ALT At the top is the title of the paper: "Embers of autoregression show how large language models are shaped by the problem they are trained to solve". Below on the left is a screenshot of ChatGPT being asked to count how many words are in a list. The correct answer is 29, but it says 30. Next to it is a plot showing ChatGPT's accuracy at counting elements in a list; in general, it does well on multiples of 10 but poorly on other numbers. The explanation offered at the bottom of the image is: In training sets, round numbers are much more common than other numbers.
ALT Plotting LLM accuracy on shift ciphers shows trends of genuine reasoning and shallow memorization.
