“An intriguing open question is whether the LLM is actually using its internal model of reality to reason about that reality as it solves the robot navigation problem,” says Rinard. “While our results are consistent with the LLM using the model in this way, our experiments are not designed to answer this next question.”
The paper, “Emergent Representations of Program Semantics in Language Models Trained on Programs” can be found here.
Abstract
We present evidence that language models (LMs) of code can learn to represent the formal semantics of programs, despite being trained only to perform next-token prediction. Specifically, we train a Transformer model on a synthetic corpus of programs written in a domain-specific language for navigating 2D grid world environments. Each program in the corpus is preceded by a (partial) specification in the form of several input-output grid world states. Despite providing no further inductive biases, we find that a probing classifier is able to extract increasingly accurate representations of the unobserved, intermediate grid world states from the LM hidden states over the course of training, suggesting the LM acquires an emergent ability to interpret programs in the formal sense. We also develop a novel interventional baseline that enables us to disambiguate what is represented by the LM as opposed to learned by the probe. We anticipate that this technique may be generally applicable to a broad range of semantic probing experiments. In summary, this paper does not propose any new techniques for training LMs of code, but develops an experimental framework for and provides insights into the acquisition and representation of formal semantics in statistical models of code.
Exactly. If we give an LLM with no training data a large group of specimens, will it organize them into logical groups? Does it even understand the concept of organizing things into discrete groups?
That’s something that’s largely encoded into our brain structures due to millennia of evolution (or creation, take your pick) where such organization is advantageous. The LLM would only do it if we indicated that such organization is advantageous, and even then would only do it if we gave it a desired output. An LLM will only reflect the priorities of its creator, or at least the priorities baked in to the training data. It’s not going to suggest that something else entirely be considered, because it only considers things from the lenses we give it.
Humans will question assumptions, will organize things without being prompted, and will generate our own priorities. I firmly believe an LLM cannot, and thus cannot be considered self-deterministic, and thus not sentient. All it can do is optimize for the priorities we give it, and while it may do that in surprising ways, that doesn’t mean there’s “thinking” going on, just that it’s a complex system we don’t fully understand (even if we created it). Maybe human brains work in a similar way (i.e. completely deterministic given a specific genome and “training data”), but we know LLMs work that way, so until we prove that humans work similarly, we cannot equate them. It’s kind of like the P = NP question, we know LLMs are deterministic, we don’t know if humans are. So the question isn’t “can LLMs think” (we know they can’t), but “can humans think.”