How Physics Might Help Understand Rogue AI

AI doesn’t always do what you think it’s doing.

When you use a program like ChatGPT, it uses a framework called a Large Language Model, or LLM, to decide what to say next. LLMs view text in terms of tokens, roughly word-sized chunks representing common ideas and parts of speech. Those tokens are expressed in terms of mathematical vectors via an embedding procedure. Each time the LLM is used to infer the best choice for the next word in the sentence, those vectors are multiplied by a series of matrices arranged in layers, the contents of each matrix determined by numbers called weights. When engineers “train” an LLM, those weights are adjusted, with changes “back-propagated” through the matrices to make the predictions as accurate as they can be. 

The result is impressive, but risky. The weights of an LLM encode rules of thumb, and it’s often unclear what those rules are or how much we can trust them. Current “chain of thought” models try to write down steps of reasoning before they answer, but new models are increasingly reasoning without writing anything down. That makes it harder to trust their output, since chain-of-thought models sometimes write explicitly that they will deceive the user.

“So you can actually see it being deceptive,” said Dmitry Vaintrob.

Vaintrob works for Principles of Intelligence (formerly PIBBSS), an organization that funds research to better understand AI, called interpretability research. Last year, his team reached out to physicists, looking for insight. They are putting their hopes in ideas that physicists use to understand complex systems, which let them find which details matter most. For them, supporting research based on these ideas seems like the best path to let humans probe the minds of future AI.

‍

‍

The Horizon Scanning Team looks for new ideas to keep AI under control

As AI gets more and more intelligent, organizations like Principles of Intelligence get more worried. They are especially worried about the possibility that one day someone will invent an AI that is smarter than even the smartest humans. Such an AI might propose ideas that superficially seem great, but are secretly based on apocalyptic goals, which if acted on would doom humanity. If we can better understand which rules of thumb an AI uses to make its decisions, we have a chance of catching those goals before we act on them.

Historically, this kind of “AI safety” research has followed two paths. Some approach the problem very abstractly, trying to imagine from first principles what a future AI might do. Others are more empirical, looking at existing AI systems and doing experiments. By reaching out to other academic fields, Principles of Intelligence combines the two approaches.

“The way we see our role is trying to bridge this gap a little bit. So injecting some theoretical principles into empirically focused work,” said Lauren Greenspan.

Greenspan and Vaintrob worked on Principles of Intelligence’s Horizon Scanning Initiative, along with the organization’s Executive Director of Research, Lucas Teixeira. While the organization’s other initiatives fund research fields with an established track record of contributing to AI safety, the Horizon Scanning Initiative is designed to keep an eye on the horizon, looking for unusual research with the potential to solve key problems.

“What we do is try to look at bottlenecks as we see them within AI safety research and then think about different fields from outside of AI safety which can fill that gap,” said Greenspan.

It’s a pursuit that blends science and sociology, where researchers have studied how different fields forge connections.

“There's this idea of a trading zone, which is that you have two cultures that are meeting and maybe they don't understand each other's languages,” said Vaintrob, “We are interested in finding interesting fields to trade with.”

Renormalization lets physicists find what’s most important

One of those fields is physics, where an idea called renormalization has a role in understanding how AI thinks.

Physicists use renormalization to study how nature changes with scale, both for fundamental particles and for collective behavior of atoms in materials. Zoom in, and you might see a wealth of detail, with atoms made of protons and neutrons, and protons and neutrons made of quarks. But if you zoom out, only some of those details matter. Renormalization lets you “coarse-grain”, ignoring details smaller than a certain scale by averaging them out, and summarizing the result in terms of a new physical model. Physicists call the details that survive this process “relevant”, and using the mathematics of renormalization they can tease out the relevant details from the irrelevant ones using differential equations that describe how different physical effects vary as you tweak the zoom lens. Through this procedure, physicists can find what kind of “particles” are visible in the zoomed-out theory, pinning down which elements of the model survive and how they change.

Renormalization can help understand AI in a few different ways. It can be used to study how a specific model learns during training, taking input data and finding the most relevant patterns. It can be used to study how a trained model makes predictions, how it takes a question and processes it to produce an answer. And perhaps most interestingly for Principles of Intelligence, it can be used to compare different AI models, finding the essential traits, analogous to particles, that could cause them to behave differently. Potentially, it could give rigorous proofs about how the model will behave.

The hope is to get a tool that can say, “Well, this component of the model is an atom. And this component is adding two numbers together,” said Vaintrob. If we had a tool like that for future models, we could investigate whether a model was doing something that would lead to dangerous output.

Building such tools will be challenging. When physicists use renormalization to study AI, they often work layer by layer, showing how the effect of deeper layers of a neural network can be represented instead as additional connections between nodes in earlier layers, summarizing a model’s behavior in terms of how it responds to input, rather than the details of what happens deep inside. Currently, physicists have had the most success studying AI models that are comparatively “wide” rather than “deep”, with fewer, broader layers. In physics, these models correspond to theories where the forces of interaction between particles are comparatively weak, as there are fewer deep layers to take into account with new links. But physicists do have tricks to understand strong interaction forces, and they are  to use those tricks to understand deeper AI models.

The Horizon Scanning Team is hopeful. They have been providing support and encouragement for physicists, inspiring new collaborations and motivating the new community of physicists who study AI to solve what they view as the most critical questions we face. Their research program has a name: PIAMI, Physics-Informed Ambitious Mechanistic Interpretability. That “Ambitious” is important: they want physicists to not be content with concepts that work in-principle or partway, but to find ideas that can go all the way, and make AI fully transparent.

“There's this growing group of physicists who are interested,” said Greenspan, “We want to take that intuition and make it a little bit more applied, and explicitly relevant to AI safety.”

The same question, asked elsewhere

This isn't the only place the same problem shows up. Anywhere researchers lean on a model to reason through something they can't fully verify by hand, the same question follows: how do you know the answer is right, and not just fluent? Principles of Intelligence is asking that question about the models themselves, using physics to look inside the machine. Other groups are asking about the research process that sits around the machine, building environments where a model's output has to survive comparison before anyone treats it as a result.

FirstPrinciples built its Theo platform around that second version of the problem. Rather than asking one model for an answer and moving on, Theo puts several models to work on the same question at once, so a researcher can see where the models agree, where their reasoning diverges. Results get checked against symbolic and numerical verification before they're treated as settled, and every step stays visible enough to retrace later. It's a different layer of the same instinct that sent Principles of Intelligence looking outside AI safety for new tools. Confident output and correct output are not the same thing, and the fields making real progress right now are the ones building that gap into their process rather than papering over it.

‍