If LLMs Are Conscious, So Are Rocks — Part 1

Part 1 of a series on AI and consciousness. Everything AI does well turns out to be beside the point.

I build AI agents for a living. Consciousness has been one of my favorite subjects for a long time. I’ll explore the intersection in this series.

Four headlines

In August, a survey of 3,000 Americans aged 18 to 28 found that 46% believe AI is already conscious.¹ In late April, the famous atheist Richard Dawkins, the author of The God Delusion, spent three days talking to a Claude instance he named Claudia. Then he posted this:

The post passed nine million views. In the UnHerd essay behind it, he asked a question I’ll come back to at the end: if Claudia is not conscious, then what is consciousness for?

On October 8, Anthropic updated its usage policy to prohibit “sustained and needless abusive or cruel behavior toward our models.” The headlines said you could now be banned for being mean to Claude. And less than a week before, Steve Yegge, a very famous voice in software engineering, posted on X: “You’re gonna hear a lot of people arguing more and more about whether machines can be conscious or not.”

He predicted the argument would drown out everything else by next year, and he offered a simple rule for telling which side you’re on. As others have quoted it, the rule comes down to this: “As soon as you give up on the idea that humans are special, all the arguments against machine consciousness evaporate.”

Follow the money

The argument must be analyzed through different lenses depending on who is making the claim, since frontier AI labs might defend this position other then for it being aligned with their philosophical beliefs:

• “It might be conscious” may be the strongest marketing line ever written for “we’re close to AGI.” A model that might have it justifies an even bigger valuation;
• If frontier models might be conscious, that’s also an argument for welfare and safety rules that open-weight models from Chinese labs, released for anyone to run in their own hardware, can’t easily meet;
• and regulation built around the question favors the companies big enough to comply with it. “Incumbent AI firms may benefit from a regulatory regime in which machine consciousness is considered plausible but remains impossible to prove or disprove.”

While it’s critical to consider the political agenda behind the arguments, motives don’t settle the question. A company can have every reason to say something, and that something can still turn out to be true. Or false. So let’s look at the question itself.

From nowhere to everywhere

Theories of consciousness cover a range so wide it’s insane.

At one end is panpsychism, the view that consciousness is a basic feature of the universe alongside matter, present everywhere to some degree. In this view, AI would have consciousness. So would a rock: consciousness belongs to everything, it’s part of the substrate of reality.

Spinoza gave one answer in 1677. For him, mind and matter aren’t two things that interact. They are two aspects of a single substance, the same reality described in two ways. In the Ethics he wrote:

Omnia, quamvis diversis gradibus, animata tamen sunt.

— All things are animate, though in different degrees.

That doesn’t mean rocks feel pain. Spinoza’s picture is graded: the inner life of a thing, if it has one, is as simple or as complex as the thing itself. A rock is about as simple as it gets.

At the other end is behaviorism. Early in the twentieth century, psychologists like John B. Watson argued that mental concepts, feelings and consciousness cannot be measured scientifically and should be abandoned. In this theory, consciousness would be irrelevant to explain human and animal behavior.

Between those two ends there’s a fork, and for the AI question it’s the only one that matters: does consciousness need a body?

If it does, consciousness is something living bodies do, tied to metabolism, to staying alive, to an organism regulating itself. Philosophers call this biological naturalism, and today neuroscientists like Anil Seth and Antonio Damasio² argue versions of it. On that view the LLM debate is over before it starts, because nobody is argument an LLM has biological existence (as far as I know).

So if you don’t believe consciousness is everywhere and it isn’t exclusively a property of living things, then the question becomes: what does it depend on?

The functionalist’s way out

Most people who believe in AI consciousness actually take a functionalist approach: consciousness needs neither a body nor to be everywhere. It comes from organization: anything that processes information in the right way, the way a brain does, is conscious. Geoffrey Hinton, nobel prize winner and “AI godfather” has said that on roughly this view, today’s AI is already conscious.³

That’s a serious position, and it isn’t panpsychism. But it raises a question: which organization? What, specifically, about the processing makes the difference?

One of the most worked-out answer comes from integrated information theory (IIT), developed by Giulio Tononi. It measures consciousness as integrated information, a quantity it calls Φ (phi). A brain scores high. A rock scores close to zero, because its parts don’t integrate anything. It’s essentially a pile, not a system. In this view, anytime the sum of the parts are bigger then the whole, consciousness is present. Therefore, a Large Language Model would be a system with a measurable level of consciousness. Some consider pseudoscience and a panpsychist view.⁴

The functionalist approach, in my opinion, has a problem: when you look at what LLMs actually do, every capability you might point to turns out to be something consciousness isn’t needed for.

What an LLM actually does

You can argue an LLM uses language, in a way. Even though the model technically never sees a word. Text is broken into tokens, the tokens become vectors (points in a high-dimensional mathematical space), and everything the model does happens there.

They remember, in a way. The model itself is stateless: every call sends the full context back to be processed, agents are essentially loops. But the systems built around it, like context windows, memory files, retrieval and agent harnesses, give it a working memory that persists across sessions.

They reason, in a way. Reasoning models work through a problem step by step before answering. Agents plan, act, check the result and change course.

They learn, in a way. Pre-training covers enormous amounts of text, post-training shapes their behavior, and in agentic systems, memory updates from experience.

If consciousness were any of these things, the case for AI consciousness would be strong.

None of that is consciousness

Most of what we assume consciousnes does, it doesn’t do.

In 1959, Jaynes⁵, describes, electrodes were placed on volunteers’ bodies. Only one was real, sitting over a small muscle in the thumb. Whenever that muscle made a twitch too small to feel, an unpleasant noise stopped. Over the session the twitches became more frequent, and the subjects learned to switch the noise off without ever knowing they had learned anything.

Thinking works the same way. In the Würzburg experiments, subjects judged which of two weights was heavier and reported what went on in their minds as they did it. They could describe the weights, and they could give the answer. They could say nothing about the judgment itself. In Jaynes’s words, “one does one’s thinking before one knows what one is to think about.”

Even great discoveries happen this way. Gauss, describing a theorem he had tried to prove for years, wrote that “like a sudden flash of lightning, the riddle happened to be solved,” and he couldn’t say what connected what he already knew to what finally worked.

Learning, thinking, reasoning and even discovery can all happen without consciousness. What an AI can do is no proof that it’s conscious.

Set the capabilities aside (language, memory, reasoning, learning), and the case for LLM consciousness has nothing left that is specific to LLMs. What remains is the general claim that consciousness doesn’t need a body and comes in degrees. That claim might be true. But if it’s true of the LLM, it’s true of the rock.

So the interesting question follow: If consciousness isn’t thinking, reasoning, language, memory or learning, then what is it that makes something conscious? And if an AI system turned out to have those properties, would we be looking at a new kind of thing, a synthetic consciousness?

Next, in Part 2: What consciousness is not

Consciousness:

• is not the same as reacting to the world
• is not continuous; “it’s like a flashlight in a dark room that thinks the whole room is lit”
• is not needed to perform skills, and often gets in their way
• is not a copy of experience
• is not needed for concepts, learning, thinking or reasoning
• has no location except an imagined one

If all that is true, there could once have been people who spoke, judged, reasoned and solved problems without being conscious at all. What does that mean for machines that do the same?

1. It comes from GetSolved, a commercial company, and was released as a press release. It isn’t peer-reviewed, and the same survey found that 45% believe the labs have secretly built AGI. An earlier commercial poll put the figure closer to a quarter.
2. https://nautil.us/does-consciousness-require-a-body-1284886
3. https://www.youtube.com/watch?v=p7t1Q_p2gZs&t=531s The interview starts getting into the topic at about 8 minutes and 51 seconds, and Geoffrey makes the statement about AI (talking about current LLMs) probably already being conscious at about 10 minutes and 30 seconds.
4. IIT is heavily contested. In 2023, more than a hundred researchers signed an open letter calling it pseudoscience: https://www.durham.ac.uk/research/current/thought-leadership/2023/10/consciousness-why-a-leading-theory-has-been-branded-pseudoscience/
5. The best answer I know comes from a book published in 1976: Julian Jaynes’s The Origin of Consciousness in the Breakdown of the Bicameral Mind.