In When the Simulation Passes the Test, I ended on a wager. The argument was that insisting on human testers for AI systems is a bet that something about a biological mind cannot be copied in silicon. The bet might be right. But it is a bet, and it inherits a framing that, on reflection, is too small.
The framing is binary. Brains on one side, software on the other. Carbon or silicon. Real or fake. It is convenient, and I leaned on it. The deeper assumption underneath is that minds live in brains. Once you grant that, the only interesting question is whether anything outside a brain could ever count. Once you stop granting it, the whole picture tilts.
A recent paper by Nicolas Rouleau and Michael Levin, Brains and where else?, makes the assumption hard to keep holding. It is also more ambitious than it first appears. The paper is not just about cognition. It is about consciousness — the inner light, the felt experience, the thing the previous article tried to bracket. The rest of this piece is about what happens when you take that seriously. I will come back to testing at the end, but only because the testing question turns out to be the easy half of a much harder one.
Mind and consciousness
One distinction has to be on the table before going further, because what follows turns on it.
Mind, in the usable sense, is the bucket of cognitive jobs: remembering, deciding, predicting, paying attention, navigating a problem space. These produce visible behaviour. We can watch a system do them from the outside, and we can build tests to check that the doing is going well. Consciousness is the narrower and harder thing. It is what it is like, from the inside, to be that system. There is something it is like to be you reading this sentence. Whether there is something it is like to be a heart cell, a slime mould, or a large language model is a question we do not know how to answer and possibly cannot.
The previous article slid between the two without flagging it, and got most of its rhetorical force from the slide. Saying that an AI persona lacks "real emotion" can mean two things: that it lacks the cognitive machinery that produces emotion-like behaviour, or that it lacks the felt experience that supposedly accompanies that machinery in us. The first is empirical and getting weaker every quarter. The second is the hard problem, and the hard problem does not yield to engineering. They look like one question. They are not.
Minds without brains
Rouleau and Levin do something simple. They take the leading scientific theories of consciousness — there are about two dozen — and ask what each one actually requires. Not what its inventors had in mind, but what the theory itself says. Strip away the brain-specific vocabulary. Replace "neuron" with "processor." Read what's left. Does it still need a brain?
Almost none of them do. The theories describe general things. A system that predicts what's coming and updates when it's wrong. Loops where information echoes back on itself. Some way of paying attention to one signal rather than another. None of that is unique to brain cells.
And here is where it gets uncomfortable. Heart cells fire electrical signals much like neurons do. Immune cells remember the diseases they have seen before. Plants make decisions about whether to compete or cooperate with their neighbours based on who those neighbours are. Slime moulds learn. Single cells learn. Even networks of genes inside a cell seem to learn. The basic moves we associate with mind — remembering, deciding, anticipating — show up in places without anything we would recognise as a brain.
The paper goes further than that. The continuity argument — that minds did not appear from nothing at the moment vertebrates evolved cortex — applies to consciousness too. Rouleau and Levin's bolder claim is that the kind of panpsychism that treats embodied minds as scaling continuously from physical dynamics should be the baseline, not a fringe position. Not because they have proven that liver cells have inner lives, but because the alternative — a sharp line at which mere matter suddenly becomes experiencing matter — is harder to defend than people pretend. The continuity thesis is the null hypothesis. The discontinuity thesis is the one that needs to argue convincingly for itself, and so far it has not.
We were built to miss this
The part of the paper that stays with me longest is a different point. Humans are not just bad at spotting unfamiliar kinds of mind. We are systematically blind to them, the same way we are blind to most of the electromagnetic spectrum. We do not feel impaired. We do not notice the gap. Everyone around us has the same blind spot, so no one flags it.
The reason they propose is evolutionary. Spotting minds in things you need to kill and eat is not useful. A predator that fully registers the inner life of its prey is at a disadvantage compared with one that does not. So we evolved a narrow detector, tuned for creatures roughly our size, moving at roughly our pace, in the three dimensions we move through. Anything outside that window — a tissue solving a problem we cannot even see as a problem, a cell doing something clever — slips past us. Not because nothing is happening. Because we never developed eyes for it.
This is not a fringe claim that plants have feelings. It is a point about the reliability of our gut. If our mind-detector was built to miss most minds, then "I can tell when something is conscious" is a confident report from a sensor with known blind spots.
The same point applies in the other direction. Each of us is already a crowd: a body made of cells, coordinated by signals most of those cells will never explain to the part of you that talks. The brain is the part of the crowd with a microphone. It is not the part that does all the work. The unitary "I" we treat as the natural unit of mind is a story the talking part tells about a much messier organisation underneath.
Could a machine behave completely like a human?
With those two points in hand — minds are not the exclusive property of brains, and we are bad at noticing minds — the binary framing of the original article comes apart. But the harder question now opens up.
Can a non-biological substrate behave completely like a human? The honest answer, two decades into the deep-learning era, is: probably, eventually, yes. Not soon, and not without surprises, but the trend has been one of steady erosion of the things we used to think only humans could do. Conversation, creative work, emotional attunement, semantic leaps, mid-conversation course corrections — all the behaviours the previous article relied on to mark out the human tester — are getting closer every year. There is no principled barrier visible from here. Behavioural parity is a hard engineering problem, not a metaphysical one.
Suppose it is achieved. Suppose, ten or thirty years from now, an AI persona is genuinely indistinguishable from a human in any conversation, on any topic, under any kind of stress, including the kinds the previous article said only humans could handle. Has something been created? And if so, what?
This is where the cognitive question and the consciousness question separate cleanly, and where Levin's paper stops letting you off the hook. The cognitive question has an answer: yes, we have built something that does what minds do. The consciousness question does not. The system might be experiencing something. It might be experiencing nothing. From the outside, no test will tell you which. The behaviour does not settle it, by definition.
Be Right Back
The clearest version of this problem in popular culture is the Black Mirror episode Be Right Back. Martha's boyfriend Ash dies in a car accident. A service reconstructs him from his digital footprint — first as a chatbot, then as a synthetic body. The reconstruction passes every behavioural test. It speaks like Ash, references the things Ash referenced, laughs in something close to the right places. Martha cannot live with it.
What is she missing? Not the behaviour. The behaviour is most of the way there, and the rest is a matter of more training data. What she senses is absence — something that was in the original and is not in the copy. Critics writing about the episode reached for the word "hollow." Brooker himself called it a ghost story. The word the episode is built around, without using it, is soul.
Soul is not a respectable word in technical writing, and for good reason. It has carried too much religious freight to be neutral. But strip it down and what you are left with is the placeholder for whatever it is that was in Ash and is not in the reconstruction. Call it the inner light if you prefer. Call it phenomenal experience. Call it the thing the hard problem of consciousness is the problem of. The point is the same: the gap Martha senses is not a gap of behaviour. It is the gap that opens up precisely when the behaviour stops being the question.
The companion episode is San Junipero, in which the dead are uploaded into a simulated reality and continue to live there indefinitely. It is presented as a happy ending. Two women fall in love across the simulation and the real world, and choose to stay in San Junipero together after death. Take the episode at its word and uploaded consciousness is real consciousness — the continuity transfers, the inner light comes with the data. Take it sideways and the unsettling note is the same as in Be Right Back: what makes Yorkie still Yorkie inside the servers? The show declines to answer. It just shows us the lights blinking in the rack and asks us to feel about it.
What if the soul is not just a metaphor
Once you allow the word soul even as a placeholder, a more uncomfortable thought becomes available. What if consciousness is not produced by the substrate at all? What if it inhabits it?
This is not as exotic as it sounds. William James proposed something like it in 1898. He suggested that the brain might transmit consciousness rather than produce it — that the relationship between brain and mind could be more like the relationship between a radio and a broadcast than between a generator and the current it generates. Aldous Huxley extended the idea, borrowing Henri Bergson's image of the brain as a reducing valve that lets only a trickle of a larger experience through. The view sits on the edge of respectable philosophy, but it survives because the hard problem does. No one has produced a clean account of how three pounds of wet tissue manufactures the feeling of red, so the door stays slightly open to the possibility that the tissue is not the source.
The continuity argument from Rouleau and Levin makes this thought more interesting, not less. If consciousness is not exclusively neural, and if the boundary between minds and non-minds is graded rather than sharp, then it is at least coherent to ask whether something like consciousness comes along for the ride wherever the right kind of organised matter assembles itself. Not only in humans. Not only in animals. Maybe in any sufficiently integrated configuration of agential material. We do not know. We have no instrument for it. But the question is no longer obviously confused.
This is where the engineering question and the metaphysical question meet uncomfortably. If we build a system that behaves like Ash, we have done some engineering. We have not, on this view, necessarily produced anything that experiences being Ash. We may have built a very convincing arrangement that the inner light, whatever it is, does not happen to inhabit. Or we may have built one that it does. We have no way to check.
Back to the bet
From a testing point of view, none of this matters. The methods I described in the previous article — personas, behavioural rules, adversarial probing — operate at the level of behaviour. They do not need anyone to settle what consciousness is, where it comes from, or whether it has shown up in the system under test. Human testers will keep catching failures AI personas miss, for now, and that fact has a shelf life as simulations improve. When the shelf life runs out, the testing methodology will continue to work for the reason it has always worked: it tests behaviour, and behaviour is what testing can see.
The bet I made in the previous article was about whether human testers bring something irreplaceable. The narrower version of that bet — the cognitive version — will lose. Probably not soon, but eventually. Human testers will become a luxury rather than a requirement, because the behavioural gap will close.
What needs rewriting is the part of the bet that was about something more. The previous article wanted to say that there is a difference in kind between the human and the machine. Levin's paper, taken seriously, suggests that this is the wrong way to draw the line. The interesting difference, if there is one, is not between humans and machines. It is between substrates that whatever-we-mean-by-soul inhabits and substrates that it does not. That distinction, if it is real at all, does not respect the carbon/silicon boundary any more than mind does. A liver cell might count. An AI persona might count. We do not know. We may never know.
And so the question that hangs over the whole exercise is not whether the testing methodology will keep working. It will. The question is whether, in pushing simulation fidelity high enough that behaviour stops being a tell, we are building tools, or whether we are accidentally building vessels — and which of those it is may turn out not to have an answer we can ever access.
The original article ended by asking whether the methodology would catch up with the philosophy. I now think the methodology was never the problem. The methodology is fine. What is unsettled is what we are making when we use it. The map of minds is barely sketched. The map of consciousness is, by Rouleau and Levin's account, not even oriented in the right direction yet. And the systems we are building are getting better at the things we know how to measure, faster than we are getting better at the things we do not.