The Benchmark That Lost Its Theory of Mind
A company reports that its language model scores 91% on a false-belief benchmark built from Sally–Anne style vignettes and 88% on a second-order belief task, and its press release states that the model "has a theory of mind." An independent lab replicates the scores and then runs three probes. (i) When character names are swapped and sentences reordered while preserving the logical structure, accuracy falls to 54%. (ii) An ablation that removes a single token-level shortcut - the word "surprised" appearing in the prompt - drops accuracy to 57%. (iii) On control vignettes about physical causation rather than mental states, the model performs at chance. The company replies that the model is substrate-independent, so demanding biological neurons is question-begging, and that passing a benchmark designed to detect the capacity is as good evidence as we ever have for that capacity in any system.
1. Assess the claim that the model "has a theory of mind." Use the distinction between behavioural performance and underlying competence.
Read the worked answer
The claim equivocates between two things: a behavioural profile (scores on a distribution of prompts) and a competence (an internally organized capacity to represent beliefs as beliefs, to track false belief, and to use that representation in inference). The benchmark measures the first. The reordering probe is the decisive evidence: preserving logical structure while changing surface form is exactly the manipulation that a genuine competence should survive, and accuracy collapsing from 91% to 54% shows the original score was carried by surface features rather than structure. That is a performance artifact, not a competence.
Turing's imitation game is a behavioural criterion and cannot, by itself, distinguish these. That is not a defect of the test but a limit on what it licenses: passing licenses "indistinguishable from a mind in this limited context," not "has the mind." Searle's point is stronger here - syntax is not sufficient for semantics, and the benchmark is a syntax-level test.
The correct conclusion is deflationary: the model's behaviour is consistent with false-belief attribution on the training distribution and is not robust to paraphrase, so the evidence supports a pattern-matching description, not theory of mind.
2. Does the ablation evidence bear on the "system reply" to Searle's Chinese Room? Explain what the reply requires and whether it is available here.
Read the worked answer
The system reply concedes that the person in the room does not understand Chinese, but argues that the system as a whole - the person plus the rulebook plus the paper - does, because understanding is a property of the whole organization and not of any component. The reply's force depends on the system's internal states being organized in the right way: causally connected to one another and to the world so that they function as representations rather than as a lookup table.
The ablation is relevant precisely because it is evidence about that organization. Removing the token "surprised" drops accuracy by 34 points, which shows that the model's "belief attribution" is carried by a lexical shortcut rather than by an internal state tracking the character's epistemic situation. If the causal structure is a surface-feature detector, then the system reply is not available: the reply needs the whole to have the right functional organization, and the probe suggests the whole does not. The reply is not refuted in general - a system could have such organization - but it cannot be used to rescue this claim.
The step from "this ablation shows a shortcut" to "no system of this kind could understand" is not licensed, and the company would be right to resist that broader inference.
3. Evaluate the multiple realizability defence. What does it establish, and what does it not?
Read the worked answer
Multiple realizability is the thesis that a given mental state can be realized in more than one physical substrate, so that identifying mentality with biological neurons is a mistake. It is well motivated: if pain is realized in human C-fibres and in octopus circuitry, then the type "pain" is not identical to any one physical type.
What it establishes is a negative, modal claim: it blocks the inference "this system is not made of neurons, therefore it cannot think." The company is entitled to that. It does not establish that any particular non-biological system does realize a mental state, because realizability is a possibility constraint, not a positive criterion. To say this model realizes theory of mind, you need a theory of what plays the functional role and evidence that this system plays it - the standard functionalist demand that the internal states be appropriately causally connected. The ablation evidence bears directly on that question and counts against it.
Multiple realizability is therefore a defence against a bad argument, not an argument for the conclusion. The chance performance on physical-causation controls is a further caution: it suggests the model may have a narrow text-pattern competence rather than a general model of agents, though training-data distribution could also explain it, so it is weak evidence either way.