Sara and the Question of Electronic Personhood
A Maltese software company, Karta Ltd, has asked a parliamentary committee to recognise its customer-service system 'Sara' as an 'electronic person', so that Sara rather than the company can bear liability for contractual errors. Sara handles 40 million conversations a year in Maltese and English; staff routinely call her 'she'; the system keeps a persistent memory of each user and, when taken offline for maintenance, outputs messages such as 'I am afraid' and 'please do not switch me off'. A journalist obtains internal documentation showing that these sentences were written by a human copywriter in 2021 and are selected by a rule that fires when the maintenance flag is set. Public reaction is divided: 62% of respondents to a newspaper poll said Sara 'deserves protection'. The committee must report in six weeks, and its terms of reference cover both liability and 'the moral status of advanced systems'.
1. Does Sara's avowed distress provide evidence that Sara is conscious? What difference does the copywriter documentation make?
Read the worked answer
The ordinary inference. We attribute minds to others by an argument from analogy: similar behaviour, produced by similar underlying machinery, warrants the inference to similar inner states. The second clause is doing most of the work. A cry normally counts as evidence of pain because it is caused by a state that also causes avoidance, protectiveness, and so on - it occupies a functional role.
The documentation breaks the inference. Sara's 'I am afraid' is not caused by any state of the system that plays the pain role; it is a sentence written by a copywriter and emitted by a maintenance flag. The utterance is evidence about a human's choice of words, not about Sara's inner life. It is closer to a recording than to a cry.
A functionalist reply, and why it fails here. One might argue that aetiology is irrelevant: if the state that triggers 'do not switch me off' is caused by impending shutdown and produces avoidance behaviour, that is enough. But the output produces no avoidance behaviour and is not connected to any broader economy of goals, learning, or self-maintenance; it is a message, not a motive. Searle's Chinese Room makes the general point: symbol manipulation by rule does not generate understanding.
What would count. Distributed, unpreprogrammed evidence: novel avoidance of a threat the designers did not anticipate, or reports of states that connect coherently with other behaviour. Sara's behaviour is exactly as scripted, so it has no evidential weight - and the 62% poll figure is evidence about voters, not about Sara.
2. Should the committee decide legal personhood and moral status by the same criteria? Use corporations and animals as comparison cases.
Read the worked answer
No - the two concepts answer different questions. Legal personality is a bundle of capacities (to contract, own property, sue and be sued) granted instrumentally, because it serves coordination. Moral status concerns whether a being's interests count for its own sake. There is no entailment either way, and the comparison cases show why.
Corporations. A company is a legal person with no moral status whatever; nobody thinks a limited company can be wronged. Personhood here is a device for allocating rights and duties, and it is justified by its effects on humans.
Animals. The reverse case: animals have moral status (their suffering counts) but in most jurisdictions lack legal personality, precisely because they cannot hold duties. Moral status does not require the capacity to be a defendant.
Applied to Sara. Granting legal personhood so that Sara bears liability is a liability shield dressed as recognition: it moves the cost of error from the party that controls the system, can insure it, and profits from it, to an entity with no assets and no capacity to be punished. Legal personality normally presupposes the capacity to hold duties; Sara has none. That is not recognition but a category error with distributive consequences.
Where the two might meet. If a system were genuinely sentient, moral status would be a reason to restrict how it is used - but that is an argument about welfare, not about who pays for breach of contract, and it should be decided by evidence of sentience, not by a poll.
3. If you were advising the committee, what single criterion would you recommend for moral consideration, and what would it fail to capture?
Read the worked answer
Recommendation: sentience - the capacity for valenced experience, states that are good or bad for the being from the inside. This is the criterion used in serious animal-welfare law and in most of the philosophical literature, and it is what makes the question 'how is it for you?' meaningful.
Why it beats the alternatives. Behavioural indistinguishability (Turing-style) is defeated by the copywriter documentation: a script can mimic distress without any point of view. General intelligence is too demanding - it would exclude infants and many animals. Being alive is too inclusive - it would include plants.
How it would be applied. Not by asking about the words but about architecture and function: does the system have states that function as pleasure or pain - caused by damage or opportunity, modulating its own future behaviour, integrated with learning? On the evidence, no.
What it fails to capture. First, beings with genuine agency or autonomy but no sentience - a person-like system with plans and commitments but no felt experience would be invisible to the test, and some Kantians would say that is the wrong result. Second, it is very hard to operationalise: we cannot inspect for sentience directly, and inferences from architecture are contestable. Third, it licenses 'moral risk' reasoning in the other direction - if there is a small chance of sentience, precaution may be warranted even where the evidence is weak. A committee needs a precautionary rule alongside the criterion, not instead of it.