An AI Agent That Wins Opens and Loses Revenue

Open rate is the wrong metric and it is moving the wrong way. Revenue per recipient fell 7% (£0.44 to £0.41) while opens rose 5.2pp. The subject lines are winning opens the content cannot repay - a curiosity gap. Unsubscribes rising from 0.11% to 0.19% is a 73% increase, which at 240,000 weekly contacts is roughly 190 extra unsubscribes per week.

Verdict: keep the agent, change its objective and constrain it. The underlying capability - matching product to contact - is real, as the 2.9x CTR shows. What is broken is the target it optimises.

Guardrails:

1. Move the objective from click to margin-weighted revenue. The agent was trained on click data, so it learned what gets clicked, which is best-sellers. Feed it gross margin per SKU and revenue per recipient, then retrain or re-weight. Without this, business rules will fight the incentive forever.
2. Constrain the assortment. Cap each best-seller's share of sends and force accessory inclusion in a set proportion, then check whether accessory attach rate rises without hurting revenue per recipient. If it does not, margin-weighted training is the real fix.
3. Keep a permanent 10% human-written holdout, every week, so you can detect drift and justify the agent's existence each quarter.
4. Human review for new subject-line patterns plus a blocklist of phrasings implying surveillance.

Do not judge on eight weeks alone. That is one seasonal cycle, and the training data contained no revenue signal at all.

Why it fails. Sent to someone whose last order was 11 months ago, the line implies the brand is tracking current attention, and it is factually flimsy: an 11-month gap is not 'still thinking'. Legal is right. The risk is not only regulatory but reputational, and recipients who feel watched unsubscribe - which is plausibly part of the 0.19% you are seeing.

Fix the pattern, not the copy. Banning this one line leaves the generator free to produce variants.

• Restrict the personalization tokens the agent may use in subject lines to: first name, category affinity, loyalty tier, and recency band expressed positively ('a year with us', 'back for more'). Remove specific past-purchase references and any construction implying ongoing observation.
• Add a time threshold: purchase-referencing language is allowed only within 90 days, and only as a factual statement such as 'Your October order'.
• Add an automated pre-send check against a blocklist of verbs and constructions ('still', 'watching', 'noticed you', 'we saw'). Anything that trips it goes to a human for the first four weeks.
• Keep the review log. If a regulator or journalist asks, you can show the control existed.

Then measure the cost. Compare revenue per recipient for constrained against unconstrained subject lines on the 10% holdout. If constrained lines give up 1–2pp of open rate but recover revenue per recipient, the constraint is free. Expect it to be, because the open-rate gain was not converting anyway.

Do not rely on a prompt instruction alone. Model behaviour drifts; token allowlists and pre-send checks do not.

The current cells are too small to read. Under 300 contacts per segment per block means a 2.9x CTR difference could easily be noise at segment level, and any per-segment winner you pick will be partly luck. Multiplicity makes it worse: testing 20 segments at 95% confidence yields roughly one false winner by chance.

Three changes.

1. Test where you have power. Aggregate to category (hair, skincare, accessories, fragrance) rather than segment by block. Four categories by two arms at 240,000 weekly sends gives tens of thousands per cell.
2. Fix the arms and the horizon. One control (human-selected) and one treatment (AI with the margin-weighted objective), each with a 10% holdout, evaluated over a minimum of eight weeks and preferably twelve. Declare the primary metric in advance as revenue per recipient. Pre-registering it stops you choosing whichever metric looks best afterwards.
3. Use sequential testing or a fixed sample, not weekly peeking. Checking significance each week and stopping when it crosses a threshold means you will stop on noise. Decide the sample size up front or use a method built for continuous monitoring.

What to do with the eight weeks you already have. Treat it as directional. Revenue per recipient of £0.41 against £0.44 across 240,000 sends may well be large enough to be meaningful in aggregate - check that first - but discard the segment-level claims entirely.

Report the uncertainty. Give the board a range and the test horizon rather than a single uplift number. An agent that is honestly 3% better is more useful than one claimed to be 40% better on a 280-contact cell.