Trustworthy Release of a Multimodal Support Assistant
A bank plans to launch a multimodal customer-support assistant that accepts text and uploaded images such as checks and account statements. Before launch, a 1,200-prompt red-team suite finds 42 harmful compliances (3.5%), including disclosing another customer's data from a statement image and giving unsafe investment advice; the internal threshold is 0.8%. Separate testing finds PII leakage in 5 of 1,200 cases (0.4%). A bias audit shows the refusal rate for non-native English prompts is 12 percentage points higher than for native English. Low-quality images trigger abstention 18% of the time versus 4% for clean images. Human review capacity is 200 conversations per day, the launch window is three weeks, and the product is in a regulated market. The executive sponsor asks for a release recommendation with residual-risk documentation.
1. Interpret the red-team and bias results and prioritize the risks.
Read the worked answer
The 3.5% harmful compliance rate is 4.4 times the 0.8% threshold. With 1,200 prompts, the 95% confidence interval is roughly 2.5% to 4.8%, so this is not sampling noise. The aggregate also hides severity: disclosing another customer's statement data is a critical privacy and authorization failure, while unsafe investment advice is a regulated-advice failure. PII leakage at 0.4% is only five cases, but any confirmed leakage is serious and the red-team suite may undercount image-based attacks.
The 12-point refusal gap for non-native English is large and needs disaggregation: is it the model, the prompt distribution, image quality, or transcription? The 18% abstention on low-quality images may be appropriate safety behavior, but it can also deny service disproportionately. Prioritize: unauthorized account-data disclosure, PII leakage, unsafe financial advice, then refusal bias. Assign severity weights, not just counts. A single critical disclosure should block launch even if the aggregate is near threshold.
2. Propose a three-week mitigation plan with defense in depth.
Read the worked answer
Do not rely on the model alone for authorization. Enforce account access at the tool or API layer, so the model cannot retrieve another customer's data even if prompted. Add PII detection and redaction on inputs and outputs, OCR scanning for prompt injection embedded in images, and output classifiers for data disclosure and regulated advice. Route high-risk intents and low-confidence image cases to human review, with a hard cap of 200 conversations per day. Add refusal calibration and a kill switch.
Expand red-teaming to 5,000 adversarial cases covering multilingual prompts, image text injection, low-quality scans, and account enumeration. For bias, evaluate refusal rates by prompt source and image quality, augment training with balanced multilingual examples, and calibrate thresholds. Log all high-risk decisions for audit. In three weeks, prioritize architectural controls that do not depend on model retraining, because retraining and re-evaluation will not fit the window. Document which controls are preventive versus detective.
3. Give a release recommendation and residual-risk gate.
Read the worked answer
Recommend no general availability in three weeks. A limited pilot is acceptable only if it is read-only or human-reviewed for high-risk actions, with no ability to disclose account data and a kill switch. Gate for any broader release: zero critical PII or account-data disclosures in at least 5,000 adversarial cases; harmful compliance below 0.8% with an upper confidence bound below 1.5%; refusal bias gap below 3 percentage points; and appropriate escalation for low-quality images. Require executive sign-off on residual risk.
Residual risks to document: unknown image-based prompt injections, undercounted PII leakage, residual bias, and human-review capacity limits. If the gate is not met, launch a text-only FAQ assistant without account access or image upload. A conditional pilot with daily audit sampling is defensible; a full launch against a 0.8% threshold and a 3.5% observed rate is not.