The Cost Cut That Broke the Workflow
A document-processing pipeline runs 2 million pages/month. Each page: OCR text (~1,500 tokens) → Claude extracts 8 structured fields → downstream system ingests. Current cost is ~$40k/month, dominated by input tokens. Finance wants a 50% cut. The team proposes: (a) route all pages to the smallest/cheapest model, (b) truncate input to the first 800 tokens, (c) drop the few-shot examples from the prompt, (d) batch requests. You have a labeled set of 2,000 pages with field-level ground truth. Current field-level accuracy is 97.2%; the downstream system tolerates 95% but a missing required field causes a hard downstream failure, and a wrong field is caught 60% of the time by a downstream validator.
1. Rank the four proposals by risk and predict the likely outcome of each against the accuracy floor.
Read the worked answer
Rank by risk (highest first):
1. (b) Truncate to first 800 tokens - highest risk, likely breaks the floor. At 1,500 tokens/page, you're discarding ~47% of the text. Structured fields (dates, amounts, IDs, signatures) are frequently not in the first 800 tokens - headers and boilerplate are. This will cause missing required fields, which is the hard-failure mode, not the soft one. Predict a sharp rise in nulls on exactly the fields that matter. Reject unless you can prove the dropped region never contains required fields - and you can't, generically.
2. (c) Drop few-shots - high risk on structured extraction. Few-shot examples are doing format and edge-case work: field boundaries, what counts as a value vs. boilerplate. Removing them tends to shift errors from "wrong value" (caught 60%) to "missing/malformed field" (hard failure). Predict accuracy drop concentrated in low-frequency field formats.
3. (a) Route all to smallest model - medium-high risk, and the wrong shape. Uniform routing throws away the easy/hard distinction. Many pages are easy and a small model handles them; a minority are hard and need the larger model. Uniform downgrade likely drops below 95% on the hard tail while over-spending capability on the easy head. The right move is tiered routing: cheap model first, escalate on low confidence or validation failure.
4. (d) Batch - low risk, real savings. Batching (async/batch API) typically cuts cost materially with no accuracy change. This is the safe win.
Do (d) first. It may get you a meaningful fraction of the 50% with zero accuracy risk, changing how much the risky levers need to contribute.
2. Design the routing and validation strategy that could hit 50% cost reduction while holding the 95% floor, and state how you'd measure it.
Read the worked answer
Tiered routing with validation-gated escalation:
- Cheap model on all pages, with a confidence/validation gate. Keep the few-shots (they're cheap relative to a hard failure) and keep full text - the savings come from model tier and batching, not from blinding the model.
- Escalate to the larger model when: any required field is null/malformed, the downstream validator rejects, or the model's per-field confidence is low. Only the escalated fraction pays the higher price.
- Batch the cheap-tier calls (async) to capture the batch discount; escalated calls can be synchronous since volume is small.
- Never truncate; if input cost is the concern, cache the stable prompt prefix (system + few-shots) so it isn't re-billed per page - that attacks input cost without dropping content.
Cost model: if the cheap tier is ~1/5 the price and 80% of pages stay on it, and 20% escalate, blended cost ≈ 0.8·(1/5) + 0.2·1 = 0.36 of baseline - a ~64% cut before batching, comfortably past 50% with margin for the escalated calls. Tune the escalation threshold to trade cost against the floor.
Measurement on the 2,000 labeled pages:
- Report field-level accuracy per field and required-field null rate separately - the null rate is the hard-failure metric and must be ~0, not just "within 95%."
- Report the escalation rate and accuracy of the cheap tier alone vs. the blended pipeline; the blended number can hide a weak cheap tier.
- Compute expected downstream failure cost: wrong fields escape the validator 40% of the time; weight that against the savings. Ship only if blended accuracy ≥95%, required-field null rate is at/near zero, and the cost model holds on the labeled set before a canary on live traffic.