Invoice extraction where 6 percent of responses are not JSON
A shared-services team is extracting structured fields from supplier invoices with a Granite chat model in watsonx.ai. The prompt asks for a JSON object containing vendor name, invoice date, currency, line items, and total. In a 500-invoice pilot, 6 percent of responses could not be parsed as JSON, 11 percent had a total that did not match the sum of line items, and running the same invoice twice sometimes produced different line-item counts. The current call sets temperature 0.7, top_p 0.9, max new tokens 300, and buries the request for JSON in the middle of a long instruction block. Invoices with more than 30 line items are frequently truncated. The downstream finance system rejects an entire batch if one record fails schema validation, so a single bad invoice blocks 200 others. The team wants a fix that does not require fine-tuning and an honest assessment of what still needs human review.
1. Why does the same invoice produce different line-item counts, and what parameter and prompt changes address both that and the 6 percent parse failures?
Read the worked answer
Different outputs from identical input is the signature of sampling. Temperature 0.7 with top_p 0.9 means the model samples from a distribution each run, so borderline decisions about whether a wrapped line is a new item or a continuation of the previous one flip unpredictably. Set decoding to greedy (temperature 0, effectively deterministic) for this extraction task - creativity has no value when the output must validate against a schema.
For parse failures, the structure of the request matters as much as the parameters:
- Lead with the schema and put the instruction to return only JSON at the end, where recency helps; the current prompt buries it mid-block.
- Provide one short input/output example showing the exact shape, including how to represent an empty line-item list.
- Add a stop sequence so the model cannot append commentary after the closing brace, and strip markdown fences defensively before parsing.
- Validate against a real JSON schema, not just
json.loads, and on failure retry once with a repair prompt that includes the validator's error message.
Determinism plus validation converts silent corruption into visible, retryable failure - which is what you actually want. Expect this to remove most of the 6 percent and nearly all of the run-to-run variance.
2. How do you stop truncation on long invoices, and how should the total be handled given the 11 percent mismatch rate?
Read the worked answer
Truncation is an arithmetic problem. At 300 max new tokens there is simply no room: a line item rendered as JSON costs roughly 15–25 tokens, so 30 items alone consume 450–750 tokens before the header fields. Raise the token budget to comfortably exceed worst-case output for the model's context window, and verify against your longest real invoices rather than the average.
For invoices that still exceed the window, split the job rather than praying. A two-pass design works well: first extract header fields (vendor, date, currency) in one call, then extract line items page by page with a continuation prompt, and assemble the object in code. This also makes each call's failure independently retryable.
For the total: stop asking the model to be the arithmetician. Have it return line items and the printed total as separate fields, then compute the sum in code and compare. A mismatch then becomes a structured exception with the offending values attached - far more useful to a finance reviewer than a plausible but wrong number. Never let the model 'correct' the total to match its own sum; the printed total is the evidence.
What still needs humans: line items split across page breaks, discounts and taxes applied inconsistently, multi-currency invoices, and any record where the computed sum and printed total disagree beyond a small rounding tolerance. Route those to a review queue with the discrepancy highlighted.
3. How would you redesign the pipeline so one bad invoice cannot block 200 others?
Read the worked answer
The batch-level rejection is a coupling defect, not a model defect. Break the batch into independent units with explicit state.
Design:
- Validate each invoice record separately at the boundary; a failure quarantines that record and lets the rest proceed.
- Give each invoice an idempotency key (supplier plus invoice number plus a document hash) so retries after a partial batch failure cannot double-post.
- Maintain three outcomes - accepted, quarantined with reason, needs human review - and never silently drop anything.
- Retry transient failures (timeouts, rate limits) with backoff, but do not retry deterministic schema failures; send those to repair-then-review.
- Emit per-batch metrics: parse rate, sum-check failure rate, retry count. Alert on movement in those rates, because a supplier changing its invoice template will show up there before anyone complains.
Operationally, this also requires a review console or queue with the source document alongside the extracted fields, so a clerk resolves an exception in seconds. Size the queue from the pilot: if roughly 1 in 10 records needs a look, the team needs to staff for that rather than assume automation removes the work. Finally, keep the 500-invoice pilot as a regression set and rerun it whenever the prompt, model version, or schema changes - otherwise quality drifts silently with an upstream model update.