AI-Generated Test Conditions for Partial Refunds
NorthCart, an online retailer, is rebuilding its returns portal. The business requirement says customers may return items within 30 days for a full or partial refund; tax must be recalculated; original shipping fees are non-refundable; gift cards are refunded as store credit. A business analyst asks an LLM to generate test conditions from this paragraph and 11 user stories. The LLM returns 60 conditions in 4 minutes. 38 are variations of “valid return” with different product names; only 3 mention tax; none cover day 30 versus day 31, partial refunds when one item in a multi-item order is returned, damaged goods, or payment-method differences. The test lead is under pressure to show 80% requirement coverage and considers accepting the set because of its volume. Two experienced testers are available for review, and the team has a decision-table template and a tax calculation service that can serve as an oracle.
1. Evaluate whether the 60 generated conditions meet the 80% coverage target. What analysis techniques should the team apply?
Read the worked answer
Evaluation. The set does not meet a meaningful coverage target. Counting 60 conditions rewards volume, not coverage. Thirty-eight near-duplicates of “valid return” add little decision coverage, while tax, boundary timing, partial refunds, damaged goods, and payment methods are missing. The 80% figure is undefined unless it is measured against a risk-based set of test conditions traced to requirements.
Techniques. Use equivalence partitioning for refund type (full, partial, none) and payment method; boundary value analysis for return timing at day 29, 30, and 31; a decision table for combinations of return reason, item condition, order composition, tax jurisdiction, and payment method; state transition testing for requested, approved, received, refunded, and rejected returns; and pairwise combination for lower-risk dimensions. Trace each retained condition to a requirement or risk. Deduplicate by semantic intent, not wording, and have the two testers review the resulting model before judging coverage.
2. Identify the highest-risk missing test conditions and specify expected results or oracle sources for them.
Read the worked answer
Missing high-risk conditions include: return on day 30 accepted and day 31 rejected; one item returned from a multi-item order where tax and shipping must be allocated correctly; full refund when all items are returned; gift-card purchase refunded as store credit rather than cash; damaged goods requiring evidence and possible shipping exception; original shipping non-refundable in partial and full returns; tax recalculation after discount and jurisdiction rules; and repeated refund attempts for the same item.
Expected results must come from business rules and the tax calculation service, not from the LLM. For example, the oracle for tax should be the tax service’s calculated amount for the refunded line items, and the oracle for shipping should be the original shipping charge remaining non-refunded. For store credit, assert the credit balance and that no cash refund is issued. Each condition should state preconditions, input data, expected monetary amounts, and post-state so that failures are diagnosable.
3. How should the team use the LLM to improve coverage without accepting its output as the oracle?
Read the worked answer
Use the LLM as a structured idea generator. Prompt it with explicit dimensions: return timing boundaries, order composition, payment method, tax jurisdiction, item condition, refund type, and negative cases. Ask for a traceability column, a rationale for each condition, and a list of assumptions or missing information. Then have the testers map the output to a decision table, remove duplicates, add combinations the model missed, and reject conditions that cannot be traced to a rule.
Do not ask the model to calculate expected refunds or tax. The tax service and business rules are the oracle. Keep a record of rejected and accepted conditions so the team can refine prompts and demonstrate coverage. This preserves the speed benefit while preventing volume from being mistaken for adequacy.