Model Extraction Through a Forecasting API
Helio Predict sells API access to a proprietary demand-forecasting model used by 90 retailers. Over 11 days, one 'analyst' account made 2.4 million queries with systematically varied price, promotion, weather, and holiday inputs on a dense grid; 92% came from 14 IP addresses in two countries and volume was 40× baseline. The tier has a 60 requests/minute limit but no aggregate cap. A junior data scientist noticed the pattern; the SOC has no ML-specific detection and the account passed KYC as a small consultancy. Helio's CISO wants to terminate and publish a breach notice. Legal asks whether model weights are a trade secret and whether the queries are 'unauthorised access'. The model is trained on retailer transaction data under contracts that prohibit reverse engineering.
1. Assess the likely attack and explain why standard API security did not detect it.
Read the worked answer
The pattern is consistent with model extraction or model stealing: an adversary queries the API over a dense input grid to train a surrogate model that approximates Helio's predictions. The adversary gains a functional clone, which can be used to undercut Helio, avoid fees, or probe the model for weaknesses. Standard API security did not detect it because it watches authentication failures, rate-limit violations, IP reputation, and payload size, not semantic query behaviour. The account stayed within 60 requests per minute and passed KYC; the volume was high only in aggregate. The 14 IP addresses and 40× baseline are visible in logs but not in the rules. Detecting extraction requires modelling query diversity and coverage: repeated queries across a structured grid, boundary probing, low entropy in requested outputs, and an account whose usage does not match its stated business. The SOC also lacked an ML-specific detection use case, so no one owned the signal until a data scientist noticed it manually.
2. Recommend immediate containment and longer-term controls, balancing false positives for legitimate high-volume users.
Read the worked answer
Immediate containment: preserve logs and put the account on legal hold before terminating; throttle or suspend the account through a controlled process; rotate any keys or tokens it holds; check for other accounts with similar query signatures; and avoid tipping off the adversary until evidence is secured. Do not publish a breach notice before confirming what data, if any, was exposed. Longer-term controls: add aggregate quotas and query-diversity limits per account, not just per-minute rate limits; require business justification and use-case registration for high-volume tiers; deploy extraction detection that scores coverage of the input space, repeated boundary queries, and account-to-use mismatch; consider output perturbation or rounding where it does not harm utility; embed canary inputs or watermarks in the model; and use contractual terms that prohibit reverse engineering and scraping. To manage false positives, create a review path for legitimate benchmarking or integration testing, allow pre-approved high-volume jobs, and use stepped authentication rather than automatic bans. Detection thresholds should be tuned against known good users and reviewed by both security and product.
3. Advise on legal and regulatory notification, evidence preservation, and the CISO's proposed press release.
Read the worked answer
First, preserve evidence: logs, query payloads, account records, KYC files, and network data, with chain of custody. The model weights and training pipeline can be trade secrets under the EU Trade Secrets Directive if they are secret, have commercial value, and are subject to reasonable protection; the contractual prohibition on reverse engineering supports that position but does not replace technical protection. Whether the queries are 'unauthorised access' is uncertain. The account had valid credentials, so criminal unauthorized-access theories are weaker than in a break-in; the stronger civil theory is breach of contract or trade-secret misappropriation. Regulatory notification depends on facts not yet established. If the queries contained personal data, GDPR breach assessment may be triggered, but price and weather inputs may not. NIS2 could apply if Helio is an essential or important entity, but that is a separate analysis. A press release now could destroy negotiation leverage, alert the adversary, and create defamation or disclosure risk. I would recommend a legal hold, a factual incident report to the board, customer communication only if required, and law-enforcement contact after counsel assesses the evidence. The CISO's instinct to act is right; the sequence is wrong.