Eligibility API for three channels with very different appetites
A national health insurer exposes a Member Eligibility capability to three consumers: a consumer mobile app with peaks of 400 requests per second at Monday 08:00, a provider portal expecting 99.95% monthly availability, and an overnight clearinghouse that submits 20,000 single-record queries between 01:00 and 03:00. Today a single Experience API calls a CICS mainframe region directly over HTTP Request. The CICS region saturates at about 300 transactions per second and takes 45 minutes to recover when it does. The mobile team owns its own client credentials and polls eligibility on every screen render. Protected health information flows through the API. API Manager currently has only two globally applied policies: Client ID Enforcement and one SLA tier of 10,000 calls per minute. There is no System API layer.
1. Redesign the solution into API-led layers. Justify what becomes a System API, a Process API and an Experience API, and state explicitly what must not live in each layer.
Read the worked answer
System API - MemberEligibility System API. One API in front of CICS, using the appropriate CICS connector or transaction gateway, owning connection pooling, mainframe error code translation into a canonical contract, and a hard concurrency ceiling. A System API's job is to make the system of record consumable and protectable; it must not contain channel formatting, pricing or eligibility business rules, and it must not be bypassed. The single most important outcome is that the mainframe only ever sees traffic from this one deployable, so the 300 tps ceiling is enforced in one place.
Process API - Eligibility Resolution Process API. Orchestrates member, coverage and accumulator lookups, applies the business rules (active coverage, effective dates, grace periods), defines the cache contract, and exposes a channel-neutral model. It must not know about mobile screen shapes, and it must not contain transport-level detail like CICS record layouts - that belongs in the System API.
Experience APIs - three of them. Mobile Eligibility, Provider Eligibility, and a Bulk Eligibility API for the clearinghouse. Each shapes the response to its consumer, masks PHI fields the channel does not need, and carries its own SLA tier. They must not call CICS directly, and they must not reimplement eligibility rules, because three Experience APIs with three copies of the rules will diverge within a quarter.
The clearinghouse is a special case: 20,000 individual synchronous calls in two hours is a design smell, not a scaling problem. Give it a bulk or chunked contract on the Process API, either paginated reads or an MQ-based asynchronous submission, so it stops competing with the mobile peak for CICS transactions.
2. Design the API Manager policy set. How do you stop the mobile polling problem without waiting for an app release, and why is the existing 10,000 per minute SLA tier inadequate?
Read the worked answer
Why the global tier fails: a single aggregate SLA tier of 10,000 calls per minute lets any one client consume the whole allowance and starve the other two, and it gives no per-consumer accountability. The provider portal's 99.95% commitment cannot be met when a mobile release storm can exhaust shared capacity. Rate limits must be per client application, and the tier values must be derived from each consumer's contract: mobile roughly 450 rps steady-state with burst capacity, provider 40 rps, clearinghouse a low per-second rate with a high daily quota because its work is concentrated in two hours.
Policy stack, outermost first: OAuth 2.0 Access Token Enforcement with scopes (read:eligibility, read:eligibility:bulk); per-client SLA tier rate limiting; a spike control policy to smooth bursts rather than reject them abruptly; JSON threat protection and schema validation at the Experience layer; and a custom policy or expression-based limit if you need per-end-user rather than per-client limiting, since the built-in rate limiting keys on client ID.
Stopping the polling without an app release, in order of effort:
- Request collapsing at the Process API - coalesce identical (member, date) queries arriving within a 3-5 second window so 400 rps of duplicate screen renders becomes a handful of CICS calls.
- Conditional responses - return ETag and Cache-Control on eligibility responses so well-behaved clients get 304s, and keep the cache TTL short (seconds to a minute) for status fields.
- Quotas with clear feedback - 429 with Retry-After, advertised quotas in the developer portal, and a direct conversation with the mobile team backed by per-client analytics from API Manager.
- Escalate only if needed: per-token or per-device limits, which require the mobile team's cooperation anyway.
Do not silently throttle a consumer-facing channel without telling the owning team; unexplained 429s become an incident, not a fix.
3. How do you protect the CICS region while still meeting the provider portal's 99.95% availability commitment? Include what you would refuse to cache.
Read the worked answer
Protection first. Enforce a concurrency ceiling in the System API connection pool set safely below the 300 tps saturation point (for example 200-250 concurrent), add a circuit breaker so that when CICS response times degrade the API fails fast with a 503 rather than queueing until the region collapses, and move the clearinghouse onto the asynchronous or bulk path so it never competes for synchronous transactions. Request collapsing from the previous answer does double duty here.
Availability arithmetic. 99.95% monthly gives roughly 21 minutes of allowed downtime. A CICS region that takes 45 minutes to recover from saturation makes that target unachievable on the happy path alone, so you need three things: fast failure instead of slow failure (the circuit breaker), a degraded mode that serves last-known-good eligibility with an explicit staleness indicator and a documented staleness ceiling, and a conversation with the business about what the SLA actually covers. If the portal contract must hold through a mainframe outage, the only honest options are mainframe-side redundancy or a formally renegotiated SLA with an agreed degraded-mode definition. Do not promise 99.95% on a single CICS region and hope.
Caching decisions - cache at the Process API, not the Experience layer, so all three channels share one consistent view.
- Cache: plan attributes, coverage product definitions, provider network status - slow-changing reference data with a 15-60 minute TTL and event-driven invalidation when the source publishes a change.
- Refuse to cache: accumulator and deductible balances (money and utilisation, changes with every claim), termination and reinstatement events where a stale yes is a compliance problem, and any authorisation or entitlement decision. Serving a stale 'active' to a provider at the point of care is a clinical and legal risk, not a performance trade-off.
Instrument cache hit rate, CICS p99 and 429 rate as first-class SLOs; without them you cannot tell whether the protection is working until it fails.