Forty brokers, one static secret, and a PCI assessor
A payments-adjacent insurer runs a claims-status API consumed by 40 external broker agencies and by an internal claims portal. Each broker holds a static client ID and secret, all created in the production business group. Database and SFTP credentials for the applications live in application.properties committed to a Git repository readable by 30 engineers. The API terminates TLS at a load balancer and re-encrypts to the Mule runtime. Mutual TLS was promised to brokers two years ago and never delivered. Claim payloads include policy numbers and bank account numbers for reimbursement, and the PCI assessor has placed the API in scope. Application logs currently record full request payloads at INFO level for troubleshooting and are retained for 400 days. A penetration test found that malformed input returns a full stack trace including SQL error text.
1. Prioritise the findings and produce a remediation sequence. Justify the order on risk and effort rather than following the assessor's checklist.
Read the worked answer
Rank by exploitability multiplied by data sensitivity, then by cost. The cheapest high-exposure fixes go first because they buy risk reduction this week.
Days 1-10 - stop the bleeding, all reversible:
- Remove payload logging. Log correlation IDs, status and timings; mask claim and bank fields with DataWeave before any log call. Reduce retention from 400 days to the policy minimum (typically 30-90). This is the single largest exposure: full claim payloads with bank accounts sitting in a log platform for over a year, readable by everyone with log access.
- Replace stack traces and SQL text with a generic error contract plus a correlation ID; log the detail server-side only. This closes an information-disclosure and injection-signalling finding at near-zero cost.
- Enable secrets scanning in CI and pre-commit hooks so no new credential enters Git.
Weeks 2-6 - credentials and transport:
- Treat every credential in Git history as compromised. Rotate the database and SFTP credentials, move them to secure property placeholder backed by a secret manager, restrict the decryption key to the runtime, and purge history. Rotation first, purge second - purging without rotating is theatre, since 30 engineers have clones.
- Enforce TLS 1.2+ end to end and validate certificates on the re-encrypt leg; the load balancer terminating TLS is fine, but an unvalidated backend leg makes the re-encryption decorative.
Quarter - identity and least privilege:
- Migrate the 40 brokers from shared static secrets to OAuth 2.0 client credentials, one client app per broker, and replace the shared database account with per-application least-privilege accounts.
Also worth sequencing deliberately: tokenising bank account numbers so they never enter the runtime, which can remove that data from PCI scope and is the highest-value medium-term move. Put the assessment narrative in terms of scope reduction, not just control gaps.
2. Design authentication and authorisation for the 40 external brokers versus the internal claims portal. Which grants, scopes, token lifetimes and transport controls, and what do you do about a broker that cannot implement OAuth?
Read the worked answer
Brokers (server-to-server, 40 principals). OAuth 2.0 client credentials, one client application per broker in API Manager - never a shared client ID, because per-client identity is what makes rate limiting, audit and revocation possible. Scope per capability (claims:status:read, claims:documents:read) and keep the scope set minimal per broker based on the contract. Access token lifetime 15-30 minutes; there is no refresh token and no user context, and validation is done by the OAuth 2.0 Access Token Enforcement policy against the IdP's introspection or JWKS, checking audience and scope, not just signature. Add mTLS as a second factor wherever the broker's stack supports it - credential compromise is the realistic threat and two factors one-remove that risk. Do not use the password grant, and do not let brokers hold tokens longer than necessary 'for convenience'.
Internal portal. OIDC authorisation code with PKCE against the corporate IdP, producing a user-context token. Authorise on user attributes and roles, not on a shared service account, and propagate identity downstream via token exchange rather than impersonation so the audit trail stays intact. If a backend System API needs its own credential, use a separate client with its own scopes - do not reuse the portal's token.
Enforcement point. API Manager policies on each API instance, with the IdP as the single source of truth for identity; the Mule layer holds no user store.
The broker that cannot do OAuth. Segment it: a dedicated API instance for legacy brokers, protected by IP allowlisting plus rotated static credentials, with strict rate limits, no access to sensitive scopes, and a contractual migration date. Isolation limits blast radius and gives the business a path to yes without compromising the target architecture. What you must not do is keep a shared production-wide secret, or put the exception on the main instance where it weakens everyone's controls.
3. Design the secrets management and data protection changes: configuration, logging, transport and data at rest. What governance keeps it from regressing?
Read the worked answer
Configuration and secrets. No credential in source control, ever. Use secure property placeholder backed by a secret manager (Vault or the platform equivalent), with per-environment paths and short-lived credentials where the backend supports it. The decryption key or Vault token is injected by the runtime at deploy time through the platform's own secret mechanism, never in the artifact. Rotate on a schedule and on any suspicion; log rotation events. Because the repository had 30 readers, assume historical exposure and rotate rather than trusting a purge.
Logging. Structured JSON with correlation ID, client ID and outcome. Mask policy, claim and bank fields at the logging boundary - a shared DataWeave logging library so masking is the default, not a habit. Separate the audit log (who accessed what, retained longer, immutable) from the diagnostic log (short retention). No tokens, no authorisation headers, no payloads. Alert on any log line matching a bank-account or policy-number pattern; that is how you find the flow someone forgot.
Transport. TLS 1.2+ to external clients, mTLS for internal hops where feasible, certificate validation and hostname verification enabled on every outbound HTTP Request, no trust-all truststores. Keep the load balancer termination if operations need it, but the backend leg must be validated and encrypted.
Data protection. Field-level protection for bank account details - either tokenise via a vault service so the runtime never stores the value, or encrypt with JWE using keys the runtime cannot read in bulk. This is what can move the API out of PCI scope, which is worth more than any individual control. Encrypt at rest for Object Store and any queue or file landing zone that transiently holds claim data.
Governance to stop regression: policies and secure-property patterns enforced through the pipeline rather than review; secrets scanning and dependency scanning as blocking checks; a mandatory security review for any new external consumer; quarterly access review of the Git repository and the secret manager; and a defined exception process with expiry dates, because an exception without an expiry date is a permanent control failure.