MGWC study materials
MGWC study resources

Eight teams, four environments, deployments from a laptop

MGWC · Put your knowledge to work in context.

All case studies

Eight teams, four environments, deployments from a laptop

Eight Mule teams at a logistics company deploy to Dev, QA, Stage and Prod. Three teams run on CloudHub 2.0 and five on Runtime Fabric on-premises for data-residency reasons. Deployments are executed from developer laptops using the Anypoint CLI, and API Manager policies are applied by hand after go-live. Last quarter a hand-applied rate-limit policy in production was set to 100 calls per minute instead of 10,000 and broke a partner integration for 40 minutes. Environment-specific properties sit in each application's src/main/resources, including the production database password. There is no automated regression suite; Stage is used mostly for demos. Release cadence is weekly for two teams and quarterly for the rest. The toolchain is GitHub Enterprise, Jenkins and Artifactory, and a new platform lead wants everything pipeline-driven within two quarters.

1. Design the pipeline stages and gates for both CloudHub 2.0 and Runtime Fabric. What is built once and promoted, and where do the environment differences come from?

Read the worked answer

Build once, promote the artifact. The pipeline compiles, runs MUnit, packages the deployable JAR with mule-maven-plugin, and publishes it to Artifactory under an immutable version derived from the commit. Every environment deploys that same artifact by version reference. Rebuilding per environment is the root cause of most 'it works in QA' failures and makes a rollback ambiguous. Environment differences come only from externalised property sources: CloudHub 2.0 secure properties and application properties set at deploy time, Runtime Fabric config maps and Kubernetes secrets, and a secret manager for credentials. Nothing environment-specific remains in src/main/resources except the reference to where properties come from.

Stages and gates:

  1. Validate - RAML/OAS lint, DataWeave and compile, property lint (no plaintext secrets) - blocking.
  2. Unit test - MUnit with mocked endpoints, coverage threshold - blocking.
  3. Package and publish - Artifactory, immutable version, plus Exchange publication of the API contract as a versioned asset - blocking.
  4. Static and dependency scan - secrets, CVEs, licence - blocking on high severity.
  5. Deploy Dev - automatic; integration and contract tests against a stubbed or ephemeral backend.
  6. Deploy QA - automatic on green build; functional regression suite - blocking.
  7. Deploy Stage - automatic; performance smoke against p95 thresholds, policy drift check, and a consumer contract verification - blocking.
  8. Deploy Prod - approval (linked change record) plus the same policy drift check and a post-deploy smoke with automatic rollback on SLO breach.

CloudHub 2.0 versus Runtime Fabric: identical stages, different deployment descriptors. Use Maven profiles or a per-target deployment configuration so the plugin targets cloudhub2 or the Runtime Fabric target, with replica counts, resource profiles and autoscaling kept as deployment configuration, not code. Runtime Fabric on-premises has no CloudHub-style elastic autoscaling, so sizing is replicas and resource requests, and the Stage performance gate matters more for those five teams. Keep one pipeline template parameterised by target rather than two divergent pipelines.

Cadence: the weekly teams run the full pipeline on merge to main. The quarterly teams run the same pipeline on their release branch - the pipeline is not the bottleneck, infrequent releases are.

2. How do you make API Manager policy application deterministic and reviewable across environments? Include API instance promotion, autodiscovery, and the limits of a policy-as-code approach.

Read the worked answer

Autodiscovery is the link between artifact and policy. Each application declares its api.id in the environment's property source, so a deployed app registers itself against the correct environment-specific API instance in API Manager. That means the api.id must be externalised and validated at startup, or an app will silently register against the wrong environment's API and inherit the wrong policies - a failure mode that is invisible until a rate limit misbehaves.

Policy as code. Keep the policy set for each environment in Git as a declarative file: which policy, which order, which configuration values, with per-environment templating for numbers like rate limits. The pipeline step, running under a dedicated service account with least privilege and its credentials in the secret manager (never a personal login), applies the policy set via the platform APIs after deployment, then reads back the applied policies and asserts they match the declared state. Fail the pipeline on drift. Producers of this pattern must accept that policy application is eventually consistent - poll until ready before asserting, or the check becomes flaky and gets disabled.

Promotion. The API definition is published once to Exchange with a version; API Manager instances in each environment reference that version. Promotion between environments means deploying the artifact and applying the environment's policy set - not copying an API instance. Keep the API instance identity stable per environment so client registrations and analytics history survive.

Limits to be honest about: the platform does not version policy configuration itself, so Git plus a scheduled drift detector is the real source of truth - a pipeline-only check misses out-of-band console changes. Custom policies need their own build, test and deploy pipeline, and they are harder to unit test. Plan-level or SLA-tier configuration may need separate handling from instance-level policies. And some changes are genuinely risky: require approval and a mandatory assertion that the configured limit is at or above the contractual minimum, which turns the 100-versus-10,000 incident into a failing test rather than a postmortem action item.

Drift detection should run daily against all four environments, alert on any policy or API instance difference, and open a ticket automatically - unauthorised console changes are the norm in the first year, not the exception.

3. What test strategy must exist before manual production approval can be removed, and how do you guarantee rollback for a production defect?

Read the worked answer

Test pyramid, sized to the risk rather than to coverage vanity:

  • MUnit unit and flow tests with mocked outbound endpoints, covering mappings, choice branches, error handlers and the retry classification logic. This is where most defects are caught and it runs in minutes.
  • Contract tests generated from the RAML/OAS in Exchange, run against a mock server and against the deployed API, so a breaking contract change fails the build rather than the consumer.
  • Integration tests against a real queue and stubbed backend in a disposable environment, covering the asynchronous paths, dead-lettering and idempotency behaviour that unit tests cannot reach.
  • Post-deploy smoke in every environment, including the health endpoint, one happy path and one error path.
  • Non-functional gates: a performance smoke on the critical path asserting p95 against a threshold, and the API Manager policy assertion described earlier.

Rollback: keep the previous artifact version available and deployable at all times. On CloudHub 2.0 redeploy the prior version; on Runtime Fabric, use rolling updates with readiness probes and keep the previous replica set until the new one is healthy, so rollback is a redeploy of a known artifact rather than a rebuild. Blue-green where the platform supports it. The hard constraint is state: any database, queue or schema change must be expand-contract and backwards compatible for at least one release, or rollback is impossible without data surgery. API contracts get the same treatment - version the API and keep the prior version serving consumers until they migrate, never break in place.

Removing manual approval is a track-record decision, not a tooling decision. Require a period of, say, 20 consecutive successful production deployments through the pipeline, at least one rehearsed rollback, and automated gates that demonstrably cover the failure modes you have actually seen, including the policy misconfiguration. Then replace the approver with a linked change record, automatic deployment, a named on-call owner and an automated rollback trigger on error-rate or latency SLO breach. The gate becomes the automated checks; the approval becomes accountability rather than inspection. Keep a documented exception path for genuinely high-risk changes - schema migrations, first production cutover of a new partner - because pretending every deployment is equal is how organisations lose the discipline they just built.