Fifty-Five Minute Pipeline with Flaky Parallel Shards
Your product has one declarative Jenkinsfile in a shared library. The pipeline takes 55 minutes: checkout, build (8 min), a parallel test stage with eight shards (30 min), package, and deploy. Each shard runs on a label matching four agents with two executors. Integration shards are flaky at roughly 12%, and when one shard fails the whole test stage is marked failed. Engineers currently press Build again and wait for all eight shards, even when only one failed. The pipeline uses stash/unstash to pass the build output to shards. A recent shared-library change broke 60 pipelines at once, and there is no version pinning. You need faster feedback and a safe way to resume after a single flaky shard.
1. Why does 'Restart from Stage' fail or not help in this pipeline, and how would you configure it correctly?
Read the worked answer
Restart from Stage needs options { preserveStashes() } (or preserveStashes(buildCount: 5)) at the pipeline level. Without it, the stash from the build stage is not retained after the run, so the restarted test stage cannot unstash the build output and either fails or rebuilds. Restart also re-executes the selected stage and every downstream stage, and it cannot resume one branch of a parallel stage. In this pipeline, restarting the test stage would rerun all eight shards, which is exactly the 30-minute cost you are trying to avoid. Enable preserveStashes, make each stage idempotent, and do not depend on workspace state that is not stashed or checked out. Test restart on a deliberately failed clone before relying on it.
2. Redesign the test stage to isolate flakiness and reduce the cost of rerunning one failed shard. What declarative constructs do you use?
Read the worked answer
Make each shard a separate parallel stage with its own options { timeout(time: 20, unit: 'MINUTES'); retry(2) }, and set failFast false so one shard does not cancel the others. Publish JUnit results in each shard's post { always { junit ... } }. For flaky integration tests, add test-level retries where the framework supports them (pytest-rerunfailures, Jest retries) so a transient failure does not fail the stage. To rerun only a failed shard, preserve the stash and add a parameterized recovery job that selects one shard and runs the same commands; do not make engineers rerun the whole matrix. A declarative matrix is tidy for sharding but restart still reruns all cells, so it does not solve single-shard recovery. Use lock or lockable-resources for the shared database and keep shards independent.
3. How do you stop a shared-library change from breaking 60 pipelines at once, while still allowing controlled rollout and rollback?
Read the worked answer
Pin the library at the call site: @Library('[email protected]') _ or a commit SHA, not @Library('platform-lib'). Set the global shared library's default version to a tag and allow per-job override, so a library merge does not instantly change 60 pipelines. Use semantic tags and a canary pipeline that runs build, test, and a deploy dry-run against the new tag. Roll out by changing the default only after the canary passes, and keep the previous tag for one-click rollback. Document that master or main is never a valid production pin. For folder-scoped libraries, configure the version in the folder and pin it the same way. This turns a global breakage into a controlled rollout.