A lookalike package reaches laptops and CI through the internal mirror
Ravi, a backend engineer at a payments processor, adds netfetchh 3.2.1 to the checkout service. During review a security champion compares it with the well-known netfetch. netfetchh was first published nine months ago, has 41,000 downloads, and its sole maintainer account was created ten months ago. Its sdist ships a setup.py that, at install time, fetches a prebuilt binary from a personal domain over unverified HTTPS and executes it. The package depends on real netfetch and re-exports its API, so application code works after the swap. The requirements.txt has no hashes, the internal PyPI mirror proxies upstream and caches packages for a year, and 40 developer laptops plus two CI runners have already installed it while testing a feature branch. Ravi argues: 'Our code only imports the library; the install script is a build detail, and anyway pip runs it in an isolated build environment.'
1. Assess the evidence and set out the immediate response.
Read the worked answer
Treat this as a probable supply-chain compromise until disproven. The signals stack: a name one letter from a popular package, a ten-month-old account, a package younger than the account, download counts consistent with automated installs, and, decisively, an install-time fetch and execution of an unverified binary from a personal domain. That is remote code execution on every machine that installs it.
Immediate actions, in hours:
- Block netfetchh in the registry proxy and internal mirror, and purge cached artifacts; the mirror may have served it to others.
- Inventory exposure: 40 laptops, 2 CI runners, and any container images built from those branches. Rebuild or rebuild-and-replace the runners rather than cleaning them.
- Rotate credentials reachable from those machines: registry tokens, CI secrets, cloud keys, SSH keys, and checkout-service deploy credentials.
- Preserve evidence: pip logs, the binary's hash, the sdist, and network logs.
- Check persistence: cron entries, systemd units, shell profiles, and outbound connections to the personal domain.
- Notify security, follow the incident process if production secrets were exposed, and report the package to the registry.
Do not simply uninstall and move on; the payload already ran with developer or runner privileges.
2. Evaluate Ravi's argument that import-time behavior is what matters and build isolation makes the install step safe.
Read the worked answer
The argument fails on three points.
- pip executes the build backend. For an sdist, pip runs setup.py or the declared backend, which is arbitrary Python at build time. PEP 517 build isolation isolates dependencies, not trust: the build still runs on the machine with the user's permissions and network access. The fetch happens before any import.
- Installation is the execution point. The application never needs to import the module for the payload to run. 'We only import it' describes runtime behavior and misses the install-time vector entirely.
- The mirror multiplies exposure. A caching proxy with one-year retention can keep serving a malicious sdist to teams that never intended to use it. Caching without scanning is an amplifier.
Preventive controls: hash-pinned requirements with --require-hashes; an allowlist of approved packages and maintainers; a mirror that quarantines first-seen packages; and builds from wheels produced in a controlled pipeline rather than executing upstream sdists on laptops.
Process gap: the package entered through a feature branch with no review gate. Add a dependency-review step for name similarity, maintainer age, install scripts, repository link, and provenance before anything enters requirements. For this incident, reject the package, use netfetch, and rebuild from a clean environment.
3. The platform team says the internal mirror cannot be removed because air-gapped builds depend on it. What controls would make the mirror defensible?
Read the worked answer
Keep the mirror, change what it is allowed to serve.
- Allowlist-first: only packages explicitly approved by the dependency-review process are available; everything else is quarantined for review rather than proxied automatically.
- No execution of upstream build scripts in protected environments: pre-build wheels in an isolated, network-restricted builder, publish those wheels to the internal index, and have CI install from the internal index with hashes only.
- First-seen quarantine: hold new packages and new versions for a review window; alert on name similarity to popular packages and on maintainer account age.
- Retention and provenance: shorten the one-year cache, store the source URL and hashes with each artifact, and keep an audit log of who requested what.
- Egress control: builders and developer machines should not reach arbitrary personal domains; allow only the registry endpoints.
- Scanning: malware and secret scanning on ingest, with a removal runbook and a way to invalidate cached copies everywhere.
- Exception path: an air-gapped offline bundle is generated from the approved set, with hashes, so the air-gapped side inherits the same controls.
Mirrors concentrate trust. Once it is an allowlist with hashes and a controlled builder, it becomes an asset rather than an amplifier.