Turning On Authentication and TLS Without Taking the Platform Down

The plan fails on two counts. First, internal authentication: the moment one member starts with x.509 or a keyfile and its peers do not, or the certificates do not chain correctly, the members cannot authenticate to each other and the set loses its majority. A simultaneous restart of everything is the one approach guaranteed to produce a total outage with no way back in. Second, authorization with no users defined means nobody, including the DBA, can connect - the localhost exception only lets you create the very first user when the deployment has none, and it disappears once that user exists.

Order of operations:

1. While authorization is still disabled, create the administrative user (userAdminAnyDatabase plus clusterAdmin as needed) on the primary, and create cluster-wide administrative users on the config/operational set. Verify the credentials work.
2. Distribute member certificates signed by the internal CA to all six hosts, with the correct subject and organization fields for cluster membership, and set clusterAuthMode and the TLS options.
3. Rolling restart: secondaries first, confirm each rejoins and rs.status() shows it healthy and authenticating, then rs.stepDown() the primary and restart it. Repeat per replica set.
4. Once internal auth is clean, enable authorization - db.adminCommand({setParameter:1, authorization:"enabled"}) avoids another restart cycle where your version supports it, otherwise roll again.
5. Only then create per-service users and start migrating connection strings.

Do step 1 before anything else; it is the only step that is impossible to recover from afterwards.

One user per service, scoped with a custom role rather than a built-in readAnyDatabase, plus a network control.

Role definition on the two permitted databases:

db.createRole({ role: "reportingRead", privileges: [ {resource:{db:"sales", collection:""}, actions:["find"]}, {resource:{db:"inventory", collection:""}, actions:["find"]} ], roles: [] })

Note collection:"" means all collections in that database - intentional here; do not use db:"" which would mean all databases.

User creation with authentication restrictions so the credential is useless off-subnet: db.createUser({user:"svc-reporting", pwd: passwordPrompt(), roles:[{role:"reportingRead", db:"admin"}], authenticationRestrictions:[{clientSource:["10.20.0.0/24"]}]})

Design notes worth stating:

• Do not grant listDatabases or any cluster-wide read; the service should not even know the other twelve databases exist.
• If the service must not see sensitive columns, expose a view that projects them away and grant find on the view. Verify on your server version how privileges are evaluated for views before relying on it as a security boundary; the role above remains the primary control.
• Rotate the password on a schedule and store it in the secrets manager the service already uses, not in a config file in git.
• Audit with $external or SCRAM plus the audit log if available, and alert on authentication failures from outside 10.20.0.0/24.

Finally, confirm the user cannot escalate: attempt a find on a non-granted database and a listDatabases as the service account as part of the acceptance test.

Use a transitional mode rather than a hard cutover. MongoDB supports three TLS modes: allowTLS (server accepts TLS but does not advertise it), preferTLS (accepts both and advertises TLS in the handshake), and requireTLS (rejects plaintext). preferTLS is the bridge that lets old and new clients coexist.

Rollout:

1. Distribute the CA bundle and server certificates. tlsCAFile accepts a concatenated bundle, so include both the current and the incoming CA - this is what makes rotation possible without coordination.
2. Rolling restart with tls.mode: preferTLS, secondaries first, then stepDown the primary. Clients that have not been updated keep working in plaintext.
3. Migrate clients connection string by connection string; watch mongod logs for handshake failures and serverStatus().connections for any drop in the pool.
4. When every client is on TLS, flip to requireTLS - a rolling restart, or setParameter tlsMode:"requireTLS" on versions that allow it at runtime - and verify you cannot connect without TLS.

Quarterly rotation, repeating:

• Add the new CA to the bundle on every node before issuing any new certificate. Keep the old CA present until rotation is complete.
• Issue new member certificates and replace them one node at a time, secondaries first; internal x.509 handshakes fail if a member presents a certificate the peer cannot verify, so the bundle must already trust both.
• Rotate client certificates and service credentials on their own schedule; do not rotate both cluster certificates and client certificates in the same hour.
• Only after all nodes and clients present new-chain certificates, remove the old CA from the bundle and verify a fresh connection succeeds.

Record the certificate expiry dates in monitoring with an alert at 60 days; expired certificates are the most common cause of a self-inflicted outage in this area.