Over-Permissive Role, Direct ESXi Logins, and an Expiring Certificate

Why Dev is reachable

• vCenter permissions are additive down the inventory tree. A role granted at the root with propagation applies to every child object, so the Dev folder inherits it. There is no 'deny' in the vSphere permission model that subtracts from an inherited grant - only an explicit No Access assignment, which masks inheritance for that object and its children.

Design

1. Remove the root-folder assignment for the group. Grant the role at the Prod folder only, with 'Propagate to children' checked.
2. If the group must not even see Dev, assign No Access to the group at the Dev folder - but only after confirming no other group membership grants them rights there.
3. Build a custom role rather than reusing 'Virtual machine power user'. It must include the read/interaction privileges the job needs - e.g. System.View, VirtualMachine.Interact.PowerOn/PowerOff/Reset, VirtualMachine.State read - and nothing else. Avoid VirtualMachine.Interact.DeviceConnection and anything granting console access unless justified.
4. Remember the inventory dependency: to see a VM in the vSphere Client, the user also needs read visibility on the parent objects (host/cluster, datastore, network). Grant read-only on those at a scope that does not leak Dev. This is the usual reason a 'Prod-only' design appears broken to the user.
5. Test with a real account that holds only this group, in both folders.

Reasoning - least privilege in vCenter is achieved by scoping the grant and trimming the role; the two errors here are scope (root) and role breadth (power user includes far more than power cycling).

Approach

1. Enable lockdown mode on all hosts - normal mode for most environments, strict only where the operational model supports it (strict also blocks the vSphere Web Services API and read-only users, which breaks some third-party tooling and monitoring).
2. Exception users - add a small break-glass set (a named local account such as the root account, plus a controlled service account) to the lockdown exception list. Exception users are the only accounts able to log in directly while lockdown is active. Keep that list short and reviewed.
3. Decide the AD-join question - joining hosts to AD is what lets domain accounts authenticate at the host at all. If the hosts do not need AD for anything else (host profiles, smart card, scripted access), removing the domain join closes the hole at the source. If AD join must stay, lockdown mode is the control that matters.
4. Change credentials and verify - rotate the root password into the vault, confirm host-level audit logging is going somewhere persistent, and test that vCenter-driven operations (vMotion, HA, DRS, host profiles, vLCM) still work with lockdown enabled. Some workflows depend on the user account, not on root.
5. Log and monitor - alert on direct host logins and on lockdown-mode changes; these are high-signal security events.

Reasoning - the host bypasses all RBAC enforced by vCenter. Lockdown mode is the only supported control that closes that path, and exception users are what prevents it from becoming an outage.

Check first - do not guess the issuer

• Inventory every certificate: machine SSL, each solution user certificate, and the SSO/SAML signing certificate. Use the VAMI certificate page and the certificate manager to list what is present, its issuer, expiry, and thumbprint.
• Determine whether VMCA issued it self-signed, VMCA is subordinate to an enterprise CA, or the certificate was issued externally. That single fact decides the whole procedure.
• Confirm all vCenter services are healthy and take a supported snapshot of the vCenter Server Appliance before any change. Note that certificate replacement restarts services - schedule downtime for the management plane and accept that some operations pause.

Sequence (VMCA-issued or CA-signed, machine SSL replaced)

1. If using an enterprise CA, verify the root and issuing chain is trusted by every component and by the browsers/agents that connect.
2. Replace the machine SSL certificate first, supplying a CSR and the signed certificate plus full chain.
3. Replace solution user certificates (vpxd, vmc, etc.) - these are commonly missed and cause post-change login failures.
4. Verify the SSO signing certificate validity separately; it has its own lifecycle.
5. Restart services and validate: log in with SSO, check each service's health, verify the presented chain with openssl s_client -connect <vcenter>:443 -showcerts, and confirm hosts, VAMI, and any integrations still trust it.
6. Document the new expiry, issuer, and renewal owner.

Reasoning - most certificate incidents are not a renewal problem but a discovery problem. Establish the issuance model and the full certificate inventory before changing the one certificate you already know about.