Partial Cloud Discovery, a Rotated Secret, and Missing Kubernetes Controllers
Three targets were onboarded last week by three different engineers. An AWS account was added using a cross-account IAM role; the target validates successfully, but only 38 of the 380 running EC2 instances appear and no RDS instances are discovered. An Azure subscription was added with an app registration; the target flips to Failed every three to four hours and recovers after a platform restart, and the Azure team rotated the client secret nine days ago. A Kubernetes cluster was onboarded with Helm-installed kubeturbo; nodes and pods are discovered, but no controllers such as Deployments or StatefulSets appear, no namespace hierarchy above the pods, and no actions are generated for any workload. Separately, an existing AppDynamics target reports business transaction response times, but only two of six business applications appear as application entities in the supply chain. Change control requires a ticket for any IAM policy edit, and the Azure team will not send secrets over chat.
1. Explain why the AWS target validates yet discovers 38 of 380 instances and no RDS, and how you would proceed.
Read the worked answer
Treat a successful validation as proof of connectivity, not of complete permissions. Validation exercises a subset of API calls; discovery exercises many more.
- Open the target's status and detail view. Most partial discoveries report per-service messages. An AccessDenied on
rds:DescribeDBInstancesexplains the absent RDS entities without invalidating the target at all. - Inspect the target scope: which regions are enabled, and whether a tag filter or resource filter was set during onboarding. 38 of 380 with no RDS at all is the classic signature of a single-region scope plus a policy that grants only
ec2:Describe*. - Compare the role's attached policy against the documented least-privilege policy for this release. A missing
tag:GetResourcesalone can cost you the tags and relationships the supply chain relies on. - Confirm whether the other 342 instances are stopped or terminated rather than undiscovered, and whether they belong to a second account that was never added.
Because a ticket is required, assemble the exact missing IAM actions with the error text as evidence and raise one ticket. Do not add the account with an administrator role in the meantime - that creates a credential that outlives the ticket and defeats the audit. After the policy change, revalidate and reconcile entity counts against the EC2 console.
2. The Azure target fails every few hours and recovers on restart. What is your diagnosis and remediation?
Read the worked answer
The cadence is the clue. Failure every three to four hours with recovery after a restart matches credential or token lifecycle, not a network fault.
Check two things together: the target's error text from the last failure, which should carry an AADSTS code identifying the exact condition (expired or invalid client secret, wrong tenant, or a lost role assignment), and the rotation date against the last successful validation. Nine days between rotation and onset is close to conclusive.
Most likely the secret was rotated and the platform still holds the previous value. Do not read 'it recovers after a restart' as the restart being the fix - the restart forces a fresh authentication that may transiently succeed while a cached token was doing the work before. That transient success is what makes this look intermittent rather than broken.
Remediate through the approved channel: the Azure team deposits the value in the vault, you retrieve it there, and you update the target credential, save and validate. Never accept a secret pasted into chat; that value then has to be rotated again on principle.
Reduce recurrence: request certificate-based credentials with twelve-month validity where the tenant allows it, record the expiry in the CMDB, and set a 30-day reminder. Also confirm the app registration still holds Reader at the subscription scope - secret rotations often coincide with a subscription move that silently drops the assignment.
3. Diagnose the Kubernetes controller gap and the two missing AppDynamics applications.
Read the worked answer
Kubernetes. Nodes and pods appearing while controllers do not points at RBAC or version, not reachability. kubeturbo needs cluster-scoped list and watch on the workload API groups. A ServiceAccount bound to a namespace-scoped Role, or an older manifest applied against a newer API version, produces exactly this pattern: core resources are visible, controllers are not, and with no controller entity there is nothing to scale or resize, which also explains the absence of actions.
Apply the cluster role and cluster role binding shipped with the current kubeturbo release, restart the kubeturbo pod, and confirm controller counts through the API and the target's discovery detail. Verify the kubeturbo version is supported by the 8.12 platform, and check that the Helm release went into the namespace the platform expects with the intended kubeconfig context.
AppDynamics. Two of six applications showing means the target is healthy, so the fault is selection or permission, not connectivity. Check in this order: the target's application filter, which if populated with an explicit list or a regex silently excludes the other four; the AppDynamics-side permissions of the account the target uses, which may be scoped to specific applications; and whether the four missing applications have tiers whose nodes map to VMs actually managed by your hypervisor targets - an application reporting hosts outside scope cannot join the supply chain. Confirm names in the AppDynamics controller against what the platform discovered before changing configuration.