IBM Instana Observability Administrator - Professional cheat sheet

Agent packages are distributed from the Instana repository keyed to the tenant's download key; the same key string appears in the agent install script and in the agent configuration file as com.instana.download-key.

Static agent install (Linux) uses the repository script plus the package manager; dynamic agent install uses a one-line shell command with the download key and agent key, and both result in a systemd unit named instana-agent.

A host agent needs outbound connectivity to the Instana backend endpoints (agent-acceptor and agent-ingress style FQDN endpoints on TCP 443) unless an on-premises or proxy/gateway relay is configured.

The agent key (agent.key) is a per-tenant identifier embedded in configuration; the download key authorizes package retrieval. Rotating the agent key requires redistributing configuration to every host.

Kubernetes deployment uses the Instana Helm chart (instana-agent) or the operator; the DaemonSet runs one agent pod per node and reads the agent key and endpoint from a Secret/ConfigMap.

Upgrades: Linux agents upgrade via the package manager on the configured repository; containerized agents upgrade by changing the image tag or Helm chart version and rolling the DaemonSet.

Version compatibility rule: the agent must be at or above the minimum supported version for the backend release; the Instana UI shows outdated agents and the backend rejects agents below the minimum.

Agent installation verification: check the instana-agent service status, the agent log at /opt/instana/agent/data/log/agent.log, and host appearance under Infrastructure within a few minutes.

Agent configuration lives in configuration.yaml (host agent) with sections for com.instana.agent, com.instana.plugin.* sensors, and host tags; changes require an agent restart or configuration reload.

Dynamic Focus Query syntax is a boolean expression of tag filters, e.g. entity.type:docker AND docker.name:web*, combined with AND, OR, NOT and parentheses; it filters the infrastructure and application views.

DFQ operators include equality (:), negation, wildcards (*), and the special tags entity.type, entity.label, and entity.zone; queries are saved and reusable across dashboards and alerts.

Sensor enablement is controlled per plugin (e.g., com.instana.plugin.javatrace, com.instana.plugin.nginx) via enabled: true/false; disabled sensors stop data collection but the agent keeps running.

Remote sensor configuration can be pushed from the Instana UI (agent configuration management) and overrides local file settings for supported sensors.

Tagging: host tags defined in configuration.yaml under tags: become entity tags and drive DFQ scoping, alert filters, and application perspective membership.

Agent configuration precedence: remote/UI-managed configuration takes effect over local defaults for managed keys; verify effective config in the agent's configuration view.

Sensor health is visible per host in the Instana UI; a sensor in error state usually indicates missing permissions, unsupported runtime version, or blocked local socket.

Host monitoring collects CPU, memory, disk, network, and process metrics; process monitoring requires the agent to enumerate processes and can be scoped to specific process names.

Docker and containerd monitoring requires the agent to access the container runtime socket (e.g., /var/run/docker.sock) and the host cgroup filesystem; missing socket access yields container metrics gaps.

Kubernetes monitoring: the agent DaemonSet plus the k8s sensor collects pod, node, deployment, and namespace metrics; cluster-wide objects require the agent's service account RBAC.

Entity model: hosts, containers, and processes are infrastructure entities; DFQ and alert filters address them by entity.type and tag values.

Custom metrics and custom events can be sent to the agent's local endpoint (agent port 42699) and appear as infrastructure metrics tied to the host entity.

Health checks: a host shows as offline when no agent heartbeat reaches the backend within the timeout; check agent connectivity and clock skew before assuming backend issues.

Infrastructure dashboards aggregate by tag; consistent host tagging is a prerequisite for meaningful grouping and for scoping alerts to environments.

Application perspectives are defined by a scope (DFQ over services/endpoints) plus boundary and alerting configuration; a service belongs to a perspective when its tags match the perspective scope.

Tracing requires a tracing sensor per runtime: Java uses the JVM tracing sensor (auto-instrumentation via -javaagent or agent attach), Node.js uses the Node.js sensor, .NET uses the .NET tracing sensor.

Distributed trace context propagates via standard headers (W3C traceparent/tracestate or vendor headers); missing propagation splits a trace into separate traces.

Service naming: the service name comes from the sensor's configured service name or the application's own naming; consistent naming is required for endpoint grouping.

Call monitoring shows service-to-service calls and their latency/error rates; unresolved calls appear as 'uninstrumented' or unknown destinations.

Endpoint detection is automatic for supported frameworks; custom endpoints can be defined where the framework is not recognized.

Trace sampling is configurable per sensor; lowering the rate reduces agent-to-backend volume but also reduces trace detail for rare paths.

Application alerting keys off service-level indicators (latency, error rate, throughput) computed over the perspective's scope; verify the perspective includes the intended services before alerting.

Alerting model: an alerting configuration binds a metric/event condition, a scope (DFQ or application perspective), a threshold and time window, and a severity.

Issue detection is automatic for anomalies (e.g., latency, error rate, saturation) and complements user-defined threshold alerts; both surface as issues in the Instana UI.

Alert channels include email, Slack, PagerDuty, Opsgenie, ServiceNow, webhook, and generic HTTP; each channel needs credentials or a webhook URL configured once and reused.

Alert silencing and maintenance windows suppress notifications for a scope during planned work; silences do not delete the alert configuration.

Event types include incidents, changes, and custom events; custom events can be sent via the agent endpoint or the events API and can trigger alerts.

Smart alerts group related issues so one root cause produces one notification rather than a storm; grouping is by entity and time proximity.

Notification routing: alert configurations are per-application or per-infrastructure scope, so a channel attached to the wrong scope produces no notifications.

Testing a channel sends a test payload; a channel that tests successfully but never fires usually indicates a scope or threshold mismatch.

Access control is role-based: users are assigned roles that grant permissions, and teams group users for scoping application and infrastructure visibility.

Application perspectives and infrastructure scopes can be restricted to specific teams, so a user sees only the entities their team is granted.

SSO integrates via SAML 2.0 or OIDC; the identity provider asserts user identity and group membership, and Instana maps groups to teams/roles.

Role assignment sources: manual assignment in the UI or automatic mapping from IdP group claims; automatic mapping requires matching group names or configured attribute mapping.

API tokens are separate from user credentials; tokens carry permissions of the creating user or a dedicated service account and should be scoped and rotated.

Least privilege: administrative roles (tenant/global admin) should be limited; monitoring roles grant read access to scoped entities without configuration rights.

User lifecycle: deprovisioning in the IdP does not always remove the Instana user immediately; verify removal or rely on SSO-only login with no local password.

Retention is set per data type (metrics, traces, logs) in the tenant settings; longer retention increases storage cost and query latency for long ranges.

Trace retention is typically shorter than metric retention; use metric-based alerting for long-window conditions and traces for detailed diagnosis.

Sensitive data scrubbing: capture rules and sensor configuration can exclude payloads, headers, or query parameters from traces to avoid PII collection.

Agent-to-backend traffic uses TLS on 443; on-premises deployments can terminate TLS at a gateway/proxy, and the agent must trust the proxy's CA.

Data residency: the tenant's backend region determines where telemetry is stored; verify region before onboarding regulated workloads.

Audit: configuration changes and user actions are logged; export or forward audit data where compliance requires an external record.

API tokens and agent keys are secrets; store them in a secret manager and never in plaintext in source control or container images.

First check for a missing host: agent service status (systemctl status instana-agent), then agent.log for connection errors, then outbound DNS/TLS to the backend endpoints on 443.

Diagnostic bundle: the agent can produce a support/diagnostic archive containing configuration and logs; collect it before opening a support case and redact secrets.

Clock skew breaks TLS and trace correlation; verify NTP synchronization on hosts and containers.

Proxy environments require HTTP_PROXY/HTTPS_PROXY (or agent proxy settings) plus NO_PROXY for local endpoints; agents that ignore the proxy fail to reach the backend.

Container agent issues: check the DaemonSet pod status, the agent key Secret, and whether the runtime socket is mounted; CrashLoopBackOff often means a bad key or unreachable endpoint.

Sensor not reporting: confirm the runtime version is supported, the sensor is enabled in configuration, and the instrumentation (agent attach or javaagent) actually loaded.

Agent self-monitoring: the Instana UI shows agent health and version per host; use it to find outdated or disconnected agents at scale.

Support cases need tenant name, agent version, host OS/runtime, timestamps in UTC, and the diagnostic bundle; missing these slows resolution.