I85PREP
⌘ K
OCI Observability Professional study materials
OCI Observability Professional study resources

Custom Application Logs Missing from Most OKE Nodes

OCI Observability Professional · Put your knowledge to work in context.

All case studies

Custom Application Logs Missing from Most OKE Nodes

A team runs a 12-node OKE cluster in a private subnet. Each node has the OCI Unified Monitoring Agent installed by a node bootstrap script. Application pods write JSON lines to /var/log/checkout/app.log on the node via a hostPath volume. A custom log named checkout-json in log group prod-app-logs is configured with 30-day retention and a JSON parser for fields level, requestId, and durationMs. The agent uses a dynamic group oke-log-agents whose rule matches the node compartment. After a rolling node image upgrade, only 2 of 12 nodes show records in Logging. The application pods are healthy and writing to disk on all nodes. The Logging service shows no ingestion errors in the console. The team also notices that DEBUG records are about 70% of volume and are not needed for search.

1. What are the most likely reasons only two nodes deliver logs, and what would you check in what order? Include IAM, agent, and log configuration.

Read the worked answer

Check in this order, from the node outward:

  1. Agent health on each node: is the Unified Monitoring Agent service running, and does it report the same configuration version as the two working nodes? The rolling image may have changed the service unit, the config directory, or the resource limits.
  2. Node bootstrap and config: confirm the custom log source path /var/log/checkout/app.log exists on each node and that the agent config was actually rendered on the new image. A hostPath volume can write to a different host path if the pod spec changed.
  3. Dynamic group membership: oke-log-agents must match the new node instances. If the rule uses a freeform tag or a compartment that the new image does not set, the new nodes are not members. Check instance membership directly.
  4. IAM policy: the dynamic group needs the Logging policy that allows the agent to write log content to the target log group. A policy that worked for old nodes can fail if the group rule no longer matches.
  5. Log group and custom log state: both must be enabled, the custom log must reference the correct log group and agent configuration, and the parser must not reject the records. The console may show no ingestion errors if the agent never uploads at all.
  6. Time range and search: a requestId search can miss records if the time range is wrong or the field is parsed as a different type; confirm with a known record on a working node.

2. Design a log routing and retention plan: keep 30-day searchable in Logging, archive 1 year in Object Storage, and send only WARN/ERROR to SIEM. Where do you filter and why?

Read the worked answer

Use a layered plan because filtering has different effects at each layer.

  • Agent or application layer: if DEBUG is not needed for search or archive, drop it before ingestion. This reduces Logging ingest cost and SIEM noise. The tradeoff is that the DEBUG records are gone from the archive too; if they must be retained, do not drop them here.
  • Logging service: keep 30-day retention on prod-app-logs for search and troubleshooting. Use saved searches for level = 'ERROR' and level = 'WARN' so responders do not scan raw volume.
  • Service Connector Hub to Object Storage: create a connector from the log group to a bucket with prefix such as checkout/year=2026/. Apply a lifecycle policy to move objects to lower-cost storage after 30 days and delete after 1 year. This connector can carry all records if archive completeness is required.
  • Service Connector Hub to SIEM: create a second connector with a filter on the parsed severity field, for example severity = 'ERROR' or a set that includes WARN. Target the SIEM stream in its own compartment. Filter before the target so only actionable events consume SIEM license and bandwidth.

Do not use the same connector for archive-all and SIEM-filtered unless the service supports separate filtering per target; otherwise one target's filter can silently drop records for the other.

3. A developer reports that a known `requestId` is absent from Logging search. How would you distinguish an ingestion failure from a search or parser problem?

Read the worked answer

Work from the raw event to the search result. First confirm the pod actually wrote the line on the host and that the file rotated after the request. Then check the agent on that specific node: is it running, is the custom log source enabled, and did it upload in the same minute? If the agent is healthy, compare a known-good record from the same node and minute.

If other records from the same file and minute are searchable, the problem is likely parser or query related: the requestId field may be parsed as a different case, nested under a JSON object, or indexed as a string with quotes. Search the raw log text for the value, then search the parsed field. Check the time range and time zone; a 30-minute offset will hide a recent request.

If no records from that file and minute are searchable on that node, suspect ingestion: agent backoff, log rotation before read, permission on the file, or the dynamic group/IAM path. Use Logging metrics for incoming records by log group and compare nodes. A single missing requestId with all other records present is rarely an IAM problem; a missing minute across a node is.