OCI Observability Professional study materials
OCI Observability Professional study resources

Service Connector Hub Stops Delivering Error Events to SIEM

OCI Observability Professional · Put your knowledge to work in context.

All case studies

Service Connector Hub Stops Delivering Error Events to SIEM

A security team uses two Service Connector Hub connectors in compartment prod-obs. Connector A reads log group prod-app-logs and writes all records to Object Storage bucket app-archive under prefix year=2025/. Connector B reads the same log group, applies a filter for level = 'ERROR', and writes to stream siem-errors in compartment sec-streams. The stream is consumed by a third-party SIEM. After a change window, Connector A is producing objects, but Connector B shows ACTIVE with no recent events; the SIEM has received nothing for 90 minutes. The log group receives about 4,000 records/minute, including 150 ERROR records/minute. A recent application change renamed the JSON field from level to severity. The connector's service principal is in dynamic group sch-prod; the stream is in a different compartment.

1. Diagnose why Connector B delivers nothing while Connector A works. Consider source, filter, target, IAM, and connector state.

Read the worked answer

Connector A proves the source log group is readable and that at least one connector in prod-obs can write to its target. The strongest difference is the filter: Connector B filters on level, but the application now emits severity. If the filter requires the old field, every record fails the predicate and is dropped before the stream target. The connector can still show ACTIVE because ACTIVE describes the connector lifecycle, not whether the filter matched anything.

Check in order:

  • Filter expression against a real recent record: does the field exist, is it at the expected JSON path, and is the comparison case-sensitive? Test severity = 'ERROR' and a value that should match.
  • Source scope: is Connector B reading the same log group and all custom logs, or only a subset?
  • Target stream: is siem-errors active, in sec-streams, and accepting writes?
  • IAM: Connector B's service principal in sch-prod needs read access to the source log content and write access to the stream in the other compartment. Cross-compartment targets are a common policy gap.
  • Connector metrics and errors: look for read records, filtered records, write failures, and throttling. A flat read count means source/filter; reads but no writes means target/IAM.

Fix the filter first if it references level; then confirm with a test record.

2. What IAM and compartment design would you use so the connector can write to the stream in `sec-streams` without broad permissions?

Read the worked answer

Use a dedicated dynamic group for Service Connector Hub service principals, for example sch-prod, with a rule that matches only the connectors that need this path, not every connector in the tenancy. Grant permissions in the compartments where the resources live:

  • In the source compartment: a policy that allows the dynamic group to read the specific log group or log content needed by the connector.
  • In sec-streams: a policy that allows the dynamic group to push to the specific stream, not all streams. If the platform requires compartment-level policy, scope to the compartment and document why.
  • Avoid any-user policies and avoid granting manage rights on streams; the connector only needs to produce.

Keep separate connectors for separate security domains if their filters or targets differ. A single connector with a wide IAM policy is harder to audit. Also confirm the service principal is the connector resource, not a user principal; the dynamic group rule must match the connector OCID or a defined tag on the connector. Test with a low-volume stream before pointing at the SIEM stream.

3. How would you prevent duplicate archive objects and verify no data loss after the fix?

Read the worked answer

Duplicate objects usually come from overlapping sources or targets: two connectors reading the same log group into the same bucket prefix, a connector restarted while another still runs, or a prefix scheme that does not include a unique connector or log identifier. Give each connector a distinct prefix, for example app-archive/connector-a/, and avoid two connectors writing the same source to the same prefix. If both archive connectors must exist for redundancy, use different buckets or different prefixes and document that duplicates are intentional.

To verify after the fix:

  • Compare Connector B's read count and filtered count with the expected 150 ERROR records/minute from the source.
  • Check the SIEM stream backlog and the consumer's acknowledgement; a connector write does not prove the SIEM processed the event.
  • Search Logging for ERROR records in the gap window and compare counts with the stream. If the connector has no replay of the 90-minute gap, use a one-time Logging search export to the stream or SIEM and label it as backfill.
  • Monitor connector metrics for read, filtered, write, and error counts, and alert on a sudden drop in filtered records while source volume is steady.