A five-hop order pipeline that cannot be traced, at 200 GB/day

Application Insights correlates automatically over HTTP because W3C traceparent rides in headers and the SDK reads it. The Service Bus hop silently breaks the chain: the message body carries no headers, so unless the sender writes the trace context into the message's application properties and the receiver reads it back and creates a child Activity, the second function starts a fresh root operation. That is exactly what you would see: two unrelated operation_Id trees and no way to follow order 88123 across the boundary.

Fix:

1. On send, set ServiceBusMessage.ApplicationProperties["Diagnostic-Id"] (or traceparent) from Activity.Current. Recent versions of Azure.Messaging.ServiceBus do this automatically in ServiceBusSender; verify the package version before writing custom code.
2. On receive, if not automatic, extract the property and start an activity with it as parent - ActivitySource.StartActivity(..., parentContext) - so the SDK links the child request.
3. Do not rely on correlation alone for business investigation. Set the business key as a custom dimension on every item: telemetryClient.Context.GlobalProperties["orderId"] in a middleware, or the scope-based TelemetryClient.StartOperation with properties. Then union requests, traces, exceptions | where tostring(customDimensions.orderId) == "88123" answers the support question regardless of sampling or correlation gaps.

Correlation tells you the shape of a healthy path. Dimensions answer the actual support ticket.

Partially - and the 4% figure is a red flag you should chase before touching a config value, because 4% is far more aggressive than adaptive sampling typically goes under steady load.

Two distinct things to separate:

1. Adaptive sampling is per-application and does not propagate the decision. Each function samples independently, so a sampled-in request in hop one can have a sampled-out child in hop two. That produces orphaned traces even when overall volume looks fine. Fix: use a fixed-rate sampling with the same rate on both apps, or ensure parent-based (Inheritance) sampling so children follow the parent's decision.
2. Something else is dropping data. Check the ingestion endpoint / instrumentation key (Application Insights endpoint restrictions, private link misconfiguration, or samplingPercentage set in a host.json or appsettings override that a deployment introduced), and check the requests count against APIM's own metrics, which are independent. A 4% figure smells like a broken pipeline or a stale key more than a sampling setting. Sampling also does not reduce requests to 4% while leaving exceptions intact.

Then set sampling overrides so you are never debugging from sampled data where it matters:

• 100% (no sampling) for requests with a non-success result code, exceptions, and anything carrying a specific orderId failure flag.
• Aggressive sampling for successful requests and dependencies.
• Optionally move dependencies out of App Insights entirely to reduce volume - see the cost task.

The judgement: never diagnose a data-loss complaint by adjusting sampling first. Verify the pipeline, then tune.

You cannot meet both goals with retention settings alone - the reduction must come from what you ingest and the audit need must move to cheaper storage.

Reduce ingestion (the 40%):

1. Attack the 70%. requests and dependencies at default verbosity from two function apps is enormous. Two moves: lower the sampling rate for successful requests/dependencies only (keeping failures at 100%), and stop logging HTTP client dependencies you do not need, using samplingExcludedTypes inverted - i.e. exclude Dependency from unsampled retention.
2. Ingestion-time filtering with a data collection rule. For workspace-based App Insights, DCR transformations can drop rows matching harmless patterns (health probes, a chatty /status endpoint, 200s from a known canary) before they are billed. This is cheaper than querying and discarding.
3. Basic Logs. Verbose custom tables that are rarely queried can be set to the Basic plan: much cheaper ingestion, and KQL still works, just without alerts on that table and with the restricted query surface. Good for the trace volume; not appropriate for the tables you alert on.
4. Metrics, not logs, for alerting. Alerts on platform metrics and a handful of custom metrics cost nothing to query and are not sampled.

Audit for 90 days:

• Keep 30 days interactive in the workspace, then use a workspace retention and archive policy so data is searchable via a search job or restoration beyond the interactive window - cheaper than 90 days interactive.
• Or export failures only (a scheduled query on exceptions/failed requests pushing to Blob with an immutable policy) - 90-day auditability of failures does not require 90-day retention of 200 GB/day of successes.

Do not propose: turning sampling up and calling it done, or extending the workspace retention to 90 days, which increases cost rather than cutting it.