Implement Observability Practices and Troubleshoot Issues
Signals, logs, and alerts
Know which tool answers which question. Cloud Monitoring covers metrics, dashboards, uptime checks, alerting policies, and SLO monitoring. Cloud Logging covers log storage, log-based metrics, and log sinks. Cloud Trace breaks down request latency, Error Reporting aggregates exceptions, and Cloud Profiler finds CPU and heap hotspots. Managed Service for Prometheus serves Prometheus-native workloads, and the Ops Agent collects VM metrics and logs.
Log routing: the Log Router matches entries against sinks. The _Required bucket holds admin activity audit logs with 400-day retention and cannot be disabled; _Default retains 30 days per project. Aggregated sinks at folder or org level centralize logs into a security project, and exclusion filters control cost by dropping noise before ingestion. Admin Activity audit logs are always on; Data Access logs are opt-in and billed.
Alerting traps: a policy needs a metric, a condition, a duration, and a notification channel. If you alert on a log-based metric, handle absence explicitly - otherwise idle periods look like outages. High-cardinality labels such as user ID or request ID make metrics expensive and slow; aggregate instead. Metrics scopes let one project view many projects' data, which is the usual answer to showing all projects on one dashboard.
Worked example: to page on 5xx spikes, create a log-based counter metric whose filter matches your service and httpRequest.status >= 500, then an alerting policy with a 5-minute alignment window and a threshold of 1% of total requests, routed to the on-call channel. Set the condition to not trigger on missing data so quiet periods stay quiet.
Rule of thumb: troubleshoot outside-in - start from the user-facing SLO breach, drop into traces for the slow span, then read logs for that span's request ID.