Data Operations and Support
Core concepts
Operations means running pipelines predictably: observability, orchestration, retries/idempotency, and performance tuning.
Observability: CloudWatch metrics, logs, and alarms for Glue, Lambda, Kinesis, and Redshift; CloudTrail for API-level audit of who changed what; AWS Config for resource configuration history; EventBridge for Glue/Lambda state-change events that trigger alerts or corrective workflows. Kinesis exposes IteratorAgeMilliseconds and ProvisionedThroughputExceeded; Lambda shows errors, throttles, and duration; Glue publishes per-job run metrics and supports Spark UI and continuous logging.
Orchestration: Step Functions for state machines, retries, and branching; MWAA for DAG-based scheduling; EventBridge Scheduler for time-based triggers. Design jobs to be idempotent (overwrite the same partition, or use MERGE) so retries are safe.
Tuning: Glue out-of-memory and slow stages usually mean data skew, too few partitions, or collect() on a large dataset - repartition, salt the skewed key, raise worker size/type, or enable grouping of small files. Redshift performance reviews use system tables (STL_QUERY, SVL_QUERY_SUMMARY), and demand ALTER/VACUUM/ANALYZE after bulk loads.
Traps
- CloudWatch is metrics/logs/alarms; CloudTrail is API audit. Questions about "who deleted the bucket policy" are CloudTrail.
- Rising Kinesis iterator age means consumers cannot keep up (or a hot shard), not that the producer is broken.
- VACUUM cannot run inside a transaction block; long-running transactions block vacuum.
Rule of thumb
Throughput drops, executor CPU is low, and a few tasks take far longer than the rest → data skew. Fix with salting/repartitioning before adding nodes.