Operational Excellence, Monitoring, and Cost Optimization
Core concepts
Know observability for generative AI workloads: Amazon CloudWatch metrics for Bedrock (invocations, latency, throttles, token counts), model invocation logging to CloudWatch Logs or S3, and AWS X-Ray for tracing multi-step agent flows. Understand quota management: on-demand throughput quotas, provisioned throughput for predictable load, and cross-region inference profiles to raise effective capacity. Know cost drivers: input and output tokens, provisioned model units, knowledge base sync and vector storage, and guardrail evaluations. Understand caching strategies, prompt compression, response length limits, and batching for cost control. Know infrastructure as code with AWS CDK or CloudFormation for reproducible deployments and canary releases.
Traps
- Optimizing only model choice while ignoring output token length, often the dominant cost.
- Assuming provisioned throughput is always cheaper; it only wins at sustained high utilization.
- Missing throttle errors caused by per-account quotas rather than model limits.
- Logging full prompts and responses to S3 without lifecycle policies, inflating storage cost.
- Deploying prompt changes without version control or rollback plans.
Worked example
A workload is throttled at peak. Rule of thumb: first check CloudWatch throttle metrics and quota usage; if traffic is spiky, use cross-region inference profiles or retry with exponential backoff and jitter; reserve provisioned throughput only when baseline utilization is consistently high.
Rule of thumb
Measure tokens per request and cost per successful answer, not just latency. Version prompts like code.