Model Customization, Fine-Tuning, and Continuous Evaluation
Core concepts
Know the customization spectrum: prompt engineering, few-shot examples, Retrieval-Augmented Generation, then fine-tuning and continued pretraining. Amazon Bedrock supports fine-tuning for select models, with training and validation data in JSONL, and provisioned throughput required to host custom models. Understand Bedrock model customization jobs, hyperparameter choices (epochs, learning rate, batch size), and the difference between fine-tuning (behavior/style/format) and RAG (fresh factual knowledge). Know Amazon Bedrock Evaluations: automatic metrics (accuracy, robustness, toxicity) and model-as-a-judge with custom metrics, plus human evaluation workflows. Understand A/B testing, shadow evaluation, and regression suites in CI.
Traps
- Fine-tuning to inject frequently changing facts; RAG is usually the better fit.
- Overfitting from too many epochs, visible as strong training loss but weak validation performance.
- Forgetting custom models need provisioned throughput and cannot be invoked on demand.
- Treating a one-time evaluation as sufficient; continuous evaluation catches drift after prompt or data changes.
- Confusing guardrail evaluation with model quality evaluation.
Worked example
A team wants consistent JSON formatting for a fixed schema. Rule of thumb: try few-shot prompting first; if format compliance remains below target across many inputs, fine-tune on curated input/output pairs and hold out a validation set to detect overfitting.
Rule of thumb
Fine-tune for form and behavior, retrieve for facts. Always keep a held-out evaluation set.