AWS Machine Learning Specialty (AWS) common mistakes
AWS Machine Learning Specialty · Understand the traps and how to avoid them.
When a scenario says data must land in S3 or Redshift in near real time with no application code to write, the candidate selects Kinesis Data Streams and describes writing a custom consumer (Lambda or KCL application) to persist the records.
Both services have 'Kinesis' in the name and both ingest streaming data, so candidates pattern-match on the word 'streaming' instead of on the delivery requirement. Data Streams is a substrate you must consume from; Firehose is the managed delivery mechanism.
What to do instead
Memorise the decision rule: if the question mentions delivery to S3/Redshift/OpenSearch/Splunk, automatic batching, optional Lambda transformation, or 'no custom consumer', the answer is Kinesis Data Firehose. Pick Data Streams only when the scenario needs multiple independent consumers, replay from a configurable retention window, sub-second consumer-side processing, or ordering per shard.
Sets up a training job that reads a multi-terabyte dataset from EFS or EBS, or downloads it into the notebook instance's local storage first, and then wonders why training throughput is the bottleneck.
Candidates model storage as 'just attach a filesystem' and forget that SageMaker training containers have no persistent local disk and that EFS is not built for high-throughput parallel reads from many GPU hosts.
What to do instead
Default to S3 as the source of truth and choose the access pattern deliberately: File mode for small datasets, Pipe mode or augmented manifest files for large sequential-read workloads, and Amazon FSx for Lustre linked to the S3 bucket when the scenario describes high-throughput, low-latency random access or repeatedly reading the same large dataset across many training jobs.
Computes imputation values, scaling statistics, target encoding, or feature-selection rankings on the entire dataset and only afterwards calls train_test_split, then reports the resulting holdout score as an unbiased estimate.
Scikit-learn-style code reads more naturally as 'clean, then split' and the leakage is invisible in the code review - the model looks excellent, so there is no signal that something is wrong until production.
What to do instead
Split first, fit every transformer on the training fold only, and apply it to validation/test with transform (never fit_transform). In SageMaker, express this as a Pipeline with the imputer/scaler as steps so the fit is confined to the training channel, or use the sklearn Pipeline object inside the training script.
Treats t-SNE output coordinates as engineered features for a downstream classifier, or reads distances between t-SNE clusters as evidence of how similar the groups really are.
t-SNE plots are visually compelling and cluster-shaped, so candidates assume the embedding is a faithful low-dimensional representation, missing that it preserves local neighbourhoods only and has no stable global geometry.
What to do instead
Use t-SNE strictly for visual inspection of class separability during EDA. For dimensionality reduction that feeds a model, use PCA (or a learned embedding) and choose the component count from explained variance. Read cluster similarity from the original feature space or from PCA space, not from a t-SNE scatter plot.
Reports accuracy as the headline metric for a fraud/churn/defect scenario where the positive class is a few percent of the data, and accepts a model that predicts the majority class for every record.
Accuracy is the first metric in every tutorial, and a 97% accuracy figure looks like success on the dashboard, so the candidate never checks the per-class breakdown or the confusion matrix.
What to do instead
Compute the baseline majority-class rate first, then choose a metric that reflects the business cost: F1 or recall at a fixed precision for rare-event detection, PR-AUC rather than ROC-AUC when the negative class dominates. Address the imbalance explicitly with scale_pos_weight (XGBoost), class weights, resampling, or by moving the decision threshold, and justify the choice from the question's stated cost of false negatives versus false positives.
Repeatedly evaluates candidate models and hyperparameter settings against the same holdout set, or reports the test-set score of the configuration that was selected using that test set.
There is only one labelled split in most notebooks, so the holdout quietly becomes the validation set; the candidate then reports an optimistic number as if it were an untouched estimate.
What to do instead
Use three roles: train for fitting, validation for model selection and early stopping, test touched exactly once at the end. When tuning with SageMaker Automatic Model Tuning, define the objective metric on a validation channel and keep the test channel out of the tuning job entirely; when data is scarce, use k-fold cross-validation for selection and hold out a final test fold.
Runs k-means or PCA directly on raw features where income is in the tens of thousands and age is in the tens, then interprets the resulting clusters as meaningful segments.
Distance-based algorithms are taught with clean toy datasets, so the scale-dependence of Euclidean distance and variance-based component selection is easy to forget when the columns come straight out of a data warehouse.
What to do instead
Standardise or min-max scale numeric features before any distance- or variance-based method, and verify the choice with silhouette score or explained variance. If the scenario describes elongated, non-convex, or density-varying groups (or noise and outliers that must be labelled as such), reach for DBSCAN instead of k-means and justify it from the cluster shape described in the question.
Deploys a real-time endpoint to score a nightly batch of tens of millions of records that nobody consumes interactively, and then sizes instances for throughput and worries about idle cost.
'Prediction' is mentally mapped to 'endpoint' after studying deployment, and the candidate misses that the question describes an offline scoring job with a scheduled cadence.
What to do instead
Match the inference pattern to the consumption pattern: Batch Transform (with MiniBatchSize and a data capture path) for large offline or scheduled scoring, a real-time endpoint only for low-latency interactive requests, a serverless endpoint for intermittent or spiky low-volume traffic, and Asynchronous Inference for large payloads or long processing times with queueing.
Enables a SageMaker Model Monitor schedule and sets baselines, but never turns on DataCaptureConfig on the endpoint, then cannot explain why no violations or monitoring data appear.
Monitoring is configured in a separate API call from endpoint creation, and the dependency on request/response capture is easy to skip because the monitoring schedule creation succeeds without it.
What to do instead
Create the endpoint configuration with DataCaptureConfig enabled (specify the capture percentage and whether requests, responses, or both are captured) before or alongside the monitoring schedule. Then pick the correct monitor type for the question: data quality for covariate/feature drift, model quality for accuracy drift against ground truth, bias drift for fairness metrics, and feature attribution drift for explainability shift.
Enables managed spot training to cut cost but reports job failures or unexplained restarts whenever capacity is reclaimed, because neither checkpointing nor a sufficient MaxWaitTime was configured.
Spot savings are attractive and the checkbox is one line, while the checkpointing contract - that the training script itself must write and resume from checkpoints - is not obvious from the estimator API.
What to do instead
Set use_spot_instances=True together with checkpoint_s3_uri and ensure the training script writes intermediate state to the checkpoint path and resumes from it when present; set max_wait to a value comfortably above max_run so the job can wait out interruptions rather than failing. If the workload is a one-off that must finish on a deadline, use on-demand instances and say so.