AWS Machine Learning Specialty (AWS) common mistakes
AWS Machine Learning Specialty · Understand the traps and how to avoid them.
Picking Amazon Kinesis Data Streams whenever a question mentions streaming ingestion, even when the requirement is a fully managed, no-code delivery into S3 or Redshift, or when the source is clickstream data that should go through Kinesis Data Firehose.
Both services contain the word Kinesis and candidates memorize a single 'streaming = Kinesis' heuristic instead of distinguishing the low-level shard-based stream from the managed delivery service with built-in buffering and S3/Redshift/OpenSearch destinations.
What to do instead
Memorize the split: Data Streams gives shards, custom consumers, replay, and sub-second latency; Firehose gives near-real-time delivery, automatic batching/compression/format conversion, and no shard management. Map 'no consumer code, land in S3/Redshift' to Firehose and 'custom real-time processing with replay' to Data Streams.
Choosing AWS Glue DataBrew or Glue ETL jobs for orchestration, or choosing Step Functions for data transformation, instead of separating orchestration from transformation.
Glue, DataBrew, and Step Functions all appear in ETL scenarios, and candidates conflate the transformation engine with the workflow scheduler. They also forget that Glue workflows and triggers exist.
What to do instead
Assign roles explicitly: Step Functions or Glue workflows orchestrate and handle retries/branching; Glue jobs or DataBrew perform Spark-based or visual transformation; EventBridge schedules. When a question asks for 'coordinate multiple dependent jobs with retries', the orchestration service is the answer.
Storing raw training data in Amazon EFS or instance store, or using a single small EBS volume, when the scenario calls for high-throughput, shared, POSIX-compatible access for many training workers.
Candidates default to the storage service they use on a laptop or pick S3 for everything, missing that SageMaker training jobs and multi-node workloads often need a shared file system, and that FSx for Lustre can link to an S3 bucket.
What to do instead
Use S3 as the durable source of truth for datasets and model artifacts, and FSx for Lustre as the high-throughput scratch layer linked to S3 when training needs POSIX semantics. EFS is for shared low-throughput file access; instance store is ephemeral.
Treating every numeric feature as continuous and skipping distribution checks, then being surprised by questions that require detecting skew, outliers, or class imbalance before modeling.
Candidates jump straight to algorithm selection and treat EDA as optional narrative, so they miss signals like a heavily skewed target that calls for a log transform or a 99:1 class ratio that invalidates accuracy as a metric.
What to do instead
Run a fixed EDA checklist before modeling: summary statistics, histograms and box plots, missing-value rates, correlation matrix, target distribution, and class balance. Then state the consequence: transform skewed features, choose precision/recall/F1 or AUC for imbalance, and remove or cap outliers with justification.
Imputing missing values or scaling features on the full dataset before splitting into train and test.
It feels efficient to clean the whole table once, and candidates do not realize the imputation statistics or scaler parameters leak information from the test set, inflating reported performance.
What to do instead
Split first, then fit imputers, encoders, and scalers only on the training fold and apply them to validation and test data. In SageMaker, put these steps inside the training container or a pipeline processing step so the same fitted transform is reused at inference.
Reporting accuracy on a fraud-detection or defect-detection dataset with a rare positive class, and concluding the model is good.
Accuracy is the default metric in most libraries and tutorials, and candidates do not connect a 1% positive rate with a 99% accuracy achieved by predicting the majority class.
What to do instead
For imbalanced problems, report precision, recall, F1, and AUC-PR, and inspect the confusion matrix. Choose the metric from the business cost of false positives versus false negatives, and consider class weights, resampling, or threshold tuning instead of relying on default 0.5.
Using K-means or another centroid method for arbitrary-shaped clusters or for categorical data, or selecting k without justification.
K-means is the most familiar clustering algorithm, so candidates apply it by reflex and ignore that it assumes convex, similarly sized clusters and numeric distance, and they skip the elbow or silhouette analysis.
What to do instead
Match the algorithm to the data: K-means for compact numeric clusters, DBSCAN for arbitrary shapes and noise, hierarchical for a dendrogram of nested structure, and appropriate encodings for categorical data. Justify k with an elbow plot or silhouette score and state how outliers are handled.
Choosing a deep neural network for a small tabular dataset, or choosing linear regression for a clearly nonlinear relationship, without a baseline.
Candidates associate 'machine learning' with complex models and skip the cheap baseline, so they cannot tell whether added complexity actually helped and they miss regularization or feature-engineering alternatives.
What to do instead
Always establish a baseline first (majority class, mean predictor, or logistic/linear model), then justify added complexity with validation evidence. For small tabular data prefer gradient-boosted trees with regularization; reserve deep learning for large unstructured inputs such as images, text, and audio.
Tuning hyperparameters on the test set, or using a single train/validation split on a small dataset, and then reporting the best score as generalization performance.
Candidates iterate against whatever labeled data is available, and repeated evaluation on the same holdout gradually overfits the choice of hyperparameters to that split.
What to do instead
Use k-fold cross-validation on the training data for model selection, keep a held-out test set untouched until the final evaluation, and use SageMaker automatic model tuning with an objective metric computed on validation folds. Report the final metric only from the untouched test set.
Deploying a model with a real-time SageMaker endpoint for batch scoring of a large historical dataset, or ignoring autoscaling, model monitoring, and A/B rollout options when the scenario mentions cost, drift, or safe release.
Real-time endpoints are the first deployment pattern taught, and candidates do not map workload characteristics to the four inference options or connect monitoring and shadow testing to operational requirements.
What to do instead
Match pattern to workload: batch transform for large offline scoring, serverless inference for intermittent low-volume traffic, asynchronous inference for large payloads or long processing, real-time endpoints for low-latency interactive calls. Add endpoint autoscaling, Model Monitor for data and model quality drift, and shadow or canary variants for safe rollout.