Clustering Customer Segments When the 'Right' Number Is Contested
A retail analytics team runs k-means on 40 standardized behavioral features for 500k customers. The elbow plot is ambiguous, the silhouette score peaks at k=2 (0.31) and declines slowly to k=8 (0.24), and the business wants 5-7 actionable segments. Features include spend, visit frequency, and recency, which are correlated and skewed; a few customers have extreme spend values. The team also notices that running k-means with different seeds changes cluster membership for about 15% of customers. Marketing plans to build campaigns per segment and needs stable, interpretable groups.
1. Critique the current clustering setup and identify the choices most likely to produce unstable, uninterpretable segments.
Read the worked answer
Several issues compound. (1) Skewed, heavy-tailed spend with outliers distorts Euclidean distances, so a handful of extreme customers pull centroids; standardization does not fix skew. Consider log-transforming spend/frequency and capping or winsorizing extremes before scaling. (2) Correlated features (spend, frequency, recency) double-count the same underlying dimension, inflating its weight; PCA or feature selection could reduce redundancy. (3) k-means assumes roughly spherical, equal-variance clusters, which behavioral data rarely satisfies; the slow silhouette decline suggests no strong natural k. (4) The 15% seed-dependent membership indicates the objective landscape has many near-equivalent optima - a stability problem, not just a tuning problem.
Together these explain why the 'right' k is contested: the method is imposing structure the data does not strongly support.
2. Given the business wants 5-7 segments and stability matters, propose a concrete path forward.
Read the worked answer
Separate the statistical question from the operational one. First, stabilize the representation: log-transform skewed features, handle outliers, reduce correlated dimensions (PCA or drop redundant features), and re-standardize. Then assess stability directly: run k-means with many restarts (e.g., 50 inits) and bootstrap resampling, and measure adjusted Rand index across runs. Choose the k in 5-7 that maximizes stability and interpretability, not the silhouette peak - silhouette at k=2 is statistically tidy but operationally useless.
If k-means remains unstable, try Gaussian mixture models (soft assignment tolerates ambiguity) or hierarchical clustering for a dendrogram the business can inspect. Validate segments by profiling them on held-out customers and confirming the profiles are distinct and actionable. Accept that segments are a decision aid, not a discovered truth.
3. Marketing asks whether the 15% membership churn means the model is broken. How should the team frame this?
Read the worked answer
Not necessarily broken - it means some customers sit near cluster boundaries where assignment is genuinely ambiguous. A 15% flip rate is high enough to matter operationally: if campaigns are triggered by segment, those customers may receive inconsistent treatment across reruns. The team should quantify which customers flip (likely borderline ones with middling feature values) and whether flips change campaign logic.
Framing: report stability as a first-class metric alongside cluster quality. Mitigate by fixing a seed and versioning the model for reproducibility, using soft assignments or a 'boundary/uncertain' bucket for ambiguous customers, and re-running only on a schedule with change monitoring. If stability cannot be improved after the representation fixes, that is evidence the data lacks strong discrete structure, and the team should say so rather than present unstable segments as fact.