The deduplication job that kept 14% duplicates
A content team ingests 2.3M product tags nightly. Tag is declared as data class Tag(val name: String, val aliases: Array<String>). The deduplication step does tags.distinct() and then writes to a HashSet<Tag> before a groupBy. After a refactor moved alias parsing into the data class, the job's output grows: 14% of tags remain duplicated, heap usage rises from 1.1 GB to 1.8 GB, and a downstream assertEquals(expectedSize, actualSize) test fails only when aliases are non-empty. A developer replaces Array<String> with List<String> in a branch and the duplicates disappear, but another developer warns that copy(aliases = newAliases) will now share the list and that equality semantics might still be wrong if order should be ignored. The build uses Kotlin 2.0, JVM 17, and the data class is part of a public SDK consumed by two other teams.
1. Why does `data class Tag(val name: String, val aliases: Array<String>)` break `distinct()` and `HashSet`, even though the generated `equals` and `hashCode` exist?
Read the worked answer
data class generates equals and hashCode from the primary-constructor properties. For aliases: Array<String>, generated equals uses aliases == other.aliases, which for arrays is referential equality, not content equality. Generated hashCode uses the array's identity hash. Two Tag instances with the same name and equal alias contents but different array instances therefore compare unequal and land in different hash buckets. distinct() is implemented with a LinkedHashSet, so it relies on both equals and hashCode; it keeps both. The downstream HashSet behaves the same way, and groupBy creates separate groups. The test fails only when aliases are non-empty because that is when the parser allocates a fresh array with meaningful contents; empty aliases may be a shared constant or deduplicated earlier, masking the bug. The heap increase is consistent with retaining 14% more Tag objects and alias arrays. This is not a Kotlin compiler bug: Array is a JVM array with identity equality, and the data class cannot infer whether you want structural or referential semantics. The fix must make equality and hashing structural.
2. Recommend a fix that preserves order-insensitive alias semantics. Compare `List<String>`, a custom `equals`/`hashCode`, and a wrapper value class.
Read the worked answer
Three options, in increasing order of semantic precision. (1) List<String>: generated equals/hashCode delegate to AbstractList, which is structural and ordered. This fixes duplicates if alias order is meaningful, but copy(aliases = newAliases) shares an immutable list-safe only if the list is never mutated. Prefer List over MutableList in the public API. (2) Set<String>: set equality is order-insensitive and rejects duplicates, matching 'aliases' semantics; iteration order is not guaranteed unless you use LinkedHashSet. (3) Custom equals/hashCode on the data class using aliases.contentEquals(other.aliases) and aliases.contentHashCode(), optionally sorting a copy first for order-insensitive comparison. Custom implementations keep the Array ABI but are easy to get wrong: you must also handle name, nullability, and subclass symmetry. A wrapper such as @JvmInline value class AliasSet(private val values: Set<String>) centralizes normalization and prevents raw arrays from leaking. For order-insensitive aliases, normalize once in a factory (aliases.map(String::trim).sorted()), not in equals, so hashing is stable. Avoid mutating arrays after construction; copy on an array-backed data class shares the backing array, so callers can change a tag's hash after it is in a set.
3. The data class is in a public SDK. Which change is safest for binary compatibility, and what would you do about `copy`?
Read the worked answer
Changing the primary-constructor property from Array<String> to List<String> is binary incompatible. The JVM constructor descriptor changes from ([Ljava/lang/String;)V to (Ljava/util/List;)V, component2() changes return type, and the generated copy signature changes. Already-compiled SDK consumers will fail with NoSuchMethodError or NoSuchFieldError even if source recompiles cleanly. For a public SDK, the safest binary-compatible change is to keep the Array constructor and add explicit equals/hashCode using contentEquals/contentHashCode, plus a withAliases function that defensively copies. That changes behavior (fixes equality) but not ABI. However, copy still shallow-copies the array reference; document it and consider deprecating it. If you need List semantics, introduce a new type or a new secondary constructor in a minor release, and plan the primary-constructor change for a major version. Adding a secondary constructor(name: String, aliases: List<String>) : this(name, aliases.toTypedArray()) is source-friendly but does not change copy. Also add a binary-compatibility check (japicmp or Kotlin binary-compatibility-validator) to CI so this decision is enforced.