I85PREP
⌘ K
Confluent Kafka Developer study materials
Confluent Kafka Developer study resources

Multiple Event Types in One Topic and a Breaking Field Removal

Confluent Kafka Developer · Put your knowledge to work in context.

All case studies

Multiple Event Types in One Topic and a Breaking Field Removal

Topic payments.events carries two Avro value types, PaymentAuthorized and PaymentCaptured, produced by different services. Schema Registry uses the default TopicNameStrategy, so both services try to register under subject payments.events-value and the second one fails with an incompatibility error. To unblock, an engineer sets global compatibility to NONE, registers both, and moves on. Three weeks later a producer removes the currency field (no default) from PaymentAuthorized. Java consumers are upgraded immediately; a Go service is upgraded two releases later and starts panicking on a null currency. The service account used by the Go service is a Developer on subject payments.events-value only.

1. What naming strategy should have been used, and what operational consequences does switching now have?

Read the worked answer

TopicNameStrategy assumes exactly one value schema per topic, so it cannot express two types. Use RecordNameStrategy (subject = fully qualified record name, e.g. com.acme.PaymentAuthorized) or TopicRecordNameStrategy (subject = payments.events-com.acme.PaymentAuthorized). The former lets the same record type be shared across topics; the latter keeps per-topic isolation of that type.

Consequences of switching:

  • Subject names change, so all existing schemas must be re-registered under the new subjects. Schemas are re-registered lazily on first produce/consume; set the mode/compatibility on the new subjects before traffic arrives.
  • Producers and consumers must both set value.subject.name.strategy to the matching class; a mismatch produces a SerializationException about a missing subject, not a helpful schema error.
  • RBAC must be extended. The Go service account is a Developer on the old subject name only and will get 403 on the new ones. Grant per-subject before cutover, not after.
  • Global NONE should be reverted immediately; set mode per subject (/config/<subject>) when a genuine exception is needed.

2. Was removing `currency` a schema-compatibility violation, and how should the two consumer teams have been sequenced?

Read the worked answer

Under the default BACKWARD compatibility, removing a field is allowed: a new consumer reading old data simply ignores the field, and old data still deserialises. That is why nothing blocked the registration - but it says nothing about whether applications use the field. This is a semantic break dressed as a compatible change.

Correct sequencing for a field removal:

  1. Deprecate first: keep currency in the schema, stop populating it, and tell consumers.
  2. Add a replacement with a default if one is needed; adding a field with a default is backward compatible, adding one without a default is not.
  3. Only remove the field once every consumer is off it. Here the Go team lagged by two releases, so removal should have waited or been published as a new record type.

Orientation reminder: BACKWARD = new schema reads old data (consumers upgrade first); FORWARD = old schema reads new data (producers upgrade first); FULL requires both. A team that needs to add a required field must use a two-phase rollout or FULL_TRANSITIVE plus a versioned record, not a single breaking push.

3. Design the safe path forward for this topic that respects both compatibility and least privilege.

Read the worked answer
  1. Restore discipline: set global compatibility back to BACKWARD (or FULL_TRANSITIVE for the payment records) and fix any subject left in NONE.
  2. Fix subject naming: pick RecordNameStrategy for the shared topic, pre-register both record types under their new subjects with the target compatibility, then roll producers and consumers with matching value.subject.name.strategy. Run both subject sets in parallel during migration.
  3. Restore currency if any consumer still needs it, or introduce currency_code with a default and a documented deprecation window for currency.
  4. Governance: replace the broad Developer grants with per-subject roles scoped to the new subject names (read for consumers, write for producers), and keep the service account off the global config resource so it cannot change compatibility mode again.
  5. Guard the future: a CI check that runs POST /compatibility/subjects/<subject>/versions/latest with the proposed schema before merge, so breaking changes fail the build rather than production.