An engineer with a background in classic Kafka consumers is setting up a Structured Streaming read from a Confluent Kafka topic and asks where the reading position — which offsets have already been consumed — is maintained for the stream, since that determines what happens after a restart. What should the team tell them?

1. A. In the query's checkpoint on durable storage: offsets are committed every batch, so restarts resume from it ✓ A
2. B. In the broker's consumer-group offsets topic, exactly like an application using the classic Kafka consumer 💬 B
3. C. In memory on the driver, which means every restart begins from the earliest offset unless configured otherwise 💬 C
4. D. In a Databricks-managed table that the team must inspect and repair manually after failures 💬 D

Reveal answer

Correct answer: In the query's checkpoint on durable storage: offsets are committed every batch, so restarts resume from it

Reference: https://docs.databricks.com/aws/en/connect/streaming/kafka/