A Lakeflow Job runs an Auto Loader stream on a job cluster and terminates the cluster after each run. The stream's checkpointLocation points at a scratch path on storage that the job wipes and recreates empty whenever a new cluster starts. Every morning, the target Delta table contains duplicate rows re-ingested from the full landing directory. Where is the configuration error?

1. A. The checkpointLocation points at ephemeral storage; the checkpoint must live in durable cloud storage, in a location dedicated to this stream alone ✓ A
2. B. The checkpointLocation is fine as configured, because the DBFS root of the workspace is the recommended default location for production stream state 💬 B
3. C. The checkpointLocation is irrelevant, because checkpoints are optional when the stream writes its output into a Delta table 💬 C
4. D. The checkpointLocation can be any random path, because Auto Loader rebuilds an equivalent checkpoint from the target table on every run 💬 D

Reveal answer

Correct answer: The checkpointLocation points at ephemeral storage; the checkpoint must live in durable cloud storage, in a location dedicated to this stream alone

Reference: https://docs.databricks.com/aws/en/ingestion/cloud-object-storage/auto-loader/best-practices