Two teams each start their own Auto Loader stream over the same landing directory, writing to the same Delta table, main.bronze.clicks. Each stream has its own checkpoint location and its own Lakeflow Job. Days later, the table holds exactly two copies of every ingested file. Why did the duplicates appear, and what is the correct fix?

1. A. Delta Lake should have detected the distinct checkpoints and deduplicated the writes; the duplicates point to a corrupted transaction log instead 💬 A
2. B. Auto Loader refuses a second stream targeting the same table with a lock error, so one of the jobs must have silently skipped its files 💬 B
3. C. Each stream tracks its own progress, so both discovered and ingested the same files; run a single stream per table or merge the sources ✓ C
4. D. The producer uploaded every file twice into the landing directory, because separate stream writers can never create duplicates by design 💬 D

Revelar gabarito

Resposta correta: Each stream tracks its own progress, so both discovered and ingested the same files; run a single stream per table or merge the sources

Referência: https://docs.databricks.com/aws/en/ingestion/cloud-object-storage/auto-loader/best-practices