To reduce storage costs, a cloud administrator applies a 30-day lifecycle expiration rule to every prefix under the S3 bucket that hosts an Auto Loader stream's checkpoint directory. The stream has run daily for two months against the same input directory. What is the impact, and what should the team have done instead?

1. A. No impact: the checkpoint is rebuilt automatically from the target table's Delta transaction log on the next run 💬 A
2. B. Auto Loader silently switches to file notification mode and re-lists the entire input directory tree on every micro-batch, because its checkpoint state has gone missing 💬 B
3. C. Only files from the last 30 days are reprocessed, because Delta time travel caps recovery at 30 days 💬 C
4. D. The stream state is corrupted and it must restart from scratch, likely reprocessing files; checkpoints belong in durable storage outside lifecycle rules ✓ D

Reveal answer

Correct answer: The stream state is corrupted and it must restart from scratch, likely reprocessing files; checkpoints belong in durable storage outside lifecycle rules

Reference: https://docs.databricks.com/aws/en/ingestion/cloud-object-storage/auto-loader/best-practices