A new Auto Loader stream must ingest events from a landing directory that already holds two years of history: about one million small JSON files produced by a decommissioned legacy platform. Going forward, only a few dozen new files arrive each day. The stream's very first micro-batch tried to absorb the entire backlog at once and the job ran out of memory. How should the team bound the initial load?

1. A. Split the source directory into one prefix per year and run one Auto Loader stream per year, all in parallel 💬 A
2. B. Set cloudFiles.maxFilesPerTrigger to a fixed cap, so each micro-batch ingests at most that many files and the backlog drains gradually ✓ B
3. C. Turn cloudFiles.useNotifications off for the first run, because file notification mode is what forces huge micro-batches 💬 C
4. D. Raise the worker count on the job cluster so the single first micro-batch has enough memory to process the whole backlog 💬 D

Revelar gabarito

Resposta correta: Set cloudFiles.maxFilesPerTrigger to a fixed cap, so each micro-batch ingests at most that many files and the backlog drains gradually

Referência: https://docs.databricks.com/aws/en/ingestion/cloud-object-storage/auto-loader/