Noos
PT EN
RAN-DBDEA-00120
Ranvier · DBDEA

Exam DBDEA Topics: ingestion Difficulty: medium

A new Auto Loader stream must ingest events from a landing directory that already holds two years of history: about one million small JSON files produced by a decommissioned legacy platform. Going forward, only a few dozen new files arrive each day. The stream's very first micro-batch tried to absorb the entire backlog at once and the job ran out of memory. How should the team bound the initial load?

Options (Reading Mode) Click to filter discussion →
  1. A. Split the source directory into one prefix per year and run one Auto Loader stream per year, all in parallel
  2. B. Set cloudFiles.maxFilesPerTrigger to a fixed cap, so each micro-batch ingests at most that many files and the backlog drains gradually
  3. C. Turn cloudFiles.useNotifications off for the first run, because file notification mode is what forces huge micro-batches
  4. D. Raise the worker count on the job cluster so the single first micro-batch has enough memory to process the whole backlog
Reveal answer

Correct answer: Set cloudFiles.maxFilesPerTrigger to a fixed cap, so each micro-batch ingests at most that many files and the backlog drains gradually

Reference: https://docs.databricks.com/aws/en/ingestion/cloud-object-storage/auto-loader/

Train this exam on Ranvier →

02 — The Myelin Layer · Canonical Discussion

Where the exam tries to trick you — and how those who passed solved it.

0 Passed

contributed to this question

Answer Distractor Matrix Click an option to filter or open its dissection.
✦ Consolidated Synthesis (Wiki) rev. #1 (baseline) · CC BY-SA 4.0
Start synthesis for this question →

No community synthesis consolidated for this question yet. Below is the verifiable official starting point — if you passed the exam, open the option dissection.

Official Answer (B): Set cloudFiles.maxFilesPerTrigger to a fixed cap, so each micro-batch ingests at most that many files and the backlog drains gradually

Ranvier · DBDEA

Contribute to this Question

Your contribution builds your Orbital Sheath

Peer contributions

No public contribution yet. Be the first who passed to explain why the distractors trick candidates — your contribution builds your Orbital Sheath.