A newly hired data engineer asks how work is divided when a PySpark notebook runs on a multi-node cluster: where the notebook's code and the Spark session live, and where the actual parallel processing of data happens. Which answer is correct?

1. A. All of the work, including data processing, runs on the driver node; the executors are passive and only store the underlying data files 💬 A
2. B. Each executor runs its own full copy of the notebook's code and independently processes its own portion of the data 💬 B
3. C. Driver nodes exist only on high-concurrency shared clusters; single-user clusters and dedicated job clusters run without any driver at all 💬 C
4. D. The driver node hosts the Spark session and coordinates tasks, while executors run the tasks that process data in parallel ✓ D

Revelar gabarito

Resposta correta: The driver node hosts the Spark session and coordinates tasks, while executors run the tasks that process data in parallel

Referência: https://docs.databricks.com/aws/en/compute