A PySpark job joins a 4 TB events table with a small dimension table on `user_id`. One stage has 200 tasks: 199 finish in under two minutes, but one task runs for 38 minutes. The stage-level summary metrics show: ```text Shuffle read size per task: min 14 MB | median 32 MB | max 7.9 GB Task duration: min 21 s | median 48 s | max 38 min ``` What is the most likely diagnosis?

1. A. Too few shuffle partitions: all tasks are uniformly too large, so spark.sql.shuffle.partitions should be increased to shrink every task 💬 A
2. B. Disk spill: the stage is dominated by spilling shuffle data to local disk, which the duration column reveals 💬 B
3. C. A broadcast timeout: the small table exceeded the broadcast threshold and the join fell back to a shuffle 💬 C
4. D. Data skew on user_id: one partition received 7.9 GB of shuffle data versus a 32 MB median, producing a single long-running straggler task ✓ D

Revelar gabarito

Resposta correta: Data skew on user_id: one partition received 7.9 GB of shuffle data versus a 32 MB median, producing a single long-running straggler task

Referência: https://docs.databricks.com/aws/en/optimizations/spark-ui-guide/long-spark-stage