In an exploratory notebook, an engineer wants to inspect catalog.events.bronze_events, a table with roughly 200 million rows. She runs the cell below, and a few minutes later the run fails with an out-of-memory error raised on the driver process. ```python rows = spark.read.table("catalog.events.bronze_events").collect() ``` Why did this fail, and what is the correct practice?

1. A. collect() fans the rows out across the executors for inspection — the driver out-of-memory error must have a different cause, such as network buffers 💬 A
2. B. Prepend coalesce(1) — collapsing the DataFrame into a single partition is what allows collect() to return without exhausting driver memory 💬 B
3. C. collect() ships every row into the driver's own memory — inspect with display(), take(n), or limit(), and keep full-scale processing on the cluster ✓ C
4. D. The driver never holds data in Spark — only executors allocate memory, so the failure proves the executors were undersized for the table 💬 D

Reveal answer

Correct answer: collect() ships every row into the driver's own memory — inspect with display(), take(n), or limit(), and keep full-scale processing on the cluster

Reference: https://docs.databricks.com/aws/en/pyspark/reference