A data engineer builds an intermediate DataFrame in her notebook: a read of a large Parquet dataset from cloud storage, a join with a 5 GB dimension, a filter to active regions, and a deduplication pass. Five cells later in the same notebook reuse that DataFrame, and each one currently re-runs the whole lineage, taking about six minutes per cell. She wants the reuse to be fast and the memory released when the notebook ends. Which approach fits?

1. A. Call cache() before the first action runs, so that even the initial computation that materializes the DataFrame becomes faster 💬 A
2. B. Call cache() in the very first cell and rely on the cached data surviving cluster termination, so that tomorrow's job run reuses it 💬 B
3. C. Do nothing, because Spark automatically caches every DataFrame in memory after its first action and reuses it for free 💬 C
4. D. Call cache() on the DataFrame, materialize it with an action once, and call unpersist() when the last consuming cell finishes ✓ D

Reveal answer

Correct answer: Call cache() on the DataFrame, materialize it with an action once, and call unpersist() when the last consuming cell finishes

Reference: https://docs.databricks.com/aws/en/pyspark/reference