bronze_orders accumulates re-delivered files: the same order_id appears multiple times with different ingested_at values, and silver must keep exactly the most recent row per order. A teammate proposes: ```python from pyspark.sql.functions import col latest = (bronze .orderBy(col("ingested_at").desc()) .dropDuplicates(["order_id"])) ``` What is the correct assessment of this proposal?

1. A. The proposal is correct — orderBy(col('ingested_at').desc()) guarantees that dropDuplicates(['order_id']) keeps the newest row per order, because the sort carries over 💬 A
2. B. dropDuplicates(['order_id']) keeps an arbitrary row, ignoring the sort — use row_number() over a window ordered by ingested_at DESC, keeping rn = 1 ✓ B
3. C. dropDuplicates(['order_id']) is missing columns — listing every column of the table would make it keep only the latest row 💬 C
4. D. Replace dropDuplicates(['order_id']) with distinct(), which keeps the last row encountered after an orderBy 💬 D

Reveal answer

Correct answer: dropDuplicates(['order_id']) keeps an arbitrary row, ignoring the sort — use row_number() over a window ordered by ingested_at DESC, keeping rn = 1

Reference: https://docs.databricks.com/aws/en/pyspark/reference/functions