The source system resent yesterday's events file, and the bronze table now contains rows that are exactly identical across every column — a full-row duplicate of each event. The silver table must contain exactly one row per event, and for this feed a duplicate means an exact copy of the whole row. Which PySpark operation removes these duplicated rows?

1. A. dropDuplicates() — it removes rows that are identical across all columns; dropDuplicates(['event_id']) is the variant that keeps one row per key ✓ A
2. B. dropDuplicates() cannot run without a column list — Spark requires the deduplication key to be specified explicitly in every invocation 💬 B
3. C. distinct() — it inspects the schema and picks an arbitrary business key, such as event_id, to deduplicate the rows on its own 💬 C
4. D. A row_number() window ranked over all columns — exact duplicates can only be removed by ranking the identical rows and filtering down to rank one 💬 D

Revelar gabarito

Resposta correta: dropDuplicates() — it removes rows that are identical across all columns; dropDuplicates(['event_id']) is the variant that keeps one row per key

Referência: https://docs.databricks.com/aws/en/sql/language-manual/sql-ref-functions