Noos
PT EN
RAN-DBDEA-00045
Ranvier · DBDEA

Exam DBDEA Topics: transformation Difficulty: hard

A silver pipeline must exclude cancelled orders but keep every other row, including orders whose status is still NULL because they are pending manual review. After running: ```python silver = bronze.filter(col("status") != "CANCELLED") ``` QA reports that 612 rows are missing from silver — precisely the rows with NULL status. Why did this happen, and what is the correct fix?

Options (Reading Mode) Click to filter discussion →
  1. A. filter() silently ignores rows with NULL in the filtered column — .where() handles NULL predicates correctly, so switch to .where()
  2. B. The Spark reader dropped the NULL statuses at load time — reload the table with .option('mode', 'PERMISSIVE')
  3. C. Fill the NULLs before filtering: bronze.fillna('UNKNOWN', subset=['status']) makes every row comparable, so the filter keeps them
  4. D. The predicate is NULL for NULL statuses and filter() keeps only true — add an isNull() branch or a null-safe operator
Reveal answer

Correct answer: The predicate is NULL for NULL statuses and filter() keeps only true — add an isNull() branch or a null-safe operator

Reference: https://docs.databricks.com/aws/en/pyspark/reference/functions

Train this exam on Ranvier →

02 — The Myelin Layer · Canonical Discussion

Where the exam tries to trick you — and how those who passed solved it.

0 Passed

contributed to this question

Answer Distractor Matrix Click an option to filter or open its dissection.
✦ Consolidated Synthesis (Wiki) rev. #1 (baseline) · CC BY-SA 4.0
Start synthesis for this question →

No community synthesis consolidated for this question yet. Below is the verifiable official starting point — if you passed the exam, open the option dissection.

Official Answer (D): The predicate is NULL for NULL statuses and filter() keeps only true — add an isNull() branch or a null-safe operator

Ranvier · DBDEA

Contribute to this Question

Your contribution builds your Orbital Sheath

Peer contributions

No public contribution yet. Be the first who passed to explain why the distractors trick candidates — your contribution builds your Orbital Sheath.