A nightly job joins a 900 GB transactions DataFrame to a 250 MB currency_rates dimension on currency_code and then aggregates by market. The Spark UI shows a SortMergeJoin whose shuffle of the transactions data dominates the runtime. The rates table fits comfortably in every executor's memory. ```python joined = txns.join(rates, "currency_code") ``` Which change most directly avoids shuffling the large DataFrame for this join?
Reveal answer
Correct answer: Force a broadcast hash join by wrapping the small side: txns.join(broadcast(rates), 'currency_code')
Reference: https://docs.databricks.com/aws/en/pyspark/reference/functions
Noos Visual & Functional Recognition
The “Sphere Embrace”: how Noos embraces top contributors without a childish mascot.
In neurobiology, glial cells wrap axons in myelin sheaths to accelerate neural impulses. On Noos, as your explanations earn accelerations and enter the Wiki Synthesis, the Noos Sphere wraps your Identicon with Keplerian orbits and two quantum nodes on its shoulders.
Pure 5×5 geometric Identicon (no personal photo, zero tracking). 1px frame.
Single dashed orbit with 1 bright polar node at the apex. Highlights those who passed the exam.
Two coaxial ellipses (±35°) wrap the avatar with two lateral nodes.
The Sphere Embrace: The Noos Sphere hood wraps the top of the Identicon and rests two luminous nodes on its shoulders.
02 — The Myelin Layer · Canonical Discussion
Where the exam tries to trick you — and how those who passed solved it.
contributed to this question
No community synthesis consolidated for this question yet. Below is the verifiable official starting point — if you passed the exam, open the option dissection.
Official Answer (D): Force a broadcast hash join by wrapping the small side: txns.join(broadcast(rates), 'currency_code')
Contribute to this Question
✦ I PASSED!Peer contributions
No public contribution yet. Be the first who passed to explain why the distractors trick candidates — your contribution builds your Orbital Sheath.