A nightly job pages through a partner events API that occasionally times out. The job's retry logic can re-request a page, so the same event_id sometimes arrives twice in one night; the API cannot promise exactly-once delivery. Each event must appear exactly once in the target Unity Catalog table. Which design makes this ingestion reliable?

1. A. Pause the pipeline until the partner can guarantee exactly-once delivery for every event_id, before writing anything into the table 💬 A
2. B. Append the raw responses with plain INSERT INTO and require analysts to filter duplicate event_ids out of every downstream dashboard and report 💬 B
3. C. Accept at-least-once delivery and write idempotently: MERGE INTO the target on event_id, updating matches and inserting rows that are new ✓ C
4. D. Disable the job's retry logic so a timed-out call is never repeated, accepting that lost responses mean permanently missing events 💬 D

Reveal answer

Correct answer: Accept at-least-once delivery and write idempotently: MERGE INTO the target on event_id, updating matches and inserting rows that are new

Reference: https://docs.databricks.com/aws/en/delta/merge