A partner REST API returns orders in pages, and every response includes a next_page cursor. A nightly Lakeflow Job must fetch only orders that are new since the previous run, because reprocessing old pages wastes API quota. The job restarts on failures and its compute terminates after each run. How should the ingestion position carry over between runs?

1. A. Replay from the first page every night and let a MERGE against the target re-deduplicate the whole order history again on every run 💬 A
2. B. Persist the newest cursor in a small Delta control table, read it at start, and pass it to the next run as a job parameter ✓ B
3. C. Write the cursor to a file on the job cluster's driver disk, which survives across runs even though the cluster itself terminates 💬 C
4. D. Store nothing: the partner API remembers each client's last position on the server side, so the job can simply ask for whatever orders are new 💬 D

Reveal answer

Correct answer: Persist the newest cursor in a small Delta control table, read it at start, and pass it to the next run as a job parameter

Reference: https://docs.databricks.com/aws/en/jobs/configure-job