A team defines a streaming table in a Lakeflow Declarative Pipeline using the read_files table-valued function over a landing directory, and the table's refresh schedule runs every 15 minutes. Files land in that directory around the clock. A new engineer assumes each refresh re-reads the entire directory and worries about duplicate rows. What does each scheduled refresh actually process?

1. A. Only the files that have arrived since the last successful refresh, because the streaming table keeps durable state tracking what it has already ingested ✓ A
2. B. Every file in the directory, with duplicate rows removed afterwards by a deduplication step that the pipeline runs automatically 💬 B
3. C. Nothing, until an external scheduler updates a control table listing which files the previous refresh already consumed 💬 C
4. D. Only new files if the table was declared with AUTO CDC INTO; streaming tables fed by read_files always reload in full 💬 D

Reveal answer

Correct answer: Only the files that have arrived since the last successful refresh, because the streaming table keeps durable state tracking what it has already ingested

Reference: https://docs.databricks.com/aws/en/ldp/concepts/streaming-tables