A Lakeflow Job task calls a partner REST API that uses OAuth 2.0 client-credentials authentication: the task presents a client ID and a long-lived client secret to obtain a short-lived access token at runtime. The security team requires that the secret be stored in one managed place, referenced from the task without the value appearing in run history or logs. Where should the client secret be stored?

1. A. As a string parameter on the Lakeflow Job, so the task reads the secret from its parameter list at the start of every run 💬 A
2. B. In a Databricks secret scope, fetched at runtime with dbutils.secrets.get, which redacts the value in output and logs ✓ B
3. C. In an environment variable set in the cluster's Spark configuration, where it is readable by anyone who can attach to that cluster 💬 C
4. D. In the Git folder repository that holds the pipeline code, with branch permissions restricted to the data engineering team 💬 D

Reveal answer

Correct answer: In a Databricks secret scope, fetched at runtime with dbutils.secrets.get, which redacts the value in output and logs

Reference: https://docs.databricks.com/aws/en/security/secrets