The full answer
The job now runs with the service principal's privileges, not yours, and the service principal lacks some of what you had. Find exactly which object and privilege the error names, grant that to a group the service principal belongs to, and repeat until the job runs. Do not fix it by running production as a person or by granting broad admin rights; the whole point of the service principal is a small, known set of permissions.
Why it worked for you
A Lakeflow job uses the privileges of its Run as identity for every task. By default that is the creator, so while you developed it, the job quietly used everything you can reach: tables you were granted years ago, catalogs your team owns, the main catalog that everyone can use. Switching Run as to a service principal reveals all of those hidden dependencies at once.
Find what is missing
Read the error: it names the privilege and the object, for example USE SCHEMA on sales.raw or SELECT on sales.raw.orders. A frequent surprise is that SELECT on a table is not enough. The identity also needs USE CATALOG on the catalog and USE SCHEMA on the schema.
Typical set for a pipeline that reads raw tables and writes curated ones:
| Object | Privilege |
|---|---|
| Source catalog and schema | USE CATALOG, USE SCHEMA |
| Source tables | SELECT |
| Target schema | USE SCHEMA, CREATE TABLE |
| Target tables it updates | MODIFY, SELECT |
| Landing files in an external location | READ FILES |
| Secrets, connections, functions it calls | the matching use or execute privilege |
Grant it the right way
- Grant to a group, such as
etl-sales-prod, and add the service principal to it. Grants then survive replacing the principal and are easy to review. - Grant at the narrowest level that is still practical. Schema-level
SELECTis inherited by every table in it, which fits a schema the job owns entirely; table-level grants fit shared schemas. - Write grants as code (SQL scripts or Terraform) next to the job, so a new environment gets the same permissions and changes are reviewed.
Avoid it next time
Develop and test with the same identity, or run the job as the service principal in a test workspace before release. Then the first permission error appears during testing, not at 2 a.m. on the first production run.
What interviewers want: that Run as decides the privileges, why "works for me" hides dependencies, the catalog and schema USE privileges on top of SELECT, group-based least-privilege grants kept as code, and testing with the production identity.
Last updated 16 October 2026
Sources