A model trained last month gives different results when you retrain it with the same code and data on Azure ML. The environment definition has not changed. What is going on, and how do you prevent it?

Most likely the packages did. If the environment lists packages without exact versions (scikit-learn, or pandas>=2.0), every new image build installs whatever is newest that day, so "the same environment" can mean different libraries a month apart. Pin every package to an exact version, build the environment once, register it as a version, and point training and deployment at that version.

How Azure ML builds environments

An environment becomes a Docker image. Azure ML computes a hash from the base image, any custom Docker steps and the package list, and reuses a cached image when the hash matches. Changing the name or version label does not change the hash; changing a package, its version, or even the order of dependencies does.

The trap: a definition with unpinned packages keeps the same hash while the packages it would install change. The cached image hides the drift until something forces a rebuild, such as a new workspace, a cleaned registry or a small edit. Then the new image picks up newer libraries.

Other causes to rule out

Compare the two runs in the workspace: each job records its environment and inputs, so a diff of package versions and data versions usually finds the cause in minutes.

How to make runs repeatable

• Pin exact versions in conda.yaml or requirements.txt, including the libraries that matter indirectly (numpy, scipy).
• Pin the base image by a specific tag or digest rather than latest.
• Register the environment with an explicit version and reference it as azureml:churn-env:7 in jobs and deployments. Use the same environment for training and inference, or a version built from the same lock file.
• Version the data: use data asset versions, not raw paths.
• Change deliberately: upgrading a library means a new environment version, a retrain and a comparison against the old model.

Curated or your own?

Curated environments are fine for exploring. For anything that goes to production, build your own pinned environment; editing a curated one creates a custom environment anyway, so own it from the start.

What interviewers want: unpinned packages as the usual cause, how image caching by hash hides the drift, comparing the two runs' recorded environments and data, and pinned, versioned environments shared by training and inference.