A SageMaker hyperparameter tuning job ran 100 training jobs, 20 at a time, with Bayesian search over wide ranges. It cost a lot and the best model barely beat the defaults. How do you run the next round better?

Fix 3 things: the ranges (most of the budget went on values that could never win), the parallelism (Bayesian search learns from finished jobs, and 20 at once leaves it guessing), and wasted epochs (stop jobs that are clearly behind). Then continue from the first run with warm start instead of starting over, so the 100 jobs already paid for still count.

1. Read the first run before changing anything

Plot the objective metric against each hyperparameter across the 100 jobs. Typical findings:

• The best jobs cluster at one end of a range: the range is wrong, widen it that way.
• A parameter makes no visible difference: fix it to a sensible value, and spend the search on the ones that matter.
• The best score is close to the defaults whatever you try: the limit may be the features or the data, not the hyperparameters. AWS itself warns that tuning might not improve a model.

Also check the objective is a validation metric. Tuning on a training metric rewards overfitting.

2. Pick the strategy for the search

With Bayesian search, each new job is chosen from completed results. Running 20 at once means many jobs are picked before the earlier ones report, so the search behaves closer to random. Run only a few jobs at a time, or switch to random search if wall-clock time matters more than efficiency. For a neural network that reports accuracy every epoch, Hyperband is often the cheapest: it reallocates epochs to the promising configurations.

3. Stop losing jobs early

Set TrainingJobEarlyStoppingType to AUTO. After each epoch, SageMaker stops a job whose objective is worse than the median of earlier jobs' running averages at the same epoch. It only works if the training code emits the objective metric every epoch; with PyTorch, log it yourself in the format your metric definition matches. With Hyperband, leave this setting OFF, since Hyperband stops jobs with its own mechanism.

4. Continue with warm start

WarmStartConfig={
    "ParentHyperParameterTuningJobs": [{"HyperParameterTuningJobName": "churn-xgb-round1"}],
    "WarmStartType": "IdenticalDataAndAlgorithm",
}

• IdenticalDataAndAlgorithm: same data and training image; you may change the ranges and the number of jobs. The response's OverallBestTrainingJob is the best across this job and its parents.
• TransferLearning: when the data or the algorithm version has changed, such as a month of new data.

The rules that trip people up: the same objective metric, the same total of static plus tunable hyperparameters with the same types, at most 10 changes from tunable to static or to static values, up to 5 parents, and no chaining (round 3 must list round 1 directly if it should learn from it). So if you might tune a parameter later, include it as static from the start.

What interviewers want: reading the first run before rerunning, knowing why parallelism weakens Bayesian search, matching the strategy to the training type, how early stopping decides, and warm start with its restrictions.