Distributed training with Horovod on Databricks: does it automatically parallelize any ML model, and what does Databricks recommend now?

No. Horovod only parallelizes deep learning training code that you change to use its API, and it is no longer the recommended route on Databricks: Horovod and HorovodRunner are deprecated and are not installed in Databricks Runtime 16.0 ML and later. Databricks now points to TorchDistributor for PyTorch, the DeepSpeed distributor for very large models, tf.distribute.Strategy for TensorFlow, and Ray.

What Horovod did, and what any of these tools still require

They all use data parallelism: each worker holds a copy of the model, trains on its own share of the data, and the workers average their gradients (an all-reduce) after each step so the copies stay identical. That only works for models trained by gradient descent in a training loop, so:

• Your code must change: initialise the distributed backend, wrap the model (for example PyTorch's DistributedDataParallel), give each worker its own share of the data, and save checkpoints from one worker only.
• Traditional models such as scikit-learn's random forests are not parallelised this way.
• Arbitrary Python code is not distributed at all.

The current way with PyTorch

from pyspark.ml.torch.distributor import TorchDistributor

def train(epochs):
    import torch.distributed as dist
    from torch.nn.parallel import DistributedDataParallel as DDP
    dist.init_process_group("nccl")
    model = DDP(build_model().cuda())
    # ... training loop with a DistributedSampler ...
    dist.destroy_process_group()

TorchDistributor(num_processes=8, local_mode=False, use_gpu=True).run(train, 5)

TorchDistributor starts the training function on the cluster's workers as a Spark job and sets up the environment that PyTorch's own distributed tools expect.

Other options

• DeepSpeed distributor for models too large for one GPU's memory.
• Ray on Databricks for scaling training and tuning across frameworks.
• Spark ML (pyspark.ml.connect) for classic models on Spark DataFrames.

Before distributing at all, Databricks advises training on a single machine whenever the model and data fit, ideally one with several GPUs. Distribution adds communication overhead and complexity, and it pays off only when one machine is too slow or too small.

How to answer in an interview: say what data parallelism is, that the training code must be adapted, that it applies to gradient-trained models only, and that Horovod has given way to TorchDistributor and DeepSpeed on Databricks. Then add the judgement: measure single-machine training first, scale to several GPUs on one machine next, and use several machines only when the numbers show it is needed. That order is what keeps cost and debugging effort down.