Learning Rate

What the learning rate actually controls

The learning rate sets the size of each weight update during training. It is one of the few hyperparameters that directly shapes optimisation behaviour, because the same model, data and loss function can train very differently depending on whether updates are small and cautious or large and aggressive.

In practical terms, the learning rate is a trade-off between speed and stability. A rate that is too high can make training bounce around the loss surface, miss a good solution, or even diverge; a rate that is too low can leave the model improving so slowly that it appears stuck.

Why it matters in model training

The learning rate affects whether optimisation converges smoothly, oscillates, or becomes numerically unstable. That makes it more than a tuning detail, because it influences training time, final accuracy, and the likelihood that a model reaches a usable minimum at all.

It also interacts with other training choices such as batch size, optimiser type, regularisation, and learning-rate schedules. A setting that works for one architecture or dataset may be poor for another, so the parameter is usually chosen by experimentation rather than by rule of thumb alone.

Common learning-rate behaviours and schedule patterns

In many workflows, the initial learning rate is only part of the story. Practitioners often reduce it over time so that early training makes rapid progress while later training refines the weights more carefully. This is why schedules such as step decay, cosine decay, and warmup are widely used.

Adaptive optimisers change the effective step size for different parameters, but they do not eliminate the need to think about the base learning rate. Even with Adam-like methods, the learning rate still controls how aggressively the optimiser moves through parameter space.

How practitioners tune it in practice

A good learning-rate choice is usually the one that reaches stable improvement fastest without causing erratic loss curves. That is why teams often run short experiments across a range of values, watch the loss trajectory early in training, and adjust when the model shows divergence, stagnation, or excessive noise.

Common misunderstanding: a smaller learning rate is not always safer. If it is too small, the model may train so slowly that it consumes more compute, remains underfit, or fails to escape shallow regions of the loss landscape.

Related resources from NHI Mgmt Group

• What should teams do when rate limits are exceeded?
• What should teams do first after learning that a kernel SMB service is exposed?
• How do teams govern AI systems that keep learning after deployment?
• What do regulators expect from AI and machine learning risk models?