Gradient Descent

How Gradient Descent Works

Gradient descent is an optimisation procedure that iteratively moves model parameters in the direction that reduces loss. The core idea is simple, but the practical behaviour depends on the loss surface, step size, and whether updates use one example, a mini-batch, or the full dataset.

Because it is iterative, gradient descent is not a single calculation but a search process. Each update uses the gradient as a local signal, so the method can steadily improve a model even when the full parameter space is too large to solve analytically. That same locality is also why the method can be sensitive to scale, curvature, and initialization.

Why Gradient Descent Matters in Machine Learning

Most modern supervised learning models rely on gradient-based optimisation because it is efficient, differentiable, and works with large parameter sets. It is the mechanism that turns a loss function into a repeatable training process, letting models learn from data rather than from hand-tuned rules.

In practice, the choice between batch, stochastic, and mini-batch variants changes the trade-off between stability and speed. Full-batch updates are often smoother but slower, while stochastic updates can be noisy but useful for large datasets and faster iteration. Mini-batches are common because they balance throughput and convergence behaviour.

Gradient descent also defines the basic training dynamics of neural networks and other differentiable models. When a model appears to “learn,” the underlying process is usually some form of gradient-based optimisation, often paired with momentum, adaptive learning rates, or regularisation to improve convergence and generalisation.

Common Failure Modes and Tuning Challenges

Although the method is conceptually straightforward, poor tuning can make training unstable or inefficient. A learning rate that is too high can overshoot minima or diverge, while a rate that is too low can make training painfully slow or leave the model stuck on plateaus.

Another issue is the geometry of the loss surface. Sharp curvature, saddle points, ill-conditioned features, and poorly scaled inputs can all slow progress or create oscillation. In deep learning, these effects are amplified because the optimisation landscape is high-dimensional and often irregular.

Gradient descent can also converge to different solutions depending on initialization and random sampling. That is not necessarily a defect, but it means training outcomes may vary unless the process is controlled carefully and evaluated with proper validation metrics.

How Practitioners Use Gradient Descent Well

Use gradient descent as part of a broader training discipline, not as a blind default. The update rule only works well when features are scaled sensibly, the objective is differentiable enough for stable gradients, and the optimisation settings match the model and dataset size.

Practitioners usually monitor training loss, validation loss, and convergence behaviour together, because a loss that keeps falling is not enough if generalisation is worsening. If optimisation stalls, the remedy is often to revisit learning rate, batch size, initialization, normalisation, or the choice of optimiser rather than to assume the model itself is flawed.

Practitioner takeaway: Gradient descent is less about one formula than about controlling a training process. Small configuration choices often determine whether optimisation is merely slow, or genuinely effective.

Related resources from NHI Mgmt Group

• How does differentially private stochastic gradient descent work?
• Why does Differentially private Stochastic Gradient Descent matter for security teams?
• Differentially private Stochastic Gradient Descent
• Stochastic Gradient Descent (SGD)