When should organisations use supervised fine-tuning before preference-based reinforcement learning for language models?

Use supervised fine-tuning when the model needs a stronger behavioural shift or when the current outputs are too weak for ranking data to be meaningful. Demonstrations help bootstrap the task with high-quality examples, while preference comparisons work better once the model is already producing plausible candidates. The choice depends on whether you need a broad reset or finer control over behaviour.

Why the training sequence matters

Supervised fine-tuning and preference-based reinforcement learning solve different parts of model shaping. SFT is the better starting point when the model needs to learn the task, style, or policy from clear demonstrations, because it changes the baseline behaviour before ranking signal is useful. Preference learning is most effective after the model can already produce plausible candidates that humans or evaluators can reliably compare.

The practical question is not which method is “better” in general, but which one gives you a usable gradient at the current stage. If outputs are still noisy, off-task, or too weak to rank consistently, preference data tends to be low signal. If the model is already roughly competent, preference optimisation can sharpen judgement, reduce undesirable responses, and improve consistency without needing as many demonstrations.

When supervised fine-tuning should come first

Use supervised fine-tuning first when you need a broad behavioural reset, not just a refinement. That is common when the base model has not yet learned the target format, terminology, safety boundaries, or domain-specific response pattern. Demonstrations give you a direct teaching signal that can move the model quickly toward the desired operating range.

SFT is also the sensible choice when the current model outputs are so poor that preference labels would mostly compare bad against bad. In that situation, the preference signal can become unstable or misleading because the model is not yet generating candidates with enough quality separation. A small, well-curated supervised set can raise the floor and make later preference optimisation more informative.

In practice, SFT is often the bridge between a general model and a model that is ready for human preference shaping. It is especially useful when the behaviour you want is easier to demonstrate than to rank, such as strict output structure, tone constraints, domain procedure, or a specific decision policy. Preference learning then works as a second-stage adjustment, not a substitute for teaching the core behaviour.

When preference-based reinforcement learning adds more value

Preference-based reinforcement learning becomes more valuable once the model can already generate multiple plausible answers and the remaining problem is choosing among them. At that point, ranking data can express subtle preferences such as helpfulness over verbosity, safer refusals over risky completions, or better trade-offs between completeness and concision. The method is strongest when the model is near the target distribution, but still inconsistent in judgment.

This is why preference learning often works better as a refinement layer. It can smooth out edge cases that are awkward to encode as demonstrations, especially when the desired behaviour is comparative rather than absolute. If the target is “better than this alternative” instead of “exactly match this example,” preference data usually gives a more natural training signal.

The sequencing choice also depends on data quality. High-quality demonstrations are easier to trust when you need to establish a canonical answer shape. High-quality comparisons are easier to trust when the candidate outputs are already close enough that human raters can express a meaningful preference. If either data type is weak, the method you choose can amplify noise rather than improve behaviour.

How practitioners should decide the order

For many language model workflows, the safest order is SFT first, then preference-based reinforcement learning. That sequence is most robust when you want a reliable base policy before you optimise nuance. It also makes evaluation easier, because you can measure whether the model learned the intended behaviour before introducing preference optimisation.

A useful decision rule is simple: if the model cannot yet produce acceptable candidates, start with demonstrations; if it can already produce acceptable candidates but still needs better selection, move to preferences. If you need both, use SFT to establish competence and preference learning to refine quality. If compute or labelled data are constrained, prioritise the stage that fixes the largest current failure mode.

Practitioners should also consider whether the target behaviour is stable or shifting. Stable tasks with well-defined outputs often benefit more from SFT earlier in the pipeline. Open-ended assistant behaviour, policy tuning, and subjective quality improvements usually benefit more from preference learning after a competent baseline exists. The right sequence is the one that gives you the cleanest training signal at each stage.

Practitioner Guidance

What to verify: Before choosing the order, inspect a sample of model outputs and ask whether raters can make consistent comparisons. If they cannot, preference training is premature and SFT should establish the baseline first.

Decision rule: If your main failure is that the model does not yet know what “good” looks like, use supervised fine-tuning first. If the model already knows the task but still makes uneven choices, preference-based reinforcement learning is the better second stage.

Practitioner takeaway: The most reliable sequence is usually teach first, refine second, because preference optimisation is most useful after the model can already produce candidates worth ranking.