A freshly pre-trained model is a text continuation engine, not an assistant. Post-training — instruction tuning followed by preference optimisation — is what teaches it to follow instructions, keep a consistent voice and decline requests it should not fulfil.
Stages
- Pre-training. Next-token prediction over a very large corpus; this is where general capability comes from, and it is far too expensive to repeat for most teams.
- Supervised fine-tuning. Curated prompt-and-response pairs teach the format and behaviour of an assistant.
- Preference optimisation. RLHF, DPO and relatives train the model on comparisons between responses, sharpening helpfulness and refusal behaviour.
- Reinforcement on verifiable rewards. For maths, code and other checkable domains, automatic graders provide the signal, which is how current reasoning models are trained.
- Distillation. A small model is trained on a large model's outputs, delivering most of the quality at a fraction of the cost.
Cheaper ways to adapt
- Prompting and few-shot examples — always try this first; it costs nothing to change and nothing to maintain.
- LoRA and QLoRA — train small low-rank adapters instead of full weights; hours on one GPU rather than a cluster.
- Preference tuning on your own data — DPO on a few thousand ranked pairs is often enough to fix tone and format drift.
- Structured output constraints — grammars and JSON schemas enforce shape without any training at all.
Practical notes
- Fine-tuning teaches behaviour and format; it is a poor way to add facts — use retrieval for knowledge.
- Build the evaluation set before you train, or you will not be able to tell whether the tune helped.
- Watch for catastrophic forgetting and for a narrow tune that degrades general reasoning.