Fine-Tuning, RLHF and Safety: How a Model Becomes an Assistant

After pretraining, a model is like a student who has read the whole library but never had a conversation. Ask it “What’s a good name for a cat?” and it might reply “What’s a good name for a dog? What’s a good name for a fish?”, because in its training data lists of questions are common. Post-training fixes this. It is a much smaller amount of data and compute than pretraining, but it decides how the model *behaves*.

Step 1: Supervised fine-tuning (SFT)#

The model is trained on examples of prompt → ideal response, using the same next-token learning as pretraining, just on far fewer, far better examples (thousands to millions instead of trillions of tokens).

sft_examples = [
    {"prompt": "Explain recursion to a 10-year-old.",
     "response": "Recursion is when something is defined using a smaller copy of itself..."},
    {"prompt": "Write a polite leave request to my manager.",
     "response": "Dear Ms. Rao, I would like to request leave on..."},
]
for ex in sft_examples:
    print(len(ex["prompt"].split()), "->", len(ex["response"].split()), "words")

Output:

5 -> 12 words
8 -> 10 words

After SFT, the model answers instead of continuing. But good answers can differ in many subtle ways (tone, accuracy, safety), and those are hard to capture with examples alone.

Step 2–4: Learning from preferences (RLHF)#

It’s easier for a person to say “A is better than B” than to write the perfect answer. RLHF (Reinforcement Learning from Human Feedback) uses exactly that:

1. Collect pairs of answers to the same prompt, with a human choice of which is better.
2. Train a reward model that gives any answer a score predicting human preference.
3. Use reinforcement learning to update the assistant so its answers get higher reward, while staying close to its SFT version (so it doesn’t learn weird tricks to game the score).

Here is a toy reward model: a scoring function built from what reviewers liked.

def reward(answer):
    score = 0
    if "step" in answer.lower():
        score += 2          # reviewers liked step-by-step answers
    if "i'm not sure" in answer.lower():
        score += 1          # and honesty when unsure
    if "guaranteed" in answer.lower():
        score -= 3          # and disliked overconfident promises
    score -= max(0, len(answer.split()) - 40) * 0.1   # and rambling
    return round(score, 1)

a = "Step 1: boil water. Step 2: add tea leaves and spices. Step 3: add milk and sugar."
b = "This guaranteed perfect chai recipe will change your life forever!"
print("A:", reward(a), "| B:", reward(b))

Output:

A: 2.0 | B: -3.0

A real reward model is itself a neural network trained on hundreds of thousands of comparisons, not hand-written rules, but it plays the same role: turning “what people prefer” into a number the training can push towards.

Watch a preference pair become a training signal#

import math
score_chosen = 2.0
score_rejected = -3.0
gap = score_chosen - score_rejected
p_correct = 1 / (1 + math.exp(-gap))
loss = -math.log(p_correct)
print(round(p_correct, 3), round(loss, 3))

Other alignment methods you’ll hear about#

Safety is a balance#

Too little alignment and a model may help with harmful requests or invent facts confidently. Too much and it refuses harmless questions (“over-refusal”). Labs measure both and tune between them, which is why model updates sometimes say “fewer unnecessary refusals” or “more careful with medical advice”.

Summary#

• SFT teaches the assistant format from good examples.
• Reward models turn human preferences into scores; RLHF pushes the model towards higher scores.
• Constitutional AI uses written principles and AI feedback; DPO learns from preference pairs directly.
• Post-training is small compared with pretraining but decides behaviour, tone and safety.

Next: how a finished model is served to millions of people, and why output tokens cost more than input tokens.