DPO checkpoint merging

What was done

We trained several models with DPO and saved a checkpoint every 200 steps to analyse: is it better to pick one of those checkpoints, or to average several of them together?

We tried 282 averages (linear merges) across four models: Qwen3 4B, Qwen3 8B, Qwen3 14B, and Llama 3.1 8B. Each average was compared with the best single checkpoint from the same training runs. The runs themselves vary learning rate and β (how strongly DPO trusts the preference data).

The averages fall into a few families:

• Top-k soups. Take the k checkpoints with the highest ELO and average them with equal weight. We did this both across the whole sweep and within separate runs.
• Trajectory soups. Average checkpoints from a single training run:
  • the last 3 or 5 saves;
  • every save in the run;
  • the run's best steps;
  • the saves around the step where ELO peaked;
  • an EMA (exponential moving average): a weighted average that counts later checkpoints more. Going backward from the end of training, each older save is weighted 0.7 times the one after it.
• Temporal interpolation. A weighted mix of two moments in one run: the step that did best on preference, and the step that did best on static capability.
• Controls. Two checks that the result is really about averaging, and not just about how far the weights moved:
  • take the SFT model and walk partway toward a chosen checkpoint (θ_SFT + λ·Δ);
  • soups whose update size is matched, so we can tell the direction of the update apart from its length.

Every merge was scored on 10 held-out benchmarks:

• Preference (chat-quality): Arena-Hard, MT-Bench, AlpacaEval. Judged by Qwen3-Next-80B.
• Static capability (knowledge and reasoning): ARC-C, GPQA, GSM8K, HellaSwag, PIQA, TruthfulQA, IFEval.

ELO was used to choose which checkpoints to merge.

Why we did it

• Does averaging checkpoints beat picking a single one, and if so, when?
• Can we tell in advance when merging will help, from how the weight updates look (how long they are, and whether they point the same way)?
• Preference seems to peak early, and capability later. If we average those two peaks, do we get a model that is good at both?
• DPO often hurts capability (alignment-tax). Can merging bring some of that capability back?

How we measured it

Find the checkpoints -> plan the merges -> merge them with a streaming engine (mergekit) -> evaluate them

Scores. For each model, every benchmark is turned into a z-score across that model's DPO checkpoints. A score of 0 is an average checkpoint; a score of 1 is one standard deviation better. We report three summaries:

• z_static: average of the static benchmarks;
• z_dynamic: average of the preference benchmarks;
• z_all: average of all ten.

What we compare against:

• Best final checkpoint by ELO. Among the last checkpoint of each run, pick the one with the highest ELO.
• Honest selection. Split the held-out benchmarks into two halves. Use one half to pick the best final checkpoint, then score that checkpoint on the other half. Average this over every way of splitting the benchmarks. This asks whether a checkpoint that looks good on some tests still looks good on others.
• Matched selection. The same split, applied to merges and to checkpoints side by side. On the selection half, pick the best merge and the best checkpoint. Score both on the evaluation half.