Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

[32B Dense] Comparison v1 vs. v2 run

Aperta
  • Qualcosa non va?
#412 7 commenti 0 reazioni 1 assegnatario Vedi su GitHub

@tvosch ci sta già lavorando.

Dal 22/9/2026.

Valutazione

Questa issue non è ancora stata valutata.

Descrizione

T4.4 - model training

Goal

Pick the run to continue for the rest of the 32B dense run: v1 or v2. The deciding point for now is at 86k iterations or 1.4T tokens of the total 10-15T.

v1 run
  • FP8
  • lr: 3e-4
  • z-loss 1e-4
  • defer_embedding_wgrad_compute: true
  • At 64k loss diverged: z-loss was lowered to 1e-5 QK-layernorm: weight decay, RMS-norm weight decay, Fused CE and z-loss gradients in FP32. This run is called B1
  • Also a 86k fork with defer_embedding_wgrad_compute: false only
  • wandb: https://wandb.ai/openeurollm-project/oellm_32b_dense
v2 run

Approach

Since comparing the two runs on either loss, validation loss or downstream evaluation with different learning rates, we expect the lower learning rate to be favoured within the constant learning rate phase -> so lower loss, higher eval score. The higher learning rate (under the condition that the run is stable) is to outperform on downstream evaluation only after proper annealing.

Annealing can be expensive so we approximate annealing with WSM. @geoalgo already reproduced this on prelude here

For simplicity we do annealing on 5 checkpoints uniformly (so averaging with equal weights) across checkpoints of 2000 iterations apart (78k–86k; 230k–238k).

Downstream evaluation is done with oellm-eval

For uniform checkpoint merging we take 5 checkpoints and average them with equal weight

branch checkpoints merged source
v2 + z-loss at 86k 78000, 80000, 82000, 84000, 86000 /e/scratch/e-sta-openeurollm/production_training/oellm_32b_v2_zloss_unfused_i60k_gbs4096_lr1.76e-4/checkpoints
defer-off fork at 86k 78000, 80000, 82000, 84000, 86000 /e/scratch/e-sta-openeurollm/production_training/oellm_32b_dense_prod_dataopt5_deferoff_3e-4_i64k_seed1234/checkpoints
B1 at 86k 78000, 80000 /e/fscratch/e-sta-openeurollm/jj1/reanimation_64k_20260911/B1-reborn/training_ckpts/checkpoints
82000, 84000, 86000 /e/fscratch/e-sta-openeurollm/jj1/reanimation_64k_20260911/B1-reborn-production/training_ckpts/checkpoints
B1 at 238k 230000, 232000, 234000, 236000, 238000 same B1-reborn-production tree

B1 at 86k spans two directories because -production starts at 82000; the two trees share a
trajectory (their iter_0082000 are bit-identical, checked before merging).

Outputs: /e/scratch/e-sta-openeurollm/vanosch1/wsm_test/merged_32b_*; HF exports
downstream_eval_32b/hf/{v2zloss,fork,b1}_merge5_* and b1_238k.

Lingua principale
Nessun dato sulla lingua
Stelle
3
Fork
0
Nessuna PR unita negli ultimi 30g

Preparare l'ambiente

Questo progetto non fornisce container di sviluppo, Dockerfile né guida per i contributori, quindi l'ambiente è a tuo carico: parti dal suo README e consulta la nostra guida al primo contributo per i passaggi generali.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di OpenEuroLLM/Taskboard

Tutte le issue di OpenEuroLLM/Taskboard

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.