Hacktoberfest 2026: los issues que los mantenedores marcaron para octubre, abiertos y aptos para principiantes. Explorar issues de Hacktoberfest

[32B Dense] Comparison v1 vs. v2 run

Abierto
  • ¿Algo no está bien?
#412 7 comentarios 0 reacciones 1 asignado Ver en GitHub

@tvosch ya está trabajando en esto.

Desde el 22/9/2026.

Evaluación

Este issue todavía no se ha evaluado.

Descripción

T4.4 - model training

Goal

Pick the run to continue for the rest of the 32B dense run: v1 or v2. The deciding point for now is at 86k iterations or 1.4T tokens of the total 10-15T.

v1 run
  • FP8
  • lr: 3e-4
  • z-loss 1e-4
  • defer_embedding_wgrad_compute: true
  • At 64k loss diverged: z-loss was lowered to 1e-5 QK-layernorm: weight decay, RMS-norm weight decay, Fused CE and z-loss gradients in FP32. This run is called B1
  • Also a 86k fork with defer_embedding_wgrad_compute: false only
  • wandb: https://wandb.ai/openeurollm-project/oellm_32b_dense
v2 run

Approach

Since comparing the two runs on either loss, validation loss or downstream evaluation with different learning rates, we expect the lower learning rate to be favoured within the constant learning rate phase -> so lower loss, higher eval score. The higher learning rate (under the condition that the run is stable) is to outperform on downstream evaluation only after proper annealing.

Annealing can be expensive so we approximate annealing with WSM. @geoalgo already reproduced this on prelude here

For simplicity we do annealing on 5 checkpoints uniformly (so averaging with equal weights) across checkpoints of 2000 iterations apart (78k–86k; 230k–238k).

Downstream evaluation is done with oellm-eval

For uniform checkpoint merging we take 5 checkpoints and average them with equal weight

branch checkpoints merged source
v2 + z-loss at 86k 78000, 80000, 82000, 84000, 86000 /e/scratch/e-sta-openeurollm/production_training/oellm_32b_v2_zloss_unfused_i60k_gbs4096_lr1.76e-4/checkpoints
defer-off fork at 86k 78000, 80000, 82000, 84000, 86000 /e/scratch/e-sta-openeurollm/production_training/oellm_32b_dense_prod_dataopt5_deferoff_3e-4_i64k_seed1234/checkpoints
B1 at 86k 78000, 80000 /e/fscratch/e-sta-openeurollm/jj1/reanimation_64k_20260911/B1-reborn/training_ckpts/checkpoints
82000, 84000, 86000 /e/fscratch/e-sta-openeurollm/jj1/reanimation_64k_20260911/B1-reborn-production/training_ckpts/checkpoints
B1 at 238k 230000, 232000, 234000, 236000, 238000 same B1-reborn-production tree

B1 at 86k spans two directories because -production starts at 82000; the two trees share a
trajectory (their iter_0082000 are bit-identical, checked before merging).

Outputs: /e/scratch/e-sta-openeurollm/vanosch1/wsm_test/merged_32b_*; HF exports
downstream_eval_32b/hf/{v2zloss,fork,b1}_merge5_* and b1_238k.

Lenguaje dominante
Sin datos de lenguaje
Estrellas
3
Forks
0
Sin PR fusionados en 30 d

Preparar el entorno

Este proyecto no incluye contenedor de desarrollo, Dockerfile ni guía de contribución, así que la configuración corre por tu cuenta: empieza por su README y consulta nuestra guía para la primera contribución para los pasos generales.

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de OpenEuroLLM/Taskboard

Todos los issues de OpenEuroLLM/Taskboard

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.