Synthetic Generation for Multilingual Instruction Tuning

Goal

Generate high-quality, multilingual synthetic instruction-tuning data for the OpenEuroLLM (OELLM) project, covering 35 languages at production scale (100k samples/language), grounded in real-world topic distributions. (we can still do hand-picked categories ?)

Description

Pipeline

All of the generations would be done by synthgen. This would be the pipeline that we could follow for the generation:

          +--------------------------------------------+
          |          build-localized-prompts            |
          +--------------------------------------------+
                               |
                               v
          +--------------------------------------------+
          |           generate (mode=localized)          |
          +--------------------------------------------+
                               |   -> outputs/loc_<gen_model>.jsonl
                               v
          +--------------------------------------------+
          |             generate (mode=judge)            |
          +--------------------------------------------+
                               |   -> outputs/judged_<judge_model>.jsonl
                               v
          +--------------------------------------------+
          |   quality-filter (per-lang score thresholds) |
          +--------------------------------------------+
                               |
                               v
          +--------------------------------------------+
          |  dedupe (exact-hash, optional embedding)     |
          +--------------------------------------------+
                               |
                               v
          +--------------------------------------------+
          |   outputs/loc_full.dedup.jsonl (survivors)   |
          +--------------------------------------------+

Generation Prompt

Example rendering (lang=fr, domain=coding, no persona drawn):

  You are a native speaker and expert writer of French as used in France.
  Create ONE high-quality instruction-tuning example...

  Domain: coding

  Silently decide a realistic user intent that fits this domain...
  No persona: write a direct, standalone request.

  If a more specific, non-obvious general-capability angle fits naturally
  with this domain, narrow to it...

  If a genuine France-specific angle fits naturally with this domain...

  If this domain would more naturally take the shape of editing/rewriting...

  If a more specific, higher-difficulty version of this task fits naturally...

  Output JSON only, no commentary, no code fences:
  {"instruction":"...","response":"...","role_used":"...","intent_used":"..."}

Judge Prompt

Example rendering (lang=fr, country=France):

  You are a strict quality filter for multilingual instruction-tuning data. You are
  fluent in French and knowledgeable about France. Only the highest-quality
  examples should be kept; low-quality data harms training.

  Give ONE holistic quality score, 0 to 10, for the example as training data...

  DISQUALIFYING TASKS — score 0-2 no matter how correct, fluent, or obedient the
  response is, if the INSTRUCTION is any of these: ...

  Otherwise the task is worthwhile — score by the SEVERITY of the single WORST
  problem in the RESPONSE...

  LANGUAGE NOTE for French: individual English loanwords or technical terms are
  FINE where French speakers use them...

  Example (French, France):
  Instruction:
  {instruction}

  Response:
  {response}

  Output JSON only, no commentary, no code fences:
  {"score": <integer 0-10>, "reason": "<one sentence>"}

Filtering and deduplication

The judge's score (0-10, from the holistic prompt above) is used in two separate stages:

1. Quality filtering. quality_filter.py joins each generated row to its judge score by id and keeps only rows scoring at or above a per-language threshold. Unscored or errored rows are dropped, not kept-by-default.

2. Deduplication. Only rows that survived the quality filter are deduped - exact-hash by default, plus embedding-based near-dup detection when --embedding-dedupe is set (as it is in the current production run). The judge itself plays no role in dedup beyond having already removed the low-quality rows before dedup runs.

Deliverable scope

The synthgen.localized pipeline produces a final deduped, quality-filtered dataset of ≥100k survivors per language across all 35 target languages. We would ideally also want to test it on Prelude, Qwen-3 and Apertus.

Related to: Issue #345