feat(signals): split activity summary updates across models from the flag payload

Aperta
#136,019 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
22/100
Tipo di issue
Funzionalità
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
python

Direzione di ricerca

Start with products/signals/backend/activity_summary/config.py, where the flag payload is parsed, and with summaries.py, where DEFAULT_MODEL is defined. Then follow logic._call_model, logic.update_activity_summary, tick.py, backfill.py and run_activity_summary to see every place that reads DEFAULT_MODEL. Done when a payload with model_variants picks a model per team and user pair, the rows and the activity summary generated event record that model, and the cost guard uses the most expensive variant.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

self-driving

Problem

The activity summary calls one model, the DEFAULT_MODEL constant in products/signals/backend/activity_summary/summaries.py. Trying another model needs a PR and a deploy, and it switches every summary at once. There is no way to run two models side by side on live traffic and compare their quality, cost, latency and failure rate.

Every other lever of the tick already lives in the activity-summary flag payload, parsed in activity_summary/config.py. The model does not.

Change

Add a weighted model split to the same flag payload. An unchanged payload keeps today's behavior: every update uses DEFAULT_MODEL.

Payload
{
  "model_variants": [
    { "model": "<default model>", "weight": 80 },
    { "model": "<candidate model>", "weight": 20 }
  ],
  "model_split_salt": "v1"
}
  • model_variants is a new field of ActivitySummaryConfig, a tuple of a frozen ModelVariant(model, weight). Weights are relative, so they do not have to add up to 100.
  • The parse keeps the fail-safe rule of the file. It drops an entry with an unknown model or a weight that is not a positive number. When no valid entry is left, the config has no split and the tick uses DEFAULT_MODEL.
  • model_split_salt defaults to a constant. A new salt reassigns every pair, which starts a new experiment.
Assignment: fixed per pair, not random per update

The flag is read once per tick for one fixed distinct id, so flag variants cannot do the split. The code picks the model from the payload.

A summary update takes the previous summary as input. A random model per update would mix the models inside one summary, and no summary could be attributed to one model. So the model is a fixed function of the pair:

  • Bucket = hash of salt:team_id:user_id, in the style of org_rollout.organization_rollout_bucket. The bucket picks a variant by cumulative weight.
  • Put the variants in a stable order (the order of the payload), so raising the weight of the last variant only moves pairs into it.
  • One pure function, for example model_for_pair(config, team_id, user_id) -> str, used by every caller.
Allowlist and cost guard
  • A model is valid only when it has an entry in MODEL_PRICES. So a typo in the payload cannot call an arbitrary model, and every model in a split has a real cost. Adding a candidate model is a small PR that adds its prices.
  • queue.max_update_cost_usd already takes a model. The daily cost guard must use the highest bound among the configured variants, so a more expensive candidate cannot pass max_cost_usd_per_day.
  • logic._call_model must price the call with the prices of the model it used, not of DEFAULT_MODEL.
Wiring

Replace each direct use of DEFAULT_MODEL with the model of the pair:

  • logic._call_model (the call and the cost).
  • The model field that logic.update_activity_summary stores on the row and on the version row.
  • The model property of the activity summary generated event in tick.py. It reads DEFAULT_MODEL today, so it would report the wrong model for a candidate pair.
  • backfill.py.
  • run_activity_summary: add an optional --model override, so a developer can build one pair with each model in a dry run without touching the flag.

A model change does not make a stored summary outdated. The next normal update of the pair uses the new model, like a prompt version that ages in. A forced rebuild stays the job of rebuild_before.

What to record
  • The row, the version row and the activity summary generated event already have model. After this change it holds the model that was called.
  • Add model_split_salt to the event, so two experiments with the same models stay apart.
  • Add the configured variants (model and weight) to the activity_summary_config_read log line.

Open questions

  • Does each candidate model support the request as sent: a JSON schema response_format, max_completion_tokens, and the gateway route signals? Check each candidate with the dry-run command before it gets a weight.
  • Should the embeddings of candidate pairs stay in the same documents as the default model's? They share the embedding model, so the proposal is yes, with model in the metadata as today.

Out of scope

  • A per-model prompt, or per-model render limits.
  • An automatic comparison or an eval harness. This issue only makes the split and records which model built each summary.
  • The personal ranking model (personal_ranking/config.py), which has its own config.

Tests

  • parse_config: no model_variants gives no split; an unknown model or a bad weight is dropped; a list with no valid entry gives no split.
  • model_for_pair: the same pair and salt always give the same model; over many pairs the shares match the weights; a new salt changes the assignment; raising the last weight only moves pairs into that variant.
  • An update of a candidate pair stores the candidate model on the row, prices the call with its prices, and reports it on the event.
  • The daily cost guard uses the most expensive configured variant.

Done when

Adding a second model with a weight to the flag payload makes that share of pairs build with it from the next tick, without a deploy, and the stored rows and the activity summary generated event show which model built each summary.

Lingua principale
Python
Stelle
40.8k
Fork
3.6k
Merge medio
16h 2m
PR unite (30g)
344

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di PostHog/posthog

Tutte le issue di PostHog/posthog

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.