feat(signals): split activity summary updates across models from the flag payload
I maintainer di solito rispondono entro 1 giorno
Nessuno ha ancora preso questa issue.
Valutazione
- Difficoltà
- 4/5
- Tempo stimato
- 3-5 giorni
- Idoneità per principianti
- 22/100
- Tipo di issue
- Funzionalità
- Chiarezza
- Abbastanza chiara
- Stato di attività
- Attiva
- Stack tecnologico
- python
- Ambito
- ai-infra-agents
Direzione di ricerca
Start with products/signals/backend/activity_summary/config.py, where the flag payload is parsed, and with summaries.py, where DEFAULT_MODEL is defined. Then follow logic._call_model, logic.update_activity_summary, tick.py, backfill.py and run_activity_summary to see every place that reads DEFAULT_MODEL. Done when a payload with model_variants picks a model per team and user pair, the rows and the activity summary generated event record that model, and the cost guard uses the most expensive variant.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Descrizione
Problem
The activity summary calls one model, the DEFAULT_MODEL constant in products/signals/backend/activity_summary/summaries.py. Trying another model needs a PR and a deploy, and it switches every summary at once. There is no way to run two models side by side on live traffic and compare their quality, cost, latency and failure rate.
Every other lever of the tick already lives in the activity-summary flag payload, parsed in activity_summary/config.py. The model does not.
Change
Add a weighted model split to the same flag payload. An unchanged payload keeps today's behavior: every update uses DEFAULT_MODEL.
Payload
{
"model_variants": [
{ "model": "<default model>", "weight": 80 },
{ "model": "<candidate model>", "weight": 20 }
],
"model_split_salt": "v1"
}
model_variantsis a new field ofActivitySummaryConfig, a tuple of a frozenModelVariant(model, weight). Weights are relative, so they do not have to add up to 100.- The parse keeps the fail-safe rule of the file. It drops an entry with an unknown model or a weight that is not a positive number. When no valid entry is left, the config has no split and the tick uses
DEFAULT_MODEL. model_split_saltdefaults to a constant. A new salt reassigns every pair, which starts a new experiment.
Assignment: fixed per pair, not random per update
The flag is read once per tick for one fixed distinct id, so flag variants cannot do the split. The code picks the model from the payload.
A summary update takes the previous summary as input. A random model per update would mix the models inside one summary, and no summary could be attributed to one model. So the model is a fixed function of the pair:
- Bucket = hash of
salt:team_id:user_id, in the style oforg_rollout.organization_rollout_bucket. The bucket picks a variant by cumulative weight. - Put the variants in a stable order (the order of the payload), so raising the weight of the last variant only moves pairs into it.
- One pure function, for example
model_for_pair(config, team_id, user_id) -> str, used by every caller.
Allowlist and cost guard
- A model is valid only when it has an entry in
MODEL_PRICES. So a typo in the payload cannot call an arbitrary model, and every model in a split has a real cost. Adding a candidate model is a small PR that adds its prices. queue.max_update_cost_usdalready takes amodel. The daily cost guard must use the highest bound among the configured variants, so a more expensive candidate cannot passmax_cost_usd_per_day.logic._call_modelmust price the call with the prices of the model it used, not ofDEFAULT_MODEL.
Wiring
Replace each direct use of DEFAULT_MODEL with the model of the pair:
logic._call_model(the call and the cost).- The
modelfield thatlogic.update_activity_summarystores on the row and on the version row. - The
modelproperty of theactivity summary generatedevent intick.py. It readsDEFAULT_MODELtoday, so it would report the wrong model for a candidate pair. backfill.py.run_activity_summary: add an optional--modeloverride, so a developer can build one pair with each model in a dry run without touching the flag.
A model change does not make a stored summary outdated. The next normal update of the pair uses the new model, like a prompt version that ages in. A forced rebuild stays the job of rebuild_before.
What to record
- The row, the version row and the
activity summary generatedevent already havemodel. After this change it holds the model that was called. - Add
model_split_saltto the event, so two experiments with the same models stay apart. - Add the configured variants (model and weight) to the
activity_summary_config_readlog line.
Open questions
- Does each candidate model support the request as sent: a JSON schema
response_format,max_completion_tokens, and the gateway routesignals? Check each candidate with the dry-run command before it gets a weight. - Should the embeddings of candidate pairs stay in the same documents as the default model's? They share the embedding model, so the proposal is yes, with
modelin the metadata as today.
Out of scope
- A per-model prompt, or per-model render limits.
- An automatic comparison or an eval harness. This issue only makes the split and records which model built each summary.
- The personal ranking model (
personal_ranking/config.py), which has its own config.
Tests
parse_config: nomodel_variantsgives no split; an unknown model or a bad weight is dropped; a list with no valid entry gives no split.model_for_pair: the same pair and salt always give the same model; over many pairs the shares match the weights; a new salt changes the assignment; raising the last weight only moves pairs into that variant.- An update of a candidate pair stores the candidate model on the row, prices the call with its prices, and reports it on the event.
- The daily cost guard uses the most expensive configured variant.
Done when
Adding a second model with a weight to the flag payload makes that share of pairs build with it from the next tick, without a deploy, and the stored rows and the activity summary generated event show which model built each summary.
- Lingua principale
- Python
- Stelle
- 40.8k
- Fork
- 3.6k
- Merge medio
- 16h 2m
- PR unite (30g)
- 344
Preparare l'ambiente
- Include un Dockerfile o un file Docker Compose
- Ha un modello di pull request
- Leggi la guida per i contributori
Come iniziare
- Leggi tutta la issue e poi la guida ai contributi del progetto.
- Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
- Fai un fork del repository e lavora su un branch.
- Apri una pull request che faccia riferimento al numero della issue.
Altre issue di PostHog/posthog
-
self-driving
Difficoltà 1/5 Meno di un'ora Idoneità per principianti 74/100
PostHog/posthog#136106 · 1 commento ·
I maintainer di solito rispondono entro 1 giorno
-
Dashboards: selecting a SQL variable value equal to its default is dropped when the dashboard saved another valueForse già presa @Inf1mum l’ha presa 3 giorni fa. Aperta
Difficoltà 2/5 1-3 ore Idoneità per principianti 75/100
PostHog/posthog#132024 · 1 commento ·
I maintainer di solito rispondono entro 1 giorno
-
self-driving
Difficoltà 2/5 1-3 ore Idoneità per principianti 78/100
PostHog/posthog#130181 · 2 commenti ·
I maintainer di solito rispondono entro 1 giorno
-
posthog-cli api: `call`, `info` and `search` read `--help` as a tool nameForse già presa @lazerg l’ha presa 8 giorni fa. Aperta
Difficoltà 2/5 1-3 ore Idoneità per principianti 67/100
PostHog/posthog#129166 · 1 commento ·
I maintainer di solito rispondono entro 1 giorno
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 78/100
PostHog/posthog#128849 · 2 commenti ·
I maintainer di solito rispondono entro 1 giorno
Tutte le issue di PostHog/posthog
Issue simili
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 72/100
samuelcolvin/dirty-equals#148 ·
-
nomination-2026
Difficoltà 1/5 Meno di un'ora Idoneità per principianti 70/100
elisa-tech/tsc#101 ·
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 63/100
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 78/100
JulianKemmerer/PipelineC#395 ·
-
The flake8 settings select checks that are not installed, and the test workflow passes inputs that setup-micromamba does not haveForse già presa @HTenkanen l’ha presa oggi. Aperta
Difficoltà 2/5 1-3 ore Idoneità per principianti 74/100
I maintainer di solito rispondono entro 1 giorno