fix(signals): cut the memory peak of the inbox ranking nn examples step
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 25/100
- Issue type
- Bug
- Clarity
- Clearly specified
- Activity status
- Active
- Domain
- data-engineering, performance
Research direction
Start with item 1: add peak_rss_metadata() to inbox_ranking_nn_examples so the baseline is measured before touching memory paths. Then read nn/dag.py around _day_inputs, load_snapshots and example_moments, plus ReportEmbeddingsFeatureSet.buildable and build_matrix, to see where snapshots and report vectors are held. Done means the examples object is row-for-row unchanged on the parity cases and peak RSS is reported before and after each item.
Written by the indexing model from the issue text.
Description
Problem
The inbox_ranking_nn_examples step of inbox_ranking_nn_training_job holds far more in memory than it uses. Its peak grew about 0.3 GiB a day and reached the run pod's 24Gi limit, so the job was OOMKilled on two nights (#136106 raises the limit to 32Gi as a stopgap).
The step writes rows, labels and holdout flags only. Most of its memory goes to data it never reads, or reads one field of:
- Report vectors, held twice.
_day_inputsinnn/dag.pykeeps the day's embeddings snapshot as an Arrow table and again as a pandas object column (report_vectors). The examples step uses them only forbuildable: does the report have a full-length vector, and when did it land. It never reads a vector's values. - The day's labels snapshot for project vectors.
_day_inputsalso reads it. The examples step never uses it. - Ninety full snapshots.
load_snapshotskeeps every report's state and labels for every day of the lookback window. At the birth grain,example_momentskeeps only reports born on the snapshot's own day, so almost all of those rows are dropped unread.
The sizes are read from the code and from the pod's memory curve (a ramp to about 12 GiB while snapshots load, then a jump to about 23.6 GiB when the embeddings load). They are not profiled. Step 1 below fixes that first.
Proposed changes
Each item can be its own PR. The examples object must not change: same rows, labels, holdout flags and carried state columns for the same partition.
1. Record peak RSS on the examples asset
Add peak_rss_metadata() to inbox_ranking_nn_examples, as the heavy dataset assets do. This gives a measured baseline for the items below and makes the trend visible before the next kill.
2. Give the examples step its own, lighter day inputs
- Do not read the labels snapshot in the examples step.
- Do not keep the embeddings Arrow table alive once the side input is built.
_day_inputsstays as it is for the candidate assets, which need all three.
3. Let buildable run without the vectors
The examples step needs three things per report: report_id, embedding_inserted_at, and whether the vector has EMBEDDING_DIMENSIONS values.
- Compute the has-vector flag in Arrow (list length), without converting vectors to Python or numpy objects.
- Let
ReportEmbeddingsFeatureSet.buildableanswer from that light frame.build_matrixkeeps reading real vectors. - Keep the as-of rule exactly as it is: a vector that landed after the moment, or has no landing time, is not buildable.
- This must not change any feature set's
feature_names,schema_versionor recipe hash.
4. Trim snapshots to the rows the birth grain can use
Add an opt-in argument to load_snapshots (default unchanged, because the main training job shares it) so the nn examples step keeps, for each day D:
- state rows of reports created on D;
- label rows of those reports on D, and on D + each head's
horizon_days.
Things that must keep working:
drop_without_training_consentreadsreport_team_idfrom state and must exclude the same reports.- A report deleted before a later snapshot keeps its label row there.
example_tablereadsreport_team_id,priorityand the carried state columns from each moment's own snapshot.- Trim before
assemble_snapshotso the provenance loop also runs on fewer rows.
Only the birth grain allows this trim. A set at another grain must get the full snapshots.
5. Cheaper reads in the loader
- Pass the column list to the Parquet read so unneeded columns are never decoded. Check the object's schema first, because older snapshots lack some columns.
- Avoid holding the raw object bytes and the decoded table at the same time for the embeddings object.
- Use categorical dtypes for the repeated string columns (
status,priority,dismissal_reason,latest_status_event) if the measurements from item 1 show they matter.
Verification
- Parity test: for a fixed set of snapshots, the examples table built with each change equals the one built without it, row for row. Cover a report with a vector that landed after its birth day, a report with no vector, a report from a team without training consent, and a report deleted before its horizon snapshot.
- On a real partition: rebuild one existing partition's examples object and compare it with the stored one before trusting a scheduled run.
- Memory: report the examples step's peak RSS before and after each item, from the metadata added in item 1.
Acceptance
- The examples step's peak is well under half of its current value on a full 90-day window.
- The peak no longer grows with the lookback window's row count.
- The examples object for a partition is unchanged.
- The job's memory limit goes back down from 32Gi (#136106) once the scheduled runs confirm the new peak.
Out of scope
- The candidate assets' memory. They stay at 8 to 11 GiB today.
- The dataset job's memory (#130394).
- Alerting on a killed run the same day.
- Dominant language
- Python
- Stars
- 40.8k
- Forks
- 3.6k
- Avg merge
- 6h 43m
- Merged PRs (30d)
- 254
Getting set up
- Ships a Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from PostHog/posthog
-
self-driving
Difficulty 1/5 Under an hour Newbie friendliness 74/100
PostHog/posthog#136106 · 1 comment ·
Maintainers usually reply within 1 day
-
Dashboards: selecting a SQL variable value equal to its default is dropped when the dashboard saved another valuePossibly taken @Inf1mum claimed this 2 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
PostHog/posthog#132024 · 1 comment ·
Maintainers usually reply within 1 day
-
self-driving
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
PostHog/posthog#130181 · 2 comments ·
Maintainers usually reply within 1 day
-
posthog-cli api: `call`, `info` and `search` read `--help` as a tool namePossibly taken @lazerg claimed this 8 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 67/100
PostHog/posthog#129166 · 1 comment ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
PostHog/posthog#128849 · 2 comments ·
Maintainers usually reply within 1 day
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
huggingface/kernels#1006 ·
Maintainers usually reply within 1 day
-
precond_kda (new op, #1552) backward kernels hit CUDA illegal memory access on Blackwell under triton 3.3.1, pass under triton 3.7.1Possibly taken @YYuJLiu claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
fla-org/flash-linear-attention#1749 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
bridgecrewio/checkov#7804 ·
Maintainers usually reply within 2 days
-
area:filesystem bug need-triage
Difficulty 2/5 1-3 hours Newbie friendliness 85/100
Maintainers usually reply within 1 day
-
triage/confirmed
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
agentscope-ai/agentscope#4068 ·
Maintainers usually reply within 2 days