fix(signals): cut the memory peak of the inbox ranking nn examples step

Open
#136,107 1 comment 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
25/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
pandas, python

Research direction

Start with item 1: add peak_rss_metadata() to inbox_ranking_nn_examples so the baseline is measured before touching memory paths. Then read nn/dag.py around _day_inputs, load_snapshots and example_moments, plus ReportEmbeddingsFeatureSet.buildable and build_matrix, to see where snapshots and report vectors are held. Done means the examples object is row-for-row unchanged on the parity cases and peak RSS is reported before and after each item.

Written by the indexing model from the issue text.

Description

self-driving

Problem

The inbox_ranking_nn_examples step of inbox_ranking_nn_training_job holds far more in memory than it uses. Its peak grew about 0.3 GiB a day and reached the run pod's 24Gi limit, so the job was OOMKilled on two nights (#136106 raises the limit to 32Gi as a stopgap).

The step writes rows, labels and holdout flags only. Most of its memory goes to data it never reads, or reads one field of:

  • Report vectors, held twice. _day_inputs in nn/dag.py keeps the day's embeddings snapshot as an Arrow table and again as a pandas object column (report_vectors). The examples step uses them only for buildable: does the report have a full-length vector, and when did it land. It never reads a vector's values.
  • The day's labels snapshot for project vectors. _day_inputs also reads it. The examples step never uses it.
  • Ninety full snapshots. load_snapshots keeps every report's state and labels for every day of the lookback window. At the birth grain, example_moments keeps only reports born on the snapshot's own day, so almost all of those rows are dropped unread.

The sizes are read from the code and from the pod's memory curve (a ramp to about 12 GiB while snapshots load, then a jump to about 23.6 GiB when the embeddings load). They are not profiled. Step 1 below fixes that first.

Proposed changes

Each item can be its own PR. The examples object must not change: same rows, labels, holdout flags and carried state columns for the same partition.

1. Record peak RSS on the examples asset

Add peak_rss_metadata() to inbox_ranking_nn_examples, as the heavy dataset assets do. This gives a measured baseline for the items below and makes the trend visible before the next kill.

2. Give the examples step its own, lighter day inputs
  • Do not read the labels snapshot in the examples step.
  • Do not keep the embeddings Arrow table alive once the side input is built.
  • _day_inputs stays as it is for the candidate assets, which need all three.
3. Let buildable run without the vectors

The examples step needs three things per report: report_id, embedding_inserted_at, and whether the vector has EMBEDDING_DIMENSIONS values.

  • Compute the has-vector flag in Arrow (list length), without converting vectors to Python or numpy objects.
  • Let ReportEmbeddingsFeatureSet.buildable answer from that light frame. build_matrix keeps reading real vectors.
  • Keep the as-of rule exactly as it is: a vector that landed after the moment, or has no landing time, is not buildable.
  • This must not change any feature set's feature_names, schema_version or recipe hash.
4. Trim snapshots to the rows the birth grain can use

Add an opt-in argument to load_snapshots (default unchanged, because the main training job shares it) so the nn examples step keeps, for each day D:

  • state rows of reports created on D;
  • label rows of those reports on D, and on D + each head's horizon_days.

Things that must keep working:

  • drop_without_training_consent reads report_team_id from state and must exclude the same reports.
  • A report deleted before a later snapshot keeps its label row there.
  • example_table reads report_team_id, priority and the carried state columns from each moment's own snapshot.
  • Trim before assemble_snapshot so the provenance loop also runs on fewer rows.

Only the birth grain allows this trim. A set at another grain must get the full snapshots.

5. Cheaper reads in the loader
  • Pass the column list to the Parquet read so unneeded columns are never decoded. Check the object's schema first, because older snapshots lack some columns.
  • Avoid holding the raw object bytes and the decoded table at the same time for the embeddings object.
  • Use categorical dtypes for the repeated string columns (status, priority, dismissal_reason, latest_status_event) if the measurements from item 1 show they matter.

Verification

  • Parity test: for a fixed set of snapshots, the examples table built with each change equals the one built without it, row for row. Cover a report with a vector that landed after its birth day, a report with no vector, a report from a team without training consent, and a report deleted before its horizon snapshot.
  • On a real partition: rebuild one existing partition's examples object and compare it with the stored one before trusting a scheduled run.
  • Memory: report the examples step's peak RSS before and after each item, from the metadata added in item 1.

Acceptance

  • The examples step's peak is well under half of its current value on a full 90-day window.
  • The peak no longer grows with the lookback window's row count.
  • The examples object for a partition is unchanged.
  • The job's memory limit goes back down from 32Gi (#136106) once the scheduled runs confirm the new peak.

Out of scope

  • The candidate assets' memory. They stay at 8 to 11 GiB today.
  • The dataset job's memory (#130394).
  • Alerting on a killed run the same day.
Dominant language
Python
Stars
40.8k
Forks
3.6k
Avg merge
6h 43m
Merged PRs (30d)
254

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from PostHog/posthog

All issues in PostHog/posthog

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.