examples/us_budgetary_impact.py consumes excessive memory (peak ~7.5 GB RSS / 82 GB VMS)
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 48/100
- Issue type
- Bug
- Clarity
- Mostly clear
- Activity status
- Quiet
- Tech stack
- python
- Domain
- performance
Research direction
Start with examples/us_budgetary_impact.py and reproduce the baseline_sim.run() and reform_sim.run() calls on the enhanced CPS 2024 dataset. Profile both runs to identify dominant allocations and determine whether intermediate data can be released or shared. Done means reducing the memory footprint or documenting the peak requirement in the example header.
Written by the indexing model from the issue text.
Description
Reported in the discussion on #264 (JOSS paper PR).
When run on the joss-paper branch (4.2.1), examples/us_budgetary_impact.py exhibits very high memory pressure during the baseline + reform simulations on the enhanced CPS 2024 dataset (2026, ~101k people / 41k households):
- Vahid: OOM-killed at peak ~30 GB RSS on a Mac (rc=137).
- Anthony (instrumented run): peak 7.48 GiB RSS, 82.35 GiB VMS;
baseline_sim.run()546.6s wall,reform_sim.run()596.5s wall; VSCode crashed mid-run from memory pressure.
This makes the canonical US population example impractical for normal local development and for JOSS reviewers attempting to reproduce results.
Suggested next steps:
- Profile
baseline_sim.run()andreform_sim.run()to find the dominant allocations (likely inpolicyengine-uscore, but worth confirming). - Determine whether intermediate arrays can be released between runs or whether the two simulations can share underlying data.
- Document peak memory requirement in the example header until the footprint is reduced.
Related: #264
- Dominant language
- Python
- Stars
- 8
- Forks
- 9
- Avg merge
- 15h 28m
- Merged PRs (30d)
- 13
Getting set up
- No Dockerfile or Docker Compose file
- No pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from PolicyEngine/policyengine.py
-
Raise the simulation cache memory warning to 16 GiBPossibly taken @anth-volk claimed this 3 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
PolicyEngine/policyengine.py#549 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
PolicyEngine/policyengine.py#473 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
PolicyEngine/policyengine.py#436 · 1 comment ·
Maintainers usually reply within 1 day
-
Certify populace-uk as the UK certified defaultMay be free again A pull request for this issue was closed without being merged. Open
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
PolicyEngine/policyengine.py#408 ·
Maintainers usually reply within 1 day
-
Difficulty 4/5 3-5 days Newbie friendliness 48/100
PolicyEngine/policyengine.py#551 ·
Maintainers usually reply within 1 day
All issues in PolicyEngine/policyengine.py
Similar issues
-
namespace operations
Difficulty 1/5 Under an hour Newbie friendliness 72/100
EclipseFdn/open-vsx.org#13737 ·
Maintainers usually reply within 1 day
-
Zero-token evaluations are treated as missing cost in selectionPossibly taken @sylvesterkaczmarek claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
google-research/rrsi#6 ·
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
aws-samples/sample-aws-genai-db-modernizer#294 ·
Maintainers usually reply within 1 day
-
feedback simulation workshop
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
githubnext/gh-aw-workshop#4174 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 86/100
WingedGuardian/GENesis-AGI#2852 ·
Maintainers usually reply within 1 day