TabPFN memory calibration grid for the admission estimator
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 52/100
- Issue type
- Feature
- Clarity
- Mostly clear
- Activity status
- Active
- Tech stack
- python
- Domain
- backend, machine-learning, performance
Research direction
Start with benchmarks/calibrate_memory.py, benchmarks/gen_calibration_data.py, memory/calibration_tabicl_a100.py, and serve/factory.py::_preloaded_observations; review memory/adaptive.py for observation and margin handling. Run the specified A100 TabPFN sweeps for each cache mode, generate the backend-specific grids, add the build_estimator preload test mirroring TestCalibrationPreload, and rerun benchmarks/probe_deployment.py to confirm admission limits improve.
Written by the indexing model from the issue text.
Description
Why
The factory-preloaded admission grid (memory/calibration_tabicl_a100.py) is TabICLv2 data measured on an A100-40GB. TabPFNBackend deployments therefore start from the conservative power-law formula plus whatever runtime observations accumulate (AdaptiveMemoryEstimator), so early TabPFN admissions are much tighter than the hardware allows and /v1/tabctx/limits under-reports what TabPFN can take. backends/tabpfn.py says as much in context_bytes_hint.
Since v0.10.0 the serving budget is the detected device capacity, so the grid only needs to describe the model's peak memory per shape; the card size is handled separately.
Tasks
- Run
benchmarks/calibrate_memory.py-style sweeps throughTabPFNBackendon an A100 for the three cache modes (kv->fit_with_cache,repr->fit_preprocessors,off->low_memory), recording peak fit bytes, resident bytes, predict-time peak atN_TEST_ROWS, and the first OOM per feature count. TabPFN's pretraining limits (~10k rows / ~500 features on the v2-series checkpoints, larger on TabPFN-3) bound the grid; record where the model rejects (422) vs where the device OOMs. - Extend
benchmarks/gen_calibration_data.pyto emitA100_40GB_TABPFN_*grids and teachserve/factory.py::_preloaded_observationsto select the grid by backend kind (today it returns empty for anything buttabicl). - Unit test:
build_estimator(ServeSettings(backends=("tabpfn",)))reports preloaded measurements inconfidence()and estimates a grid shape from measurement, not formula (mirrorTestCalibrationPreload). - Re-run
benchmarks/probe_deployment.pyon a TabPFN deployment and confirmmax_admissible_train_rows_by_feature_countloosened accordingly.
Pointers
memory/adaptive.py (preloaded observations, margins), memory/calibration_tabicl_a100.py (generated file format), ROADMAP.md item 3.
- Dominant language
- Python
- Stars
- 1
- Forks
- 0
- Avg merge
- 11h 13m
- Merged PRs (30d)
- 11
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from VectorInstitute/tabctx
-
documentation roadmap
Difficulty 5/5 Over a week Newbie friendliness 42/100
-
roadmap
Difficulty 5/5 Over a week Newbie friendliness 35/100
-
Application catalog + persistent context store (serve many applications from pre-computed contexts) Openenhancement roadmap
Difficulty 5/5 Over a week Newbie friendliness 35/100
All issues in VectorInstitute/tabctx
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 86/100
EleutherAI/lm-evaluation-harness#4207 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
ClickHouse/clickhouse-connect#1057 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 70/100
open-telemetry/sig-end-user#406 ·
-
bug ci good first issue
Difficulty 2/5 1-3 hours Newbie friendliness 88/100