TabPFN memory calibration grid for the admission estimator

Open
#8 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
52/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Active
Tech stack
python

Research direction

Start with benchmarks/calibrate_memory.py, benchmarks/gen_calibration_data.py, memory/calibration_tabicl_a100.py, and serve/factory.py::_preloaded_observations; review memory/adaptive.py for observation and margin handling. Run the specified A100 TabPFN sweeps for each cache mode, generate the backend-specific grids, add the build_estimator preload test mirroring TestCalibrationPreload, and rerun benchmarks/probe_deployment.py to confirm admission limits improve.

Written by the indexing model from the issue text.

Description

enhancement roadmap

Why

The factory-preloaded admission grid (memory/calibration_tabicl_a100.py) is TabICLv2 data measured on an A100-40GB. TabPFNBackend deployments therefore start from the conservative power-law formula plus whatever runtime observations accumulate (AdaptiveMemoryEstimator), so early TabPFN admissions are much tighter than the hardware allows and /v1/tabctx/limits under-reports what TabPFN can take. backends/tabpfn.py says as much in context_bytes_hint.

Since v0.10.0 the serving budget is the detected device capacity, so the grid only needs to describe the model's peak memory per shape; the card size is handled separately.

Tasks

  • Run benchmarks/calibrate_memory.py-style sweeps through TabPFNBackend on an A100 for the three cache modes (kv -> fit_with_cache, repr -> fit_preprocessors, off -> low_memory), recording peak fit bytes, resident bytes, predict-time peak at N_TEST_ROWS, and the first OOM per feature count. TabPFN's pretraining limits (~10k rows / ~500 features on the v2-series checkpoints, larger on TabPFN-3) bound the grid; record where the model rejects (422) vs where the device OOMs.
  • Extend benchmarks/gen_calibration_data.py to emit A100_40GB_TABPFN_* grids and teach serve/factory.py::_preloaded_observations to select the grid by backend kind (today it returns empty for anything but tabicl).
  • Unit test: build_estimator(ServeSettings(backends=("tabpfn",))) reports preloaded measurements in confidence() and estimates a grid shape from measurement, not formula (mirror TestCalibrationPreload).
  • Re-run benchmarks/probe_deployment.py on a TabPFN deployment and confirm max_admissible_train_rows_by_feature_count loosened accordingly.

Pointers

memory/adaptive.py (preloaded observations, margins), memory/calibration_tabicl_a100.py (generated file format), ROADMAP.md item 3.

Dominant language
Python
Stars
1
Forks
0
Avg merge
11h 13m
Merged PRs (30d)
11

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from VectorInstitute/tabctx

All issues in VectorInstitute/tabctx

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.