Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

Reuse a fit's CSV metadata on subsequent reads

Open
  • Is something wrong?
#1,314 4 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
48/100
Issue type
Refactor
Clarity
Mostly clear
Activity status
Active
Tech stack
r
Domain
performance

Research direction

Start by reading read_cmdstan_csv() and tracing how CmdStanMCMC and CmdStanGQ call it for repeated reads; the issue names these entry points but not specific files or tests. The proposed change separates metadata merging from reading and reuses raw-name metadata on later reads. Done means subsequent reads on a fit skip rescanning its CSV files while the exported function keeps the same arguments and return value; run relevant tests for CSV reads and both fit classes.

Written by the indexing model from the issue text.

Description

internal-code

I noticed then when reviewing #1312 and I had Claude run some tests and help generate this report.

read_csv_metadata() greps every chain file for its comment lines, so the cost of reading the metadata grows with the number of draws even though the metadata is a few hundred lines. Every read_cmdstan_csv() call pays it per file: every $draws() on a variable not yet read, every $summary() and $sampler_diagnostics() on a fresh fit, and as_cmdstan_fit().

Measured on macOS with CmdStan 2.40.0, 4 chains, 1000 draws, medians of 5:

per chain file 4 metadata scans one-variable read_cmdstan_csv() call
10 parameters, 0.16 MB 0.12 s 0.15 s
1000 parameters, 11 MB 0.44 s 0.91 s

So the scan is most of a read on a small fit and about half on a large one, roughly 10 ms per MB of CSV on each read.

The fix is for a fit to reuse its own metadata: after the first read the fit holds the merged metadata and its files don't change, so later reads pass it back to the reader and skip the scan. That leaves one scan per fit, on the first read.

Possible design: the parse-and-merge loop in read_cmdstan_csv() becomes merge_csv_metadata(files), returning the merged metadata with the per-chain inverse metric and step size lists. The body of read_cmdstan_csv() moves to an internal function with a csv_metadata argument that skips the merge when given one and returns it, and the exported function stays a wrapper with the same arguments and return value. CmdStanMCMC and CmdStanGQ (the two classes that read more than once) store the merged metadata on the first read and pass it on later ones. The processed metadata the fit already stores can't be handed back directly: the reader selects columns by the raw CSV names (theta.1) and the fit's copy has the repaired ones (theta[1]).

Dominant language
R
Stars
162
Forks
69
Avg merge
1d 10h
Merged PRs (30d)
45

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from stan-dev/cmdstanr

All issues in stan-dev/cmdstanr

Similar issues

More R issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.