Native scans use memory proportional to row count, and retired generations are never collected

Start with WP_Markdown_Native_Table_Provider::read(), validate_rows(), and bounded_rows(); inspect inc/native/class-wp-markdown-native-table-mutations.php and the surrounding inc/native code for generation cleanup. Reproduce the scan with SAVEQUERIES and measure memory, query time, and retained generation directories. Done means retired generations are collected and scans no longer materialize the full table before filtering.

Full-table scans on a high-row-count table cost memory proportional to row count rather than result size, and retired generations are never collected. On a real local site this turns 2.5 MB of matching payload into a 1.2 GB peak and a 47-second query, and leaves 3.0 GB on disk for a table whose useful content is a few megabytes.

Related: #232 (engine epic), #45 (skip file reads for metadata-only queries — adjacent but distinct).

Measured

Native engine (WP_Markdown_Native_WPDB), MARKDOWN_DB_MODE=primary, content layout post-type-hierarchy. Table is a plugin jobs log, wp_datamachine_jobs.

Single query under SAVEQUERIES:

SELECT status, created_at, completed_at FROM `wp_datamachine_jobs`
WHERE engine_data LIKE '%brain\_root%some-value%' ORDER BY job_id DESC

queries=251  total_sql_time=47.66s  peak_mem=1160MB
  47.45s  n=1   <the query above>
   0.04s  n=1   SELECT option_name, option_value FROM wp_options WHERE autoload IN (...)
   0.03s  n=1   SELECT ID, post_parent, ... FROM wp_posts WHERE ID IN (...)

One query is 99.6% of all SQL time in the request. Bootstrap for a trivial command on the same site is 171 MB / 0.74s, so this is not fixed overhead.

Measurement	Value
Rows in table	271,542
Total engine_data bytes across all rows	2.5 MB
Peak memory to scan	1,228 MB
Wall clock	~47 s
Logical bytes per row file	1,334
Disk blocks per row file	4 KB (≈3x padding waste)
Files per generation	271,542, flat directory
Non-empty generation-* dirs retained	4
Total on-disk size	3.0 GB

Row file shape:

{"_mdi_partition":{"version":1,"identity_column":"job_id","identity":"269926"},
 "row":{"job_id":"269926","status":"completed","engine_data":"…"}}

Three distinct defects

1. Retired generations are never collected

wp-content/markdown/_tables/datamachine_jobs/ holds four non-empty generation-* directories (435 MB, 551 MB, 1.0 GB, 1.0 GB) plus three orphaned 0-byte ones. I could not find a pruning path in inc/native/ — the retained occurrences in class-wp-markdown-native-table-mutations.php are row-level retention inside a mutation, not generation collection.

Roughly 2 GB is reclaimable on this one table. Independent of the other two, and cannot change query semantics.

2. Scans materialize the whole table

WP_Markdown_Native_Table_Provider::read() already declares iterable|WP_Markdown_Query_Result, so the interface permits streaming. The helpers defeat it:

• validate_rows( mixed $rows ): array|WP_Markdown_Query_Result
• bounded_rows( array $rows, WP_Markdown_Native_Table_Access $access, ?callable $hydrate = null, bool $ordered = false ): array|WP_Markdown_Query_Result

Both take/return arrays, so the full row set lands in PHP memory before predicates are applied. Memory should be O(result), and for a predicate that matches almost nothing it should be near-constant.

This is the fix that matters most: it makes peak memory independent of table size.

3. One file per row does not scale

A full scan is 271,542 open() + json_decode() pairs. That is the 47 seconds, and it is unavoidable while the layout is one file per row. The 4 KB block floor also means a 1.3 KB row costs 4 KB on disk.

This is fine at document scale — the same site's wiki is ~655 posts — and pathological at log scale. Worth reconsidering a chunked/segmented layout for high-cardinality tables, but only after #2, since streaming may make it tolerable.

Suggested sequencing

1. Generation GC — isolated, reclaims ~2 GB, no semantic risk.
2. Lazy scan iteration — the real fix; removes the need for callers to offer "fast" and "thorough" variants of the same report.
3. Re-measure, then decide whether the per-row layout still needs changing.

Consumer-side note

Two contributors are not this project's fault and are being tracked separately: the jobs table has no retention policy at 271k rows, and the query itself is a leading-wildcard LIKE over a JSON blob column, which can never use an index. Both should improve independently. Neither changes the fact that a 271k-row scan should not need 1.2 GB.

Reproduction

# with SAVEQUERIES enabled via a WP-CLI before_wp_load hook
php -d memory_limit=4G wp --path=<site> <command-that-scans-a-large-table>
# observe: peak RSS scales with row count, not with matched rows
du -sh wp-content/markdown/_tables/<table>/generation-*
ls wp-content/markdown/_tables/<table>/generation-*/ | wc -l

---

AI assistance disclosure: investigated by Anthropic's Claude running in OpenCode, directed by me. The agent profiled the query path with SAVEQUERIES, measured memory and on-disk layout, located the materialization points in the provider interface, and drafted this report. I reviewed the findings and evidence before filing.