q2 render should take a lockfile in the output directory so two concurrent renders cannot interleave their writes

Open
#900 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
48/100
Issue type
Feature
Clarity
Clearly specified
Activity status
Active
Tech stack
rust
Domain
build-system

Research direction

Start by locating the q2 render and q2 preview render entry points and how they determine the output directory; the issue does not name files or tests. Then trace render lifecycle and signal handling to see where a lock can be acquired and released. Done means concurrent renders are rejected with a useful diagnostic, stale locks do not block rendering, and preview holds the lock only during rendering.

Written by the indexing model from the issue text.

Description

Summary

Two renders of the same project can run at the same time and write the same output directory: a q2 preview --static that re-renders on file changes, and a q2 render started by hand (or by an editor task, a CI step, a second terminal). Nothing stops the second one, and the two interleave their writes. Proposal: q2 render takes a lockfile in the output directory for the duration of a render and steps back, with a diagnostic, when another render already holds it. Quarto 1 never did this; it is cheap and catches the easy mistakes.

What happened

A website project was being served with q2 preview --static, which watches the project and re-renders it when a source file changes. While it was running, I edited _quarto.yml and ran q2 render in another terminal. Both renders ran at once into _site/.

The render completed without diagnostics, but the search index was broken afterwards:

  • site_libs/quarto-search/pagefind/pagefind-entry.json named an index hash (en_3a8a4314f7),
  • the matching pagefind.en_3a8a4314f7.pf_meta file was missing from the directory,
  • in the browser, every query returned "No results", with Failed to load the meta index: Error: invalid gzip data and Pagefind: WASM Error (No pointer) on the console (the server answered the missing file with a 404 text body, which the runtime tried to gunzip).

Stopping the preview and rendering once more fixed it. I could not reproduce the missing file with a single renderer, with or without incremental renders, so the cause is the interleaving: one render's search-index step wrote its files while the other was replacing the directory.

The search index is just the piece that broke visibly. Any whole-directory output (site_libs/, the Pagefind index, llms-full.txt, alias redirects) is open to the same interleaving, and the symptoms are confusing because the final log line of each render says it succeeded.

Proposal

  • When a render starts, create a lockfile in the output directory (for example _site/.quarto-render.lock, or next to the other render bookkeeping under .quarto/). Write the pid, the start time, and the command that holds it.
  • If the lock exists and its holder is alive, do not render. Emit a diagnostic naming the holder ("another render of this project is running: q2 preview pid 12345, started 10:57:03") and exit non-zero. A --wait flag or a short bounded wait before giving up would cover the common case of a preview mid-render.
  • If the lock exists but its pid is gone, treat it as stale: remove it, say so, and continue. A crash must not wedge the project.
  • q2 preview holds the lock only while it is actually rendering, not while it is merely serving and watching, so a manual render between edits still works when the preview is idle.
  • Release the lock on success, failure, and signal.

Races remain possible (two renders starting in the same instant, network file systems, lock files are not transactions), and that is acceptable. The goal is that the obvious mistake, a hand-run render against a watched project, fails loudly instead of corrupting the output silently.

Environment

q2 0.35.0-nightly.20261021, macOS.

Dominant language
Rust
Stars
302
Forks
22
Avg merge
5h 2m
Merged PRs (30d)
213

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from quarto-dev/q2

All issues in quarto-dev/q2

Similar issues

More Rust issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.