Make fewer git calls per operation

Aperta
  • Qualcosa non va?
#325 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
45/100
Tipo di issue
Refactoring
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
git, rust
Ambito
developer-experience, performance

Direzione di ricerca

Start with the helpers in crates/newgit-core/src/source.rs and crates/newgit-core/src/compose.rs, then inspect crates/newgit-core/tests/restore_sparse_paths.rs and measure Git process counts with GIT_TRACE2_EVENT. The issue lists batching and disabling auto-maintenance as options, but choosing an approach requires understanding the repo’s operations; done means measuring call counts before and after and restoring the macOS suite to pull request CI.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

newgit starts a new git process for nearly every question it asks the repo. On Linux that costs little. On macOS a process start is expensive, and on GitHub's macOS runners it's the whole cost of CI: the suite there is CPU-bound, at ~30ms per git spawn. That's why macOS left PR CI in #321. A developer's Mac pays the same cost on every spawn, checkpoint, and undo.

How much

crates/newgit-core/tests/restore_sparse_paths.rs starts 816 git processes for 4 tests (counted with GIT_TRACE2_EVENT):

subcommand count
rev-parse 116
rev-list 97
maintenance (auto, after fetch/commit) 62
for-each-ref 57
fetch 53
update-ref 51
cat-file 37
config 29
diff-tree 23
unpack-objects 22

These are the read and ref plumbing calls that git can batch.

Where to look

Most calls go through a few helpers: git() in crates/newgit-core/src/source.rs, and git(), git_env(), git_lines() and command() in crates/newgit-core/src/compose.rs. compose.rs already uses cat-file --batch in places, so the pattern exists.

Options, cheapest first

  1. Turn off auto-maintenance in newgit's own repos. Pass -c maintenance.auto=false -c gc.auto=0 (or set it in the store's config). That removes ~8% of spawns at once. newgit should probably decide when its internal repos get gc'd anyway, rather than have git do it after every fetch.
  2. Batch what git already batches:
    • rev-parse A B C in one call instead of three.
    • update-ref --stdin for groups of ref updates. This is also atomic, which is a correctness win.
    • One for-each-ref with a format string instead of repeated lookups.
    • A long-lived cat-file --batch / --batch-check per operation.
  3. Cache within an operation. Many rev-parse and for-each-ref calls probably ask the same question more than once within a single command.
  4. Use a library for reads (gix/libgit2), which avoids spawning anything for read-only queries. This is in tension with "never reimplement git's object model": it isn't reimplementing it, but it does add a second implementation that newgit depends on. Needs a decision before anyone starts.

Done when

The git call count for a typical spawn / checkpoint / undo is measured before and after, and the macOS suite is fast enough to put back on pull requests.

Lingua principale
Rust
Stelle
0
Fork
0
Merge medio
7h 27m
PR unite (30g)
140

Preparare l'ambiente

Non abbiamo ancora controllato i file di configurazione di questo progetto. Parti dal suo README e consulta la nostra guida al primo contributo per i passaggi generali.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di Spheroman/newgit

Tutte le issue di Spheroman/newgit

Issue simili

Altre issue su Rust

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.