Turn cabal-install into a build graph orchestrator

Aperta
#12,121 1 commento 11 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 3 giorni

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
35/100
Tipo di issue
Funzionalità
Chiarezza
Abbastanza chiara
Stato di attività
Tranquilla
Stack tecnologico
haskell
Ambito
build-system

Direzione di ricerca

Inizia leggendo Distribution.Simple.Build.build_setupHooks e Distribution.Client.ProjectBuilding.rebuildTargets, poi esamina l’RFC collegato e i commit di implementazione per comprendere le informazioni di build ora disponibili. La prima fase è completata quando Cabal espone i passaggi di build elencati come regole separate e cabal-install può eseguire tali regole come un grafo continuando a usare ghc --make per i moduli Haskell.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

cabal-install: cmd/build type: discussion type: enhancement type: performance

#976 and #4174 both discuss the idea of cabal-install keeping track of a per-module build graph. However, recent architectural changes in cabal-install significantly change the design constraints.

Firstly, we now have -jsem. This allows us to recover module-level compilation parallelism when invoking cabal-install with --semaphore. (We should move towards making it the default.)

The remaining enhancements from #976 are:

  1. Cross-package pipelining. If package q depends on p, we should be able to start compiling modules of q once all the modules in p on which it transitively depends have been compiled, instead of waiting for the entire of p to have been built.
  2. Improved recompilation checking. We can avoid the overhead of ghc --make invocations (see #6658) and can rebuild at a finer granularity.

Secondly, HF RFC #060 and its implementation in c5f99331266a88a833a69b5766b69aaf68f24097 + 6867dd5da57194337bab0079bbe2f1c21e11662f provides cabal-install with a complete picture of what it is building (instead of opaque ./Setup build invocations which it can't peek into). This allows us to rethink the level of granularity that cabal-install is operating at: it can operate on the module level instead of the package level (with some opaque single ./Setup build nodes for legacy build-type: Custom packages).

New design

Let's look at what Distribution.Simple.Build.build_setupHooks currently does:

  1. pre-build: generate autogenerated files (e.g. paths module), run pre-build rules.
  2. Run pre-processors (alex, happy,...).
  3. Build all Haskell modules in one go with one ghc --make invocation.
  4. Build non-Haskell code (e.g. C files).
  5. Link.
  6. Install/register.

To cabal-install, this is one opaque blob today (see Distribution.Client.ProjectBuilding.rebuildTargets).

This ticket concretely suggests to make Cabal return the individual steps above as separate rules, and implement a graph-based orchestrator in cabal-install that executes rule graphs.

For the time being, we would continue to use ghc --make to build all Haskell modules in one go. Therefore this ticket is only the first phase in implementing per-module build graphs in cabal-install.

The gains we get from this first phase are minor but not insignificant:

  • Finer-grained recompilation checking (e.g. can entirely avoid invoking GHC for building Haskell modules if a C file changes).
  • Slightly better cross-package pipelining (e.g. we can start running preprocessors in q before Haskell module dependencies of p have been built).

This ticket is agnostic about the second phase, which concerns splitting up ghc --make invocations. We can use many individual one-shot ghc -c invocations, but this means a lot more work dealing with interface files. Or we could be more ambitious and use a persistent GHC worker where everything is kept in-memory.

Lingua principale
Haskell
Stelle
1.8k
Fork
766
Merge medio
5g 21h
PR unite (30g)
27

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di haskell/cabal

Tutte le issue di haskell/cabal

Issue simili

Altre issue su Haskell

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.