Turn cabal-install into a build graph orchestrator

Open
#12,121 1 comment 11 reactions 0 assignees View on GitHub

Maintainers usually reply within 3 days

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
35/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Quiet
Tech stack
haskell
Domain
build-system

Research direction

Start by reading Distribution.Simple.Build.build_setupHooks and Distribution.Client.ProjectBuilding.rebuildTargets, then review the linked RFC and implementation commits to understand the build information now available. The first phase is done when Cabal exposes the listed build steps as separate rules and cabal-install can execute those rules as a graph while still using ghc --make for Haskell modules.

Written by the indexing model from the issue text.

Description

cabal-install: cmd/build type: discussion type: enhancement type: performance

#976 and #4174 both discuss the idea of cabal-install keeping track of a per-module build graph. However, recent architectural changes in cabal-install significantly change the design constraints.

Firstly, we now have -jsem. This allows us to recover module-level compilation parallelism when invoking cabal-install with --semaphore. (We should move towards making it the default.)

The remaining enhancements from #976 are:

  1. Cross-package pipelining. If package q depends on p, we should be able to start compiling modules of q once all the modules in p on which it transitively depends have been compiled, instead of waiting for the entire of p to have been built.
  2. Improved recompilation checking. We can avoid the overhead of ghc --make invocations (see #6658) and can rebuild at a finer granularity.

Secondly, HF RFC #060 and its implementation in c5f99331266a88a833a69b5766b69aaf68f24097 + 6867dd5da57194337bab0079bbe2f1c21e11662f provides cabal-install with a complete picture of what it is building (instead of opaque ./Setup build invocations which it can't peek into). This allows us to rethink the level of granularity that cabal-install is operating at: it can operate on the module level instead of the package level (with some opaque single ./Setup build nodes for legacy build-type: Custom packages).

New design

Let's look at what Distribution.Simple.Build.build_setupHooks currently does:

  1. pre-build: generate autogenerated files (e.g. paths module), run pre-build rules.
  2. Run pre-processors (alex, happy,...).
  3. Build all Haskell modules in one go with one ghc --make invocation.
  4. Build non-Haskell code (e.g. C files).
  5. Link.
  6. Install/register.

To cabal-install, this is one opaque blob today (see Distribution.Client.ProjectBuilding.rebuildTargets).

This ticket concretely suggests to make Cabal return the individual steps above as separate rules, and implement a graph-based orchestrator in cabal-install that executes rule graphs.

For the time being, we would continue to use ghc --make to build all Haskell modules in one go. Therefore this ticket is only the first phase in implementing per-module build graphs in cabal-install.

The gains we get from this first phase are minor but not insignificant:

  • Finer-grained recompilation checking (e.g. can entirely avoid invoking GHC for building Haskell modules if a C file changes).
  • Slightly better cross-package pipelining (e.g. we can start running preprocessors in q before Haskell module dependencies of p have been built).

This ticket is agnostic about the second phase, which concerns splitting up ghc --make invocations. We can use many individual one-shot ghc -c invocations, but this means a lot more work dealing with interface files. Or we could be more ambitious and use a persistent GHC worker where everything is kept in-memory.

Dominant language
Haskell
Stars
1.8k
Forks
766
Avg merge
5d 21h
Merged PRs (30d)
27

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from haskell/cabal

All issues in haskell/cabal

Similar issues

More Haskell issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.