Turn cabal-install into a build graph orchestrator
Maintainers usually reply within 3 days
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 35/100
- Issue type
- Feature
- Clarity
- Mostly clear
- Activity status
- Quiet
- Tech stack
- haskell
- Domain
- build-system
Research direction
Start by reading Distribution.Simple.Build.build_setupHooks and Distribution.Client.ProjectBuilding.rebuildTargets, then review the linked RFC and implementation commits to understand the build information now available. The first phase is done when Cabal exposes the listed build steps as separate rules and cabal-install can execute those rules as a graph while still using ghc --make for Haskell modules.
Written by the indexing model from the issue text.
Description
#976 and #4174 both discuss the idea of cabal-install keeping track of a per-module build graph. However, recent architectural changes in cabal-install significantly change the design constraints.
Firstly, we now have -jsem. This allows us to recover module-level compilation parallelism when invoking cabal-install with --semaphore. (We should move towards making it the default.)
The remaining enhancements from #976 are:
- Cross-package pipelining. If package
qdepends onp, we should be able to start compiling modules ofqonce all the modules inpon which it transitively depends have been compiled, instead of waiting for the entire ofpto have been built. - Improved recompilation checking. We can avoid the overhead of
ghc --makeinvocations (see #6658) and can rebuild at a finer granularity.
Secondly, HF RFC #060 and its implementation in c5f99331266a88a833a69b5766b69aaf68f24097 + 6867dd5da57194337bab0079bbe2f1c21e11662f provides cabal-install with a complete picture of what it is building (instead of opaque ./Setup build invocations which it can't peek into). This allows us to rethink the level of granularity that cabal-install is operating at: it can operate on the module level instead of the package level (with some opaque single ./Setup build nodes for legacy build-type: Custom packages).
New design
Let's look at what Distribution.Simple.Build.build_setupHooks currently does:
pre-build: generate autogenerated files (e.g. paths module), run pre-build rules.- Run pre-processors (alex, happy,...).
- Build all Haskell modules in one go with one
ghc --makeinvocation. - Build non-Haskell code (e.g. C files).
- Link.
- Install/register.
To cabal-install, this is one opaque blob today (see Distribution.Client.ProjectBuilding.rebuildTargets).
This ticket concretely suggests to make Cabal return the individual steps above as separate rules, and implement a graph-based orchestrator in cabal-install that executes rule graphs.
For the time being, we would continue to use ghc --make to build all Haskell modules in one go. Therefore this ticket is only the first phase in implementing per-module build graphs in cabal-install.
The gains we get from this first phase are minor but not insignificant:
- Finer-grained recompilation checking (e.g. can entirely avoid invoking GHC for building Haskell modules if a C file changes).
- Slightly better cross-package pipelining (e.g. we can start running preprocessors in
qbefore Haskell module dependencies ofphave been built).
This ticket is agnostic about the second phase, which concerns splitting up ghc --make invocations. We can use many individual one-shot ghc -c invocations, but this means a lot more work dealing with interface files. Or we could be more ambitious and use a persistent GHC worker where everything is kept in-memory.
- Dominant language
- Haskell
- Stars
- 1.8k
- Forks
- 766
- Avg merge
- 5d 21h
- Merged PRs (30d)
- 27
Getting set up
- No Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from haskell/cabal
-
needs triage type: enhancement
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
Maintainers usually reply within 3 days
-
needs triage type: bug
Difficulty 2/5 1-3 hours Newbie friendliness 64/100
haskell/cabal#12568 · 1 comment ·
Maintainers usually reply within 3 days
-
needs triage type: bug
Difficulty 2/5 1-3 hours Newbie friendliness 67/100
haskell/cabal#12525 · 3 comments · 1 reaction ·
Maintainers usually reply within 3 days
-
Required version of `cabal-install` to build CabalPossibly taken @philderbeast claimed this 24 days ago. Opendocumentation type: discussion
Difficulty 2/5 1-3 hours Newbie friendliness 66/100
haskell/cabal#12477 · 2 comments · 1 assignee ·
Maintainers usually reply within 3 days
-
Document which are our LTS releasesPossibly taken @KeuqS claimed this 41 days ago. Opendocumentation type: discussion
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
haskell/cabal#12373 · 2 comments ·
Maintainers usually reply within 3 days
Similar issues
-
bug desktop
Difficulty 2/5 1-3 hours Newbie friendliness 85/100
simplex-chat/simplex-chat#8291 ·
Maintainers usually reply within 1 day
-
[BUG]: C# programs with top-level statements fail to compilePossibly taken @AsyncNigam claimed this today. Openbug
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
compiler-explorer/compiler-explorer#9625 · 1 comment ·
Maintainers usually reply within 1 day
-
reflex-0.10Open
Difficulty 1/5 Under an hour Newbie friendliness 62/100
commercialhaskell/stackage#8180 ·
Maintainers usually reply within 1 day
-
documentation
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
OpenZeppelin/canton-contracts#140 ·
Maintainers usually reply within 1 day
-
takeInterval yields nothing, dropInterval drops nothingPossibly taken @GuillaumedeVolpiano claimed this 1 day ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
composewell/streamly#3271 ·