RFC: Buck2 backend
Maintainers usually reply within 3 days
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 8/100
- Issue type
- Feature
- Clarity
- Needs clarification
- Activity status
- Active
- Domain
- build-system
Research direction
This is a design RFC, not a bounded change, and it names no source files or tests. Start by reading the open questions at the end, then the linked cabal-buck2 demo and the earlier RFC PR #12410 to see what has already been argued. A newcomer's useful contribution is to comment on those questions, not to write code; done means the maintainers reach a decision on whether build-type: Buck2 is supported.
Written by the indexing model from the issue text.
Description
(title by analogy with #9089, and this replaces an earlier RFC PR #12410)
Motivation
Cabal is a limited build system
A few of the problems I run into:
- Support for custom build steps and code-generation is limited. The new hooks mechanism provides some rudimentary support, but it's not enough for a complex project like Glean for example, which has multiple preprocessors and code generators. We currently use a
Makefilefor development and include all the generated code asextra-source-filesto make an sdist for uploading to Hackage. Developing this way is tedious. - C/C++ files' dependencies aren't tracked (#4306). This makes it really awkward to use Cabal for development where you're modifying C/C++ sources.
- C/C++ files aren't built in parallel (#7127)
These aren't problems we should be fixing in Cabal IMO; Cabal should be the front-end to a fully-featured build system.
Cabal is slow for development
Mainly because every time you run cabal build it has to redo a bunch of work and run around and figure out what the state of everything is. I know there's cabal repl but that's a whole other story - it's even more limited as a build system (I suspect the right thing is for cabal repl to run alongside a caching build system).
The Cabal store is a poor man's build cache
The Cabal store is a special-purpose build cache that only caches complete Cabal package builds and never deletes them. The store doesn't cache anything to do with the package you're building, only its dependencies.
We should outsource this job to a general-purpose build cache, which would support caching on a smaller granularity (individual build steps) and also allow for tuning the cache size rather than growing without bound. Buck2 can do this with a minimal amount of setup.
CI on github does a lot of repeated work and it's slow. There's support for caching in github workflows but it's painful to use and error-prone. If we instead had a shared build cache (storing blobs on Amazon S3 for example) everyone would get faster CI for minimal effort.
Demo
I've made a cabal buck2 external command: cabal-buck2 which allows you to build a Cabal project using Buck2. It has two modes:
- By default, all the external dependencies of the project are built by Cabal into the store, just like
cabal build all --only-dependencieswould, andcabal buck2then generates Buck2 rules to build the packages in your project against the store libraries. - With the
--source-depsoption, the entire dependency tree (except for GHC's built-in packages) is built by Buck2, bypassing the Cabal store entirely.
It also supports using a build cache, of which there are several open-source implementations. Building with --source-deps and a build cache is just like using the Cabal store (except much faster), and doesn't need any special support in Cabal or GHC. Everything to do with the store in Cabal is redundant, essentially.
Support is pretty complete: cabal buck2 can build the whole transitive dependency tree of cabal-install itself and a bunch of other things I've tried. It supports most of Cabal's features, including hsc2hs, cmm-sources, asm-sources, and so on. The biggest roadblock currently is build-type: Custom packages; these can't be built by cabal buck2 and prevent you from using --source-deps for some packages. Fortunately this is a legacy feature though.
Design questions
This is all well and good: we can have a nice development experience locally, and we can use Buck2 in CI on github. Setting up a local build cache is straightforward and we could have a shared build cache for github CI.
However
- It's still a bit of a pain to set things up; there are things we could do to make it smoother:
- ghcup support for installing Buck2
- some automation around setting up a build cache
- Having
cabal buck2be a built-in rather than an external command
- But I still can't use Buck2 to write custom build code in packages I upload to Hackage. For that we would need to support
build-type: Buck2, and commit to supporting that.
Thoughts? How does any of this interact or conflict with other plans and directions?
- Dominant language
- Haskell
- Stars
- 1.8k
- Forks
- 766
- Avg merge
- 5d 21h
- Merged PRs (30d)
- 27
Getting set up
- No Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from haskell/cabal
-
needs triage type: enhancement
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
Maintainers usually reply within 3 days
-
needs triage type: bug
Difficulty 2/5 1-3 hours Newbie friendliness 64/100
haskell/cabal#12568 · 1 comment ·
Maintainers usually reply within 3 days
-
needs triage type: bug
Difficulty 2/5 1-3 hours Newbie friendliness 67/100
haskell/cabal#12525 · 3 comments · 1 reaction ·
Maintainers usually reply within 3 days
-
Required version of `cabal-install` to build CabalPossibly taken @philderbeast claimed this 24 days ago. Opendocumentation type: discussion
Difficulty 2/5 1-3 hours Newbie friendliness 66/100
haskell/cabal#12477 · 2 comments · 1 assignee ·
Maintainers usually reply within 3 days
-
Document which are our LTS releasesPossibly taken @KeuqS claimed this 41 days ago. Opendocumentation type: discussion
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
haskell/cabal#12373 · 2 comments ·
Maintainers usually reply within 3 days
Similar issues
-
bug desktop
Difficulty 2/5 1-3 hours Newbie friendliness 85/100
simplex-chat/simplex-chat#8291 ·
Maintainers usually reply within 1 day
-
[BUG]: C# programs with top-level statements fail to compilePossibly taken @AsyncNigam claimed this today. Openbug
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
compiler-explorer/compiler-explorer#9625 · 1 comment ·
Maintainers usually reply within 1 day
-
reflex-0.10Open
Difficulty 1/5 Under an hour Newbie friendliness 62/100
commercialhaskell/stackage#8180 ·
Maintainers usually reply within 1 day
-
documentation
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
OpenZeppelin/canton-contracts#140 ·
Maintainers usually reply within 1 day
-
takeInterval yields nothing, dropInterval drops nothingPossibly taken @GuillaumedeVolpiano claimed this 1 day ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
composewell/streamly#3271 ·