Cutting CI build times: prebuilt Arrow and build configuration
维护者通常 2 天内回复
评估
- 难度
- 5/5
- 预计耗时
- 一周以上
- 新手友好度
- 35/100
- Issue 类型
- 重构
- 描述清晰度
- 需要澄清
- 活跃度
- 冷清
- 技术栈
- cmake, cpp, github-actions
- 领域
- build-system, ci-cd, performance
调研方向
首先阅读现有的 GitHub Actions 工作流、cpp-linter.yml、CMake 工具链文件，以及 issue 中描述的 fork 测量结果。将 Arrow 设置、build-flavor、Windows 配置、触发器和 sccache 方案与当前矩阵进行比较。当已实现一组约定的变更，并且完整的构建和测试矩阵确认了预期的覆盖范围和 CI 时间改进时，即视为完成。
由索引模型根据 Issue 内容生成。
描述
I have some CI changes I'd like to propose. They go a bit deeper than the recent caching work, so I wanted to open an issue first to align on direction before pursuing any of them. The pieces below are independent; taking forward only some of them works fine.
Resolving Arrow from conda-forge
Eleven jobs build the vendored Arrow and Parquet from source, but the toolchain already prefers a prebuilt one (the FetchContent declaration uses FIND_PACKAGE_ARGS); CI just never installs one. Installing libarrow and libparquet from conda-forge with setup-miniconda (ASF allowlisted) takes that compile out of each leg:
| Test leg | Build targets | Cold build step |
|---|---|---|
| Ubuntu | 951 to 606 | 21 to 16 min |
| macOS | 946 to 602 | 11 to 8 min |
| Windows | 900 to 578 | 47 to 35 min |
(fork measurements; times vary with runner load, target counts don't; the AWS, SQL catalog, sanitizer, and linter legs shrink similarly)
It is also most of our cache pressure: one push to main saves around 10 GB of sccache entries, more than the repository's 10 GB limit by itself, so eviction ends up deleting entries that have no newer replacement. Over the past week about every third push to main rebuilt at least one leg from scratch. With prebuilt Arrow, and without debug info that nothing in CI reads, saves drop under 3 GB.
Coverage holds: the AWS leg keeps building the bundled AWS SDK from source, the Meson legs don't use Arrow, and the sanitizer leg passes against the non-instrumented Arrow. The conda pin would track the version in the toolchain file and bump in the same PR.
Building one library flavor per leg
CI builds with ICEBERG_BUILD_STATIC and ICEBERG_BUILD_SHARED both ON, and the two targets compile the same sources twice (the shared build adds the export define and hidden visibility, so objects can't be reused). The tests link one flavor, so building one roughly halves what a leg compiles of our own code. A static-only fork run passes the full build and test matrix; one leg could keep both ON to keep both exercised.
Windows: build Debug like the other legs
The Unix test legs build Debug; Windows builds Release and has been the slowest leg fairly consistently. MSVC Debug needs embedded debug info (/Z7 via CMP0141) for sccache to cache the objects, and that combination is green on a fork. Windows is currently the only Release build in CI, though, so this is partly a question of what the matrix should cover.
Two smaller cleanups
The test workflows trigger on both push (all branches) and pull_request, and the concurrency groups key on different refs per event, so a branch pushed here with an open PR runs everything twice. Scoping push to main, as cpp-linter.yml already does, drops the duplicates and keeps the post-merge runs that seed the caches. Separately, the sccache steps are copy-pasted across nine jobs in five workflow files; a composite action under .github/actions/ would hold them (and the conda setup) in one place.
Questions
- Is conda-forge acceptable as a source of prebuilt Arrow in CI, and how would you want the version pin maintained?
- Single flavor on the test legs: which one, and is it enough to keep one leg building both static and shared?
- Windows on Debug: fine, or should the matrix keep a Release leg?
- Any concerns with the composite action or the push trigger scoping?
- 主要语言
- C++
- 星标
- 226
- 派生
- 132
- 平均合并
- 1 天 18 小时
- 30 天内合并 PR
- 26
环境准备
这个项目没有提供开发容器、Dockerfile 或贡献指南，环境需要你自己搭建：先看它的 README，通用步骤见我们的新手贡献指南。
从这里开始
- 先读完整个 Issue，再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库，在一个分支上完成修改。
- 提交 Pull Request，并在描述里引用这个 Issue 编号。
apache/iceberg-cpp 的其他 Issue
-
bug: ReferenceVisitor::GetReferencedFieldIds dereferences null on a bound COUNT(*)可能已有人在做 @LuciferYang 于 4 天前认领。 未关闭
难度 2/5 1-3 小时 新手友好度 82/100
apache/iceberg-cpp#978 ·
维护者通常 2 天内回复
-
bug: InMemoryCatalog skips the namespace existence check in CreateTable and RegisterTable可能已有人在做 @LuciferYang 于 4 天前认领。 未关闭
难度 2/5 1-3 小时 新手友好度 78/100
apache/iceberg-cpp#977 ·
维护者通常 2 天内回复
-
难度 2/5 1-3 小时 新手友好度 88/100
apache/iceberg-cpp#973 ·
维护者通常 2 天内回复
-
bug: expression JSON deserialization throws an uncaught exception on a non-string "type"/"term"可能已有人在做 @LuciferYang 于 4 天前认领。 未关闭
难度 3/5 1-2 天 新手友好度 75/100
apache/iceberg-cpp#979 ·
维护者通常 2 天内回复
-
难度 5/5 一周以上 新手友好度 20/100
apache/iceberg-cpp#959 · 1 条评论 ·
维护者通常 2 天内回复
查看 apache/iceberg-cpp 的全部 Issue
相似的 Issue
-
enhancement
难度 2/5 1-3 小时 新手友好度 72/100
维护者通常 1 天内回复
-
难度 2/5 1-3 小时 新手友好度 85/100
maplibre/maplibre-native#4723 ·
维护者通常 1 天内回复
-
难度 2/5 1-3 小时 新手友好度 78/100
HarbourMasters/Shipwright#7320 ·
维护者通常 1 天内回复
-
难度 2/5 1-3 小时 新手友好度 72/100
-
Make Catch2 optional when `RDK_BUILD_CPP_TESTS=OFF`可能已有人在做 @pechersky 今天认领。 未关闭bug
难度 2/5 1-3 小时 新手友好度 82/100
维护者通常 2 天内回复