epic: `builds.json` is large (somewhat by design)

未关闭
#138 1 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

还没有人认领这个 Issue。

评估

难度
5/5
预计耗时
一周以上
新手友好度
12/100
Issue 类型
重构
描述清晰度
需要澄清
活跃度
停滞
技术栈
json, shell
领域
tooling

调研方向

Start by reading ./cmd/builds (builds.sh) to see how builds.json is assembled from sources.json, then open the referenced production builds.json and measure it with jq to confirm the baseline. The issue weighs several designs (tabs, zero indent, compact output, dropping the duplicated source data, splitting files), so "done" depends on a maintainer choosing one; the smallest concrete step is the --tab switch in the writer. Read the closing thoughts and the one comment before proposing anything, since consumers outside the repository may depend on the current shape.

由索引模型根据 Issue 内容生成。

描述

At the current moment, our production builds.json file is sitting at 55.2MiB, which is kind of large for a JSON file (and large enough that GitHub complains about it on every push, although not large enough that they block it -- that limit is 100MiB). 🤏

So while this isn't an urgent problem, it's certainly something we should be looking at and thinking about how to scale further, especially as the system grows and contains more data. 👀

low hanging fruit (change almost nothing)

tabs (--tab)

If we do nothing but switch from (jq's default) two-space indent to tabs, we drop to 49MiB, which is a large enough drop that it's probably worth doing even though it's not a silver bullet. ➡️

(all further serious ideas will assume a baseline of swapping to tabs for size comparison, if keeping a monolithic file)

At our current length of builds.json (7708), that's ~6.5KiB per build object.

If that ratio holds without change, we can fit ~15737 of these inside GitHub's 100MiB limit (that's slightly more than twice as many builds total as we currently have).

With a work-in-progress prototype of some future work that I have (which needs to store more data inside builds.json), I've apparently increased this to ~7.5KiB, which takes that "GitHub maximum" number down to ~13619 (still ~1.8x our current count).

zero indentation

If we were willing to sacrifice readability, we could strip all the leading whitespace and get down to 42.7MiB, but IMO that's a pretty hefty sacrifice (I frequently open the raw file and look at it, especially while debugging, but I can absolutely adjust my workflows if I need to).

non-Git-friendly presentations

To keep this file in Git, we do need to stick to a line-based presentation, and so we might as well stick to the formatted display, but just for completeness:

--compact-output

If we did --compact-output we get all the way down to 40.8MiB.

"worse is better" ? (gzip)

If we get worse and gzip it (requiring even more work to do anything with it because jq can't handle a gzip'd JSON file directly), we can get down to 4.4MiB (tabs) or 4.2MiB (compact), but that's honestly an insane thought -- just giving the ol' gzip a shout-out because this is literally a use case it's really, really good at. 🫶

adjusting other workflows / "breaking" changes

strip out sources.json duplication

The current structure of builds.json was designed so that for a given build, essentially all the information we need exists directly in the one "build" object (ie, classic NoSQL schema design - design your data to look like how you'll need to query it). This means the sources.json content (8.3MiB total, 7.1MiB if we apply the same --tab treatment to it) is all duplicated inside builds.json, and in many cases that content gets repeated because we're intentionally exploded by architecture.

If we remove that source duplication and require loading the source data separately (--slurpfile sources sources.json, for example, assuming that file doesn't grow extremely large over time too), and change/flatten nothing else (del(.[].source)), that takes us to 38.7MiB. If we also then flatten the build objects (remove the then-unnecessary .build indirection; map_values(. * .build | del(.source, .build))), we get down to 37.9MiB.

(This result surprises me a little, because I thought we'd get more bang than 49 - 37.9 out of removing the duplication, but I guess it's pretty slim; only ~156% of sources.json's 7.1MiB in there.)

For comparison, that's drops us to ~5KiB per build object (from ~6.5KiB above), and ~6.3KiB per in my new PoC (down from ~7.5KiB), so this essentially gives us a little bit of space to play with for adding more data and still have 2x headroom on our current build count.

multiple files

A seemingly obvious answer (and actually how the first designs of this system were structured!) is to split builds into separate files. This is essentially trading the size issue for a discovery/traversal issue instead.

There are various ways we can mitigate the traversal issues, like maintaining symlink farms, but IMO the gains of doing all that work are pretty marginal.

In general, reading multiple files via jq is mostly easy enough, even if a little awkward (jq '...' *.json, --slurp, etc), but it does make any workflows that need to write data complicated because now it can't be jq '...' builds.json > some-new-file.json without potentially reintroducing this problem (if we solve it via multiple files).

For example, this means that ./cmd/builds (builds.sh) that's responsible for creating builds.json no longer simply has one canonical output, but now has to write individual JSON documents to separate files (or we have to have a shell script wrapper that does that, which is kind of heinous).

closing thoughts

After compiling all this data, I think my own opinion is that we should definitely swap to tabs, as I've noted above.

If we feel like that's not enough (for now?), I think we should consider dropping the sources.json data from builds.json and/or dropping more of that if there's parts of it that aren't necessary or could be represented in a better way (but both of those are changes that require more careful coordination because we might have other things consuming this data in various ways outside of this repository).

主要语言
Go
星标
19
派生
24
平均合并
8 天 22 小时
30 天内合并 PR
1

环境准备

这个项目没有提供开发容器、Dockerfile 或贡献指南,环境需要你自己搭建:先看它的 README,通用步骤见我们的新手贡献指南。

从这里开始

  1. 先读完整个 Issue,再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库,在一个分支上完成修改。
  4. 提交 Pull Request,并在描述里引用这个 Issue 编号。

docker-library/meta-scripts 的其他 Issue

查看 docker-library/meta-scripts 的全部 Issue

相似的 Issue

更多 Go Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。