Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

Convert SOCI indexes on the native build runners

Open
  • Is something wrong?
#857 1 comment 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
35/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Active
Tech stack
docker, python
Domain
devops

Research direction

Start by tracing publish() stage 1, SociConvertWorkflow, the per-platform build workflow, and clean.yml; inspect the existing build metadata and temporary registry tags. First determine what single-platform soci convert --standalone emits and whether its image manifest digest matches buildx's output. Done means build runners publish reusable SOCI indexes, publish assembles a valid multi-platform index with the required descriptor fields, missing indexes use the existing path, and cleanup removes the new tags.

Written by the indexing model from the issue text.

Description

cicd docker enhancement

SOCI conversion is the slowest part of bakery ci publish, so move it onto the per-platform build runners and only assemble the converted indexes at publish time.

Today, stage 1 of publish() runs per target on the single merge runner with 4 workers: create the multi-platform index, pull every layer of both platforms into an OCI layout, run soci convert --standalone --all-platforms, and push the result back to the temp registry. Stage 2 (the ordered copy to Docker Hub and GHCR) cannot start until every target finishes stage 1.

Estimated from the latest scheduled production runs (stage boundaries are not logged yet, so these are approximate):

Image Publish Stage 1 (index + SOCI) Stage 2 (copy)
workbench (16 targets, 46 GB) 30 min ~26 min ~2.5 min
connect (18) 21 min ~11.5 min ~8.5 min
package-manager (18) 7.5 min ~2.5 min ~4.75 min

Published images already contain one SOCI index per platform, each tied only to its own platform's image manifest. Nothing requires converting both platforms together, so each build runner can convert the image it just built, in parallel across the whole build matrix.

What the published layout shows (ghcr.io/posit-dev/connect:latest):

  • The platform image manifests are plain image manifests with no SOCI annotations; conversion does not rewrite them.
  • Each SOCI index is a separate manifest (config media type application/vnd.amazon.soci.index.v2+json) whose layers are the per-layer zTOCs.
  • The two are linked only by annotations on the descriptors in the multi-platform index: com.amazon.soci.index-digest on the image descriptor, and com.amazon.soci.image-manifest-digest plus artifactType on the SOCI index descriptor.
  • So a build runner only uploads the SOCI index manifest and its zTOC blobs; the image layers are already in the temp registry.

Proposal:

  • Build runner: when the workflow is pushing and SOCI is enabled for the target, convert the per-platform image after the build and push the converted result to the temp registry. Skip it entirely when not pushing.
  • Discovery: tag the converted result by its source digest (e.g. tmp:soci-<source-digest-hex>) so publish can find it without changing the build metadata, and fall back to today's path when it is missing.
  • Publish: when every platform of a target has a pre-converted index, write the multi-platform index JSON directly from their descriptors (preserving platform, artifactType, and the SOCI annotations), push it with oras manifest push, and skip SociConvertWorkflow. oras manifest index create is likely to drop those descriptor fields.
  • First iteration pulls the image from the temp registry on the build runner to reuse the existing workflow. A follow-up can write a local OCI layout instead (a second buildx type=oci output, or docker save from the containerd store) to skip the pull; both should cost seconds per image, so conversion itself is likely the dominant cost.
  • Log pull/convert/push durations on the build runner so the gain is measurable.
  • Interim, no code: run publish on a larger merge-builder and raise --jobs. This only helps if conversion is bound by CPU, disk, or network on that one runner, so pair it with the duration logging. In progress in #858.

Open questions to answer first:

  • What does soci convert --standalone emit for a single-platform input: an index wrapping the image and SOCI index, or something else?
  • Is the platform manifest in the published index byte-identical to the digest buildx pushed?
  • Does clean.yml remove the new soci-* tags from the temp repo?

Risks:

  • Assembling an index from separately converted children is the riskiest part. #793 showed that a child without platform breaks soci convert.
  • Digest-keyed discovery reuses a stale SOCI index if soci options change without the image changing.
  • Conversion moves onto each build job's critical path; total wall-clock should still drop because it runs across all runners at once.
  • Likely supersedes #772, since publish would no longer hold a local converted layout.

Related:

Dominant language
Python
Stars
2
Forks
0
Avg merge
2d 1h
Merged PRs (30d)
25

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from posit-dev/images-shared

All issues in posit-dev/images-shared

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.