xpkg build is non-deterministic: package.yaml key order varies per process, breaking multi-platform packages

Start at crossplane-runtime/pkg/xpkg/build.go, especially encode(), and trace the JSON serializer into k8s.io/apimachinery/pkg/runtime/serializer/json/json.go and doEncode. Run the provided repeated crossplane xpkg build reproduction and compare extracted package.yaml files. Done means identical output and base-layer bytes across separate builds, including the supplied interval key set.

What happened?

crossplane xpkg build is not deterministic. Building the same, unmodified package directory several times produces package.yaml files that differ in mapping key order — same byte length, different content, different digest.

The instability is not global: it only affects mappings whose key set happens to break the comparator that sigs.k8s.io/yaml sorts keys with (details below). Most packages never notice. When a package does contain such a key set, the resulting base layer digest changes on every build.

This breaks multi-platform packages. The standard build/makelib/xpkg.mk flow builds one .xpkg per platform in its own process and then composes an index from them:

build.artifacts.platform: do.build.xpkgs        # one `crossplane xpkg build` per platform
xpkg.release.publish: crossplane xpkg push --package-files .../linux_amd64/x.xpkg \
                                            --package-files .../linux_arm64/x.xpkg  ...

Because the two builds are separate processes, they disagree, and the pushed index carries two different io.crossplane.xpkg: base layers for the same package. The Upbound registry rejects such packages outright:

status: rejected
reason: base layer content is inconsistent across images

Five consecutive releases of a provider were silently rejected this way before we traced it. Nothing in the build, the push, or the CLI output signals a problem — the failure only shows up in the registry afterwards.

How can we reproduce it?

Self-contained, ~40 lines. No provider runtime image needed.

package/crossplane.yaml:

apiVersion: meta.pkg.crossplane.io/v1
kind: Configuration
metadata:
  name: repro
spec:
  crossplane:
    version: ">=v1.14.0"

package/crds/xwidgets.yaml:

apiVersion: apiextensions.crossplane.io/v1
kind: CompositeResourceDefinition
metadata:
  name: xwidgets.example.org
spec:
  group: example.org
  names:
    kind: XWidget
    plural: xwidgets
  versions:
  - name: v1alpha1
    served: true
    referenceable: true
    schema:
      openAPIV3Schema:
        type: object
        properties:
          spec:
            type: object
            properties:
              interval:
                type: object
                properties:
                  interval12Hours: {type: object}
                  interval15Mins: {type: object}
                  interval1Day: {type: object}
                  interval1Hour: {type: object}
                  interval1Min: {type: object}
                  interval30Mins: {type: object}
                  interval30Secs: {type: object}
                  interval5Mins: {type: object}
                  interval6Hours: {type: object}

Then:

for i in 1 2 3 4 5; do
  crossplane xpkg build --package-root=package -o /tmp/repro$i.xpkg
done
# extract package.yaml from each .xpkg and hash it

Result — five builds of identical input, four distinct package.yaml hashes (all 1428 bytes):

1 1a5ee444a8f15499
2 a5708ff6a57cdb87
3 1a5ee444a8f15499
4 8c4d5b2b333aab44
5 73dc09b744fb3f27

diff between two of them shows only interval* keys moving.

The same reproducer fails identically with up xpkg build, which shares the code path.

Root cause

crossplane-runtime/pkg/xpkg/build.go → encode() re-serializes every package object through the apimachinery JSON serializer in YAML mode:

do := json.NewSerializerWithOptions(json.DefaultMetaFactory, objScheme, objScheme,
    json.SerializerOptions{Yaml: true})

In YAML mode that serializer does json.Marshal(obj) followed by sigs.k8s.io/yaml.JSONToYAML(...) (k8s.io/apimachinery/pkg/runtime/serializer/json/json.go, doEncode). JSONToYAML decodes the JSON into map[string]interface{} and re-marshals it with gopkg.in/yaml.v2.

yaml.v2 sorts mapping keys with keyList.Less, a "natural sort" that treats a run of digits as a number. That comparator is not a strict weak ordering. For the key set above:

interval12Hours < interval1Day  == true
interval1Day    < interval5Mins == true
interval12Hours < interval5Mins == false

sort.Sort over a non-transitive Less returns a result that depends on the input permutation — and the input permutation here comes from iterating a Go map, which is randomized per process. Same input, same key set, different output ordering each run.

Marshalling just those nine keys 200 times through sigs.k8s.io/yaml yields 56 distinct orderings. Zero-padding the digit runs (interval01Day, interval05Mins, …) makes every digit run the same width, restores a total order, and yields exactly 1.

Note this is inherent to the key names, not to package size or complexity: scanning every sibling-key set in two real shipped packages gave 4/575 unstable for the affected provider and 0/427 for provider-aws-ec2 v2.4.0. Large, long-standing packages are unaffected purely by luck of naming.

Suggested fix

Preserve the JSON document's key order instead of round-tripping through an unordered map. encoding/json already sorts map keys byte-lexicographically (a real total order) and emits struct fields in declaration order, so the JSON produced in step 1 is already deterministic — it is only the map[string]interface{} round-trip in JSONToYAML that discards and re-derives that order.

Converting JSON → YAML through an order-preserving representation (e.g. decoding into a yaml.v3 Node tree, or json.Decoder with UseNumber into an ordered structure) would make xpkg build byte-reproducible for all inputs, not just the lucky ones.

A cheaper interim mitigation, if the encoder is hard to change: have crossplane xpkg push verify that the base layer is byte-identical across the --package-files it is given, and fail loudly rather than publishing an index that registries will reject.

What environment did it happen in?

• Crossplane CLI version: v2.0.2 (also reproduced on crossplane/cli@main)
• crossplane-runtime: v2.4.0
• Also affects up xpkg build (upbound/up), same pkg/xpkg code path

Related

• crossplane/cli#344 — different symptom (field order within RawExtension inputs), same encode() re-serialization in crossplane-runtime/pkg/xpkg/build.go.