render: stopping Function runtimes aborts on the first error, and the shared 5s budget skips container removal

Start in cmd/crossplane/render/render.go at FunctionAddresses.Stop and StopFunctionRuntimes, then check the cleanup defaults and comment in cmd/crossplane/render/runtime_docker.go. Reproduce the failure with the supplied slow-stop container and add focused tests for failed stops and removal. Done means every runtime is attempted, Remove-policy containers are removed despite a slow stop, and the default-cleanup comment matches behavior.

What happened?

After a render, a single Function container that is slow to stop causes other Function containers to be
left running, and itself to be left behind stopped-but-not-removed, even with the default cleanup policy
(Remove).

Two defects in the stop path in
render.go
combine:

1. FunctionAddresses.Stop
   returns on the first error. Every runtime not yet visited is never stopped. Visit order is Go map order,
   so which containers leak varies from run to run.
2. StopFunctionRuntimes
   gives all runtimes one shared 5s deadline. ContainerStop is called with empty StopOptions, so the
   daemon waits its default 10s for SIGTERM before SIGKILL. A Function that doesn't exit promptly on SIGTERM
   (e.g. a PID-1 process with no signal handler) exceeds the deadline, ContainerStop errors, and the
   ContainerRemove that should follow is skipped — and, via (1), so is every remaining runtime.

Expected: every runtime is attempted regardless of others failing, and a slow SIGTERM can't prevent a
Remove-policy container from being removed.

For well-behaved Functions this doesn't trigger: with three real Functions, all containers went
kill → stop → destroy within about 1s of the render finishing. The problem is that cleanup doesn't survive
a single failure.

A unit test for (1) — one runtime whose Stop errors, one healthy — fails in 174/200 iterations (the
healthy runtime is never stopped):

package render

import (
	"context"
	"errors"
	"testing"
)

func TestFunctionAddressesStopSkipsRemainingOnError(t *testing.T) {
	skipped := 0
	const trials = 200
	for range trials {
		healthyStopped := false
		fa := &FunctionAddresses{contexts: map[string]RuntimeContext{
			"slow":    {Target: "slow:9443", Stop: func(context.Context) error { return errors.New("context deadline exceeded") }},
			"healthy": {Target: "healthy:9443", Stop: func(context.Context) error { healthyStopped = true; return nil }},
		}}
		_ = fa.Stop(context.Background())
		if !healthyStopped {
			skipped++
		}
	}
	if skipped > 0 {
		t.Errorf("Stop returned before stopping every runtime in %d/%d trials", skipped, trials)
	}
}

--- FAIL: TestFunctionAddressesStopSkipsRemainingOnError (0.00s)
    stop_repro_test.go:28: Stop returned before stopping every runtime in 174/200 trials

How can we reproduce it?

Build a stand-in "Function" that ignores SIGTERM:

# tagged fnlc-slow-stop:test
FROM alpine:3.20
ENTRYPOINT ["sh","-c","trap \"\" TERM; while true; do sleep 1; done","ignore-term"]

List it in functions.yaml but don't reference it from the pipeline, so the render itself succeeds and only
cleanup is exercised. Container names are set only to make leftovers easy to find; they don't change cleanup
behaviour. Replace $T with a trial number.

apiVersion: pkg.crossplane.io/v1
kind: Function
metadata:
  name: function-go-templating
  annotations: {render.crossplane.io/runtime-docker-name: fnlc-gotmpl-$T}
spec: {package: xpkg.crossplane.io/crossplane-contrib/function-go-templating:v0.11.0}
---
apiVersion: pkg.crossplane.io/v1
kind: Function
metadata:
  name: function-dummy
  annotations: {render.crossplane.io/runtime-docker-name: fnlc-dummy-$T}
spec: {package: xpkg.crossplane.io/crossplane-contrib/function-dummy:v0.4.1}
---
apiVersion: pkg.crossplane.io/v1
kind: Function
metadata:
  name: function-auto-ready
  annotations: {render.crossplane.io/runtime-docker-name: fnlc-autoready-$T}
spec: {package: xpkg.crossplane.io/crossplane-contrib/function-auto-ready:v0.5.1}
---
apiVersion: pkg.crossplane.io/v1
kind: Function
metadata:
  name: function-slow-stop
  annotations:
    render.crossplane.io/runtime-docker-name: fnlc-slow-$T
    render.crossplane.io/runtime-docker-image: fnlc-slow-stop:test
    render.crossplane.io/runtime-docker-pull-policy: Never
spec: {package: example.org/unused:v0.0.0}

xr.yaml:

apiVersion: example.org/v1
kind: XThing
metadata:
  name: test-xr
spec:
  coolField: hello

composition.yaml (does not reference function-slow-stop):

apiVersion: apiextensions.crossplane.io/v1
kind: Composition
metadata:
  name: xthings.example.org
spec:
  compositeTypeRef: {apiVersion: example.org/v1, kind: XThing}
  mode: Pipeline
  pipeline:
  - step: templating
    functionRef: {name: function-go-templating}
    input:
      apiVersion: gotemplating.fn.crossplane.io/v1beta1
      kind: GoTemplate
      source: Inline
      inline:
        template: |
          apiVersion: nop.crossplane.io/v1alpha1
          kind: NopResource
          metadata:
            annotations: {gotemplating.fn.crossplane.io/composition-resource-name: nop}
          spec: {forProvider: {}}
  - step: dummy
    functionRef: {name: function-dummy}
    input: {apiVersion: dummy.fn.crossplane.io/v1beta1, kind: Response, response: {}}
  - step: auto-ready
    functionRef: {name: function-auto-ready}

Run a few times, waiting longer than 10s after each so the daemon can finish any stop already in progress:

$ crossplane render xr.yaml composition.yaml functions-slow.yaml > /dev/null; echo "exit=$?"
exit=0
$ sleep 12; docker ps -a --filter 'name=fnlc-' --format '{{.Names}} {{.Status}}'
fnlc-slow-3 Exited (137) 7 seconds ago
fnlc-dummy-3 Up 18 seconds
fnlc-gotmpl-3 Up 19 seconds

Across 7 trials: the slow container was left Exited (137) and not removed in 7/7; the healthy
fnlc-dummy and fnlc-gotmpl were never stopped in 2/7 (trials where the slow runtime came first in map
order). --verbose shows:

INFO	Error stopping function runtimes	{"error": "cannot stop function \"function-slow-stop\" runtime (target \"fnlc-slow-3:9443\"): cannot stop Docker container: Post \"http://.../containers/0fb4.../stop\": context deadline exceeded"}

Possible fixes:

• FunctionAddresses.Stop: attempt every runtime and return errors.Join of the failures.
• StopFunctionRuntimes: give each runtime its own timeout (optionally stopping them concurrently) instead
  of one shared 5s budget.
• For the Remove policy, use a single ContainerRemove with Force: true, so a slow SIGTERM can't skip
  the removal.

While in this code: the doc comment on AnnotationValueRuntimeDockerCleanupStop
(runtime_docker.go L90)
says it "is the default", but AnnotationValueRuntimeDockerCleanupDefault (L102) is Remove, which is what
GetDockerCleanup returns when the annotation is unset. Worth correcting in the same change.

What environment did it happen in?

• Crossplane CLI version: built from main @ 29316fea54f2ede9d2c039d9c54f0c29cbad4b65
• Platform (e.g., linux/amd64): darwin/arm64, Docker Engine 29.5.3 (Rancher Desktop)
• Crossplane version (if applicable): render engine image xpkg.crossplane.io/crossplane/crossplane:stable