[Bug] Rolling N+1 upgrade stalls indefinitely when a `redis-cli --cluster reshard` does not finish

Start by tracing handleUpgradeResharding and handleUpgradeEnding, which run the two rolling upgrade reshards, and compare them with the existing timed rebalance path using rebalanceTimeoutSeconds. Then inspect the redis-cli --cluster wrapper and its unit tests for timeout handling and truncated output; the issue also calls for a real Redis integration test under test/integration. Done when both reshards are bounded, the error output retains its cause, and the listed timeout and migration cases pass.

Summary

During a Rolling N+1 upgrade, Robin runs redis-cli --cluster reshard with the reconcile loop's context, which has no timeout. If the reshard does not finish, Robin's reconcile loop stays blocked inside that call. It does not run any further reconcile cycles, does not log errors and does not retry, so the upgrade never completes.

Rebalances (redis-cli --cluster rebalance, used by scaling and health remediation) already have a timeout, robin.config.cluster.rebalanceTimeoutSeconds. The two reshard calls of the rolling upgrade do not use it:

• handleUpgradeResharding: moves the slots of the current partition to the destination node.
• handleUpgradeEnding: moves the slots of the extra node back to node 0.

How it was found

The E2E suite failed in CI on the PR for #103 (https://github.com/InditexTech/redkey-operator/pull/103) (persistent upgrade with retained PVCs). The failing spec is not related to that change: 195 of 196 specs passed, and the only failure was an existing upgrade spec. The same spec had passed in every local run before the push.

• [FAILED] [906.695 seconds]
Authentication and Upgrade Rolling upgrade with authentication from start [It] should upgrade the Redis image while preserving auth and data [auth, upgrade]
/home/runner/work/redkey-operator/redkey-operator/redkey-operator/test/e2e/auth_and_upgrade_test.go:251

  Timeline >>
  STEP: triggering the image upgrade @ 10/19/26 07:36:56.801
  STEP: waiting for upgrade to complete @ 10/19/26 07:36:56.814
  STEP: waiting for the active config to be Applied @ 10/19/26 07:36:56.814
  [FAILED] in [It] - .../test/e2e/auth_and_upgrade_test.go:154 @ 10/19/26 07:51:56.814

Ran 196 of 202 Specs in 3900.067 seconds
FAIL! -- 195 Passed | 1 Failed | 0 Pending | 6 Skipped

Robin logs (collected by the E2E debug dump). Robin starts the reshard of partition 2 at 07:37:22. From then until the spec times out at 07:51:56, the only lines it logs are the metrics collector's. There is no Finished reconciliation cycle, no error and no retry:

time=2026-10-19T07:37:21.962Z level=INFO msg="Rolling upgrade: scale up complete, starting resharding" startPartition=2
time=2026-10-19T07:37:21.967Z level=INFO msg="Finished reconciliation cycle" nextSchedule=2 onError=false
time=2026-10-19T07:37:21.969Z level=INFO msg="Rolling upgrade: resharding slots from partition to destination" partition=2 destOrdinal=3
time=2026-10-19T07:37:22.051Z level=INFO msg="Membership healing complete" forgot=false forgotSlotOwner=false met=false
time=2026-10-19T07:37:22.057Z level=INFO msg="Rolling upgrade: migrating slots" partition=2 sourceID=28f8c0548df3b9c309b1ef6b694add0c1184eb99 destID=578b77313a27881b78265ef3449b3ae71e7f332d slots=5461
time=2026-10-19T07:37:34.396Z level=INFO msg="Starting metrics collection cycle" nodes=4 infoKeys=18
time=2026-10-19T07:37:34.413Z level=INFO msg="Metrics collection cycle completed" collectedNodes=4 failedNodes=0
time=2026-10-19T07:38:34.415Z level=INFO msg="Starting metrics collection cycle" nodes=4 infoKeys=18
...                                                    (only metrics lines, every minute)
time=2026-10-19T07:48:34.505Z level=INFO msg="Metrics collection cycle completed" collectedNodes=4 failedNodes=0

Shortly before the reshard, one node had briefly reported the destination node (578b7731…) without an address (:6379@16379). Membership healing had then re-met it. This is a plausible trigger. The logs do not show what redis-cli was waiting for.

Analysis

• When every node runs Redis 8.4 or later, redis-cli --cluster reshard uses Atomic Slot Migration (ASM). It creates a server-side task (CLUSTER MIGRATION IMPORT) and polls it every second. Unless --cluster-timeout is set, it waits up to 60 minutes, then cancels the task. The E2E images (redis:8.10, redis:8-alpine) use this path.
• Robin runs the reshard with no time bound, so the reconcile loop stays blocked for as long as redis-cli waits. During that time Robin cannot heal the topology or retry, and the upgrade cannot make progress.

Proposed fix

Run both upgrade reshards with the same timeout as rebalances: add a reshardWithTimeout helper that wraps ClusterReshard in context.WithTimeout(ctx, rebalanceTimeoutSeconds). When the timeout expires, Robin kills redis-cli, returns an error, and the next reconcile cycle retries.

Behaviour on timeout, verified against real Redis 8.10.2 (about 2 million keys):

• ASM (all nodes on Redis 8.4 or later): killing redis-cli does not cancel the server-side task. The task keeps running and then switches slot ownership in a single atomic step. While it runs, Robin's retries are rejected with ERR overlapping import exists for slot range. Once it finishes, the source owns no slots and the upgrade continues. No key is lost and nothing is migrated twice. redis-cli --cluster fix, which Robin runs before each retry, does not interfere with a running task.
• Key-by-key migration (any node older than 8.4): slots already moved stay moved. redis-cli --cluster fix closes the slot left half-migrated, and the next reshard only moves the slots the source still owns. This path follows from how redis-cli works and has not been tested with older binaries.

On a busy or large cluster where a reshard takes longer than the timeout (600 s by default), Robin stops waiting and the migration still finishes on the server. Until it finishes, each reconcile cycle logs an expected overlapping import error.

Related improvement

redis-cli --cluster errors are cut to the first 2000 characters of the output. redis-cli prints the cause last, for example ERR overlapping import exists ... or *** ... please retry, after a long list of Moving slot N lines, so the cause is lost. The fix keeps both the beginning and the end of the output.

Known limitation (possible follow-up)

If an ASM task is truly stuck (for example, the destination cannot reach the source), Robin does not cancel it, and Redis keeps retrying it on its own. Robin could cancel it with CLUSTER MIGRATION CANCEL ID <id> once the timeout has been exceeded and the task reports errors.

Tests

• Unit tests:
  • a hung redis-cli (stubbed with sleep 30) is killed after the configured timeout;
  • a reshard that finishes within the timeout succeeds;
  • truncated errors keep both the beginning and the end of the output.
• Integration test against real Redis (test/integration, label redis): kills redis-cli while an ASM task is running, retries the way Robin does (cluster fix followed by reshard), and checks that:
  • the retries are rejected as overlapping import while the task runs,
  • all slots end up on the destination,
  • every key is served exactly once.

Documentation

• upgrade.md, robin.md and dynamic-configuration.md: rebalanceTimeoutSeconds now also limits the reshards of a Rolling N+1 upgrade, and the docs describe what happens on timeout.
• The documented default of rebalanceTimeoutSeconds is fixed from 120 to 600 seconds, the value the code actually uses (cluster-health-checks.md, dynamic-configuration.md).