fix(api): isRetryableError in volume_util.go does not recognize gRPC codes.Unavailable and codes.DeadlineExceeded

Start in packages/api/internal/handlers/volume_util.go at isRetryableError and review how executeOnOrchestratorByClusterID uses its result. Then inspect or add the table-driven tests in packages/api/internal/handlers/volume_util_test.go for the listed network, context, and gRPC status errors. Done means transient node failures are retryable while InvalidArgument remains non-retryable, with the tests passing.

Problem

In packages/api/internal/handlers/volume_util.go, executeOnOrchestratorByClusterID coordinates volume operations (CreateVolume, DeleteVolume) across orchestrator nodes in a cluster. When an RPC to a node fails, the loop calls isRetryableError(err) to decide whether to log the error and retry the operation against the next candidate node in the cluster pool.

However, isRetryableError only inspects:

func isRetryableError(err error) bool {
    if errors.Is(err, net.ErrClosed) || errors.Is(err, context.DeadlineExceeded) {
        return true
    }
    return false
}

When an orchestrator node daemon is temporarily unreachable (e.g. host reboot, daemon restarting, connection refused, or node timeout), gRPC transport errors are wrapped in status.Status with codes:

• codes.Unavailable (desc = "connection error: desc = transport: Error while dialing: dial tcp ... connect: connection refused")
• codes.DeadlineExceeded (desc = "context deadline exceeded")
• codes.Canceled (desc = "context canceled")

Because errors.Is(err, net.ErrClosed) does not unwrap gRPC status.Error instances, isRetryableError(err) returns false. As a result, the loop immediately terminates on the very first offline or restarting node, failing the volume creation/deletion request and returning an HTTP 500 error to the client, even when multiple healthy candidate orchestrator nodes are ready and available in the same cluster.

Root Cause

isRetryableError failed to inspect gRPC status errors using status.FromError(err) and match transient gRPC error codes:

Reproduction Steps

1. Configure a cluster with two orchestrator nodes: node-1 (stopped/offline) and node-2 (running/healthy).
2. Issue a volume creation request: POST /volumes.
3. executeOnOrchestratorByClusterID selects node-1 first.
4. The gRPC call fails with status.Error(codes.Unavailable, "connection refused").
5. Observed: isRetryableError returns false, causing the loop to abort immediately and return an error without trying node-2.
6. Expected: isRetryableError returns true, logs the retry on node-1, and successfully executes on node-2.

Technical Context

• File affected: packages/api/internal/handlers/volume_util.go
• Subsystem: Control Plane API / Volumes / High Availability & Node Failover
• Impact: High (Cluster node failover reliability and volume provisioning resilience)

Proposed Changes