[Bug] set-ramping-version and set-current-version return success without persisting state when backing system workflow is damaged

Abierto
  • ¿Algo no está bien?
#354 3 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
4/5
Tiempo estimado
3-5 días
Aptitud para principiantes
45/100
Tipo de issue
Error
Claridad
Bastante claro
Estado de actividad
Activo
Stack tecnológico
go
Área
backend, cli, distributed-systems

Línea de trabajo

Empieza reproduciendo los comandos set-ramping-version, set-current-version y delete-version indicados contra un workflow divergente de temporal-sys-worker-deployment; después, compara sus respuestas exitosas con worker deployment describe. Lee el plan executor delete+update issue (#353) relacionado y determina si el comportamiento corresponde a este controller o al servidor de Temporal. Se considera completado cuando cada operación persiste el estado solicitado o informa de un error en lugar de indicar un éxito falso.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

need information
Summary

When a Worker Deployment's backing system workflow (temporal-sys-worker-deployment:*) is in a damaged or divergent state, CLI operations (set-ramping-version --delete, set-current-version --unversioned, delete-version) return success messages but don't actually persist state changes. describe shows identical values and unchanged timestamps before and after.

Note: This may be a Temporal server bug rather than a controller bug, but filing here since it was discovered in the context of the controller's reconcile failures and affects the same recovery workflow.

Environment
  • Temporal CLI: latest (as of 2026-05-27)
  • Temporal server: self-hosted (Cassandra-backed)
  • Controller image: v1.5.x
What happened

After the controller's plan executor bug left a Worker Deployment in a corrupted state (see: plan executor delete+update bug), we attempted manual cleanup:

$ temporal worker deployment set-ramping-version --deployment-name '...' --build-id '17.233.1-497c' --delete --yes

Successfully set the ramping worker deployment version

$ temporal worker deployment describe --name '...' --namespace account

RampingVersionBuildID: 17.233.1-497c    # unchanged
RampingVersionPercentage: 50            # unchanged
RampingVersionPercentageChangedTime: 4 days ago  # unchanged

This pattern reproduced across multiple operations:

  • set-ramping-version --delete → "Successfully set" → no change
  • set-ramping-version --percentage 0 → "Successfully set" → no change
  • set-current-version --unversioned → "Successfully set" → no change
  • delete-version → "Successfully deleted" → version still present in describe

The deployment had no active pollers (workers had been sunset). The --allow-no-pollers and --ignore-missing-task-queues flags were used.

The system workflow (temporal-sys-worker-deployment:*) existed in the namespace and was reachable via temporal workflow describe, but its internal state was divergent from what deployment describe reported.

Recovery

Deleted the backing system workflow directly:

temporal workflow delete 
  --workflow-id 'temporal-sys-worker-deployment:'
  --namespace account

This scrubbed the deployment from worker deployment list and allowed clean re-registration.

Expected behavior

Operations should either persist the change or return an error. Returning success without mutation is dangerous during incident recovery.

Lenguaje dominante
Go
Estrellas
198
Forks
84
Merge medio
2 d 19 h
PR fusionados (30 d)
34

Preparar el entorno

  • Incluye un Dockerfile o un archivo de Docker Compose
  • Sin plantilla de pull request
  • Sin guía de contribución

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de temporalio/temporal-worker-controller

Todos los issues de temporalio/temporal-worker-controller

Issues similares

Más issues de Go

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.