[Bug] Local activity resolutions regrouped on replay since 1.32.0, delivering the wrong payload
Nadie ha tomado este issue todavía.
Evaluación
- Dificultad
- 4/5
- Tiempo estimado
- 3-5 días
- Aptitud para principiantes
- 52/100
- Tipo de issue
- Error
- Claridad
- Bastante claro
- Estado de actividad
- Activo
- Stack tecnológico
- python
- Área
- backend, distributed-systems, testing
Línea de trabajo
Comienza con las rutas de resolución y programación nombradas en temporalio/worker/_workflow_instance.py, especialmente _apply_resolve_activity y _outbound_schedule_activity, y luego ejecuta la reproducción mínima con las versiones de 1.31.0 a 1.33.0. Rastrea la agrupación de activaciones en ejecución en vivo y en replay, y verifica que cada actividad local reciba su payload registrado y que los historiales se reproduzcan correctamente.
Escrito por el modelo de indexación a partir del texto del issue.
Descripción
What are you really trying to do?
Replaying histories from a workflow that starts several local activities, waits with return_when=FIRST_COMPLETED, and runs a further local activity while others are still pending. This replayed clean on 1.31.0 and stopped on 1.32.0.
Describe the bug
On 1.32.0 and 1.33.0, a history fails to replay under the same SDK version that recorded it. A local activity's recorded result is delivered to a different local activity's handle, and the payload converter raises on the type mismatch:
RuntimeError: Failed decoding arguments
caused by TypeError: Expected value to be str, was <class 'bool'>
temporalio/worker/_workflow_instance.py:900 _apply_resolve_activity
temporalio/worker/_workflow_instance.py:2358 _convert_payloads
temporalio/converter/_payload_converter.py:871 value_to_type
Live and replay group the resolutions into different activations. Tracing activate, _outbound_schedule_activity and _apply_resolve_activity on 1.33.0:
live ACTIVATE [('resolve_activity', 1)]
live RESOLVE 1 work b'"0"'
live SCHEDULE 4 expired
live ACTIVATE [('resolve_activity', 2), ('resolve_activity', 3)]
live RESOLVE 2 work b'"1"'
live RESOLVE 3 work b'"2"'
live ACTIVATE [('resolve_activity', 4)]
live RESOLVE 4 expired b'false'
live SCHEDULE 5 finish
replay ACTIVATE [('resolve_activity', 1), ('resolve_activity', 2), ('resolve_activity', 3)]
replay RESOLVE 1 work b'"0"'
replay RESOLVE 2 work b'"1"'
replay RESOLVE 3 work b'"2"'
replay SCHEDULE 4 finish
replay RESOLVE 4 finish b'false'
Live resolves seq 1 on its own, so the expired activity takes seq 4 and finish takes seq 5. Replay resolves 1, 2 and 3 in a single activation, so finish takes seq 4 and consumes the bool that was recorded for expired.
There is no custom payload converter, no interceptor, no sandbox and no data converter involved. value_to_type is byte-identical between 1.31.0 and 1.33.0 (sha256 32dfc9701260bb70a15f3a83442e065aaacffde874a0335487b0d2e48361a74e), so this is resolution delivery rather than conversion.
Minimal Reproduction
import asyncio
import sys
import uuid
from datetime import timedelta
from pathlib import Path
import temporalio
from temporalio import activity, workflow
from temporalio.client import WorkflowHistory
from temporalio.testing import WorkflowEnvironment
from temporalio.worker import Replayer, UnsandboxedWorkflowRunner, Worker
@activity.defn
async def work(index: int) -> str:
return str(index)
@activity.defn
async def expired() -> bool:
return False
@activity.defn
async def finish() -> str:
return 'finished'
@workflow.defn
class Fanout:
@workflow.run
async def run(self) -> str:
pending = {workflow.start_local_activity(work, i, start_to_close_timeout=timedelta(seconds=10))
for i in range(3)}
while pending:
done, pending = await workflow.wait(pending, return_when=asyncio.FIRST_COMPLETED)
for task in done:
await task
if pending:
await workflow.execute_local_activity(expired, start_to_close_timeout=timedelta(seconds=10))
return await workflow.execute_local_activity(finish, start_to_close_timeout=timedelta(seconds=10))
async def main():
print('SDK', temporalio.__version__, flush=True)
replayer = Replayer(workflows=[Fanout], workflow_runner=UnsandboxedWorkflowRunner())
if len(sys.argv) > 1:
for filename in sys.argv[1:]:
try:
await replayer.replay_workflow(WorkflowHistory.from_json('minimal', Path(filename).read_text()))
print('MINIMAL_CROSS', filename, 'PASS', flush=True)
except Exception as error:
print('MINIMAL_CROSS', filename, 'FAIL', type(error).__name__, str(error), flush=True)
return
async with await WorkflowEnvironment.start_time_skipping() as server:
async with Worker(server.client, task_queue='minimal', workflows=[Fanout],
activities=[work, expired, finish], workflow_runner=UnsandboxedWorkflowRunner()):
for trial in range(10):
handle = await server.client.start_workflow(Fanout.run, id=str(uuid.uuid4()), task_queue='minimal')
result = await handle.result()
history = await handle.fetch_history()
try:
await replayer.replay_workflow(history)
print('MINIMAL', trial, 'LIVE', result, 'REPLAY PASS', flush=True)
except Exception as error:
print('MINIMAL', trial, 'LIVE', result, 'REPLAY FAIL', type(error).__name__, str(error), flush=True)
if __name__ == '__main__':
asyncio.run(main())
Run it with no arguments. It starts a time-skipping environment, runs the workflow 10 times, and replays each history immediately after the live run finishes.
Environment/Versions
1.31.0 10 REPLAY PASS, 0 REPLAY FAIL
1.32.0 0 REPLAY PASS, 10 REPLAY FAIL
1.33.0 0 REPLAY PASS, 10 REPLAY FAIL
Python 3.11 on Linux, time-skipping WorkflowEnvironment, UnsandboxedWorkflowRunner. Each version was run in its own virtualenv with nothing else installed beyond temporalio and its dependencies.
Additional context
The bisect lands on 1.32.0, which carries sdk-rust#1442, "Allow LAs to wake up workflow while another is still executing", merged 2026-08-04. That PR describes replay as preserving the live grouping. The trace above looks like a counterexample to that.
This is related to #1578 but points the other way. The shape in that issue never replayed clean, while this one replayed clean on 1.31.0 and regressed on the release carrying the change, and the symptom is a misrouted payload rather than a sequence mismatch.
Histories recorded on 1.32.0 and 1.33.0 also fail replay under the SDK version that recorded them, so this is not only a concern for histories written before an upgrade.
- Lenguaje dominante
- Python
- Estrellas
- 1.2k
- Forks
- 241
- Merge medio
- 3 d 2 h
- PR fusionados (30 d)
- 49
Guía de contribución
Primeros pasos
- Lee el issue completo y luego la guía de contribución del proyecto.
- Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
- Haz un fork del repositorio y trabaja en una rama.
- Abre un pull request que haga referencia al número del issue.
Más de temporalio/sdk-python
-
bug
Dificultad 2/5 1-3 horas Aptitud para principiantes 74/100
temporalio/sdk-python#1517 · 10 comentarios ·
-
bug
Dificultad 2/5 1-3 horas Aptitud para principiantes 68/100
temporalio/sdk-python#496 ·
-
Dificultad 5/5 Más de una semana Aptitud para principiantes 25/100
temporalio/sdk-python#1890 ·
-
bug
temporalio/sdk-python#1817 · 1 comentario · 1 asignado ·
-
Cloud CI Skips Nexus Tests Abierto
temporalio/sdk-python#1704 · 1 asignado ·
Todos los issues de temporalio/sdk-python
Issues similares
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 88/100
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 82/100
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 78/100
-
enhancement
Dificultad 2/5 1-3 horas Aptitud para principiantes 72/100
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 74/100