[Bug] Local activity resolutions regrouped on replay since 1.32.0, delivering the wrong payload

Abierto
#1,881 1 comentario 0 reacciones 0 asignados Ver en GitHub

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
4/5
Tiempo estimado
3-5 días
Aptitud para principiantes
52/100
Tipo de issue
Error
Claridad
Bastante claro
Estado de actividad
Activo
Stack tecnológico
python

Línea de trabajo

Comienza con las rutas de resolución y programación nombradas en temporalio/worker/_workflow_instance.py, especialmente _apply_resolve_activity y _outbound_schedule_activity, y luego ejecuta la reproducción mínima con las versiones de 1.31.0 a 1.33.0. Rastrea la agrupación de activaciones en ejecución en vivo y en replay, y verifica que cada actividad local reciba su payload registrado y que los historiales se reproduzcan correctamente.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

What are you really trying to do?

Replaying histories from a workflow that starts several local activities, waits with return_when=FIRST_COMPLETED, and runs a further local activity while others are still pending. This replayed clean on 1.31.0 and stopped on 1.32.0.

Describe the bug

On 1.32.0 and 1.33.0, a history fails to replay under the same SDK version that recorded it. A local activity's recorded result is delivered to a different local activity's handle, and the payload converter raises on the type mismatch:

RuntimeError: Failed decoding arguments
  caused by TypeError: Expected value to be str, was <class 'bool'>
    temporalio/worker/_workflow_instance.py:900    _apply_resolve_activity
    temporalio/worker/_workflow_instance.py:2358   _convert_payloads
    temporalio/converter/_payload_converter.py:871 value_to_type

Live and replay group the resolutions into different activations. Tracing activate, _outbound_schedule_activity and _apply_resolve_activity on 1.33.0:

live   ACTIVATE [('resolve_activity', 1)]
live   RESOLVE 1 work b'"0"'
live   SCHEDULE 4 expired
live   ACTIVATE [('resolve_activity', 2), ('resolve_activity', 3)]
live   RESOLVE 2 work b'"1"'
live   RESOLVE 3 work b'"2"'
live   ACTIVATE [('resolve_activity', 4)]
live   RESOLVE 4 expired b'false'
live   SCHEDULE 5 finish

replay ACTIVATE [('resolve_activity', 1), ('resolve_activity', 2), ('resolve_activity', 3)]
replay RESOLVE 1 work b'"0"'
replay RESOLVE 2 work b'"1"'
replay RESOLVE 3 work b'"2"'
replay SCHEDULE 4 finish
replay RESOLVE 4 finish b'false'

Live resolves seq 1 on its own, so the expired activity takes seq 4 and finish takes seq 5. Replay resolves 1, 2 and 3 in a single activation, so finish takes seq 4 and consumes the bool that was recorded for expired.

There is no custom payload converter, no interceptor, no sandbox and no data converter involved. value_to_type is byte-identical between 1.31.0 and 1.33.0 (sha256 32dfc9701260bb70a15f3a83442e065aaacffde874a0335487b0d2e48361a74e), so this is resolution delivery rather than conversion.

Minimal Reproduction
import asyncio
import sys
import uuid
from datetime import timedelta
from pathlib import Path
import temporalio
from temporalio import activity, workflow
from temporalio.client import WorkflowHistory
from temporalio.testing import WorkflowEnvironment
from temporalio.worker import Replayer, UnsandboxedWorkflowRunner, Worker

@activity.defn
async def work(index: int) -> str:
    return str(index)

@activity.defn
async def expired() -> bool:
    return False

@activity.defn
async def finish() -> str:
    return 'finished'

@workflow.defn
class Fanout:
    @workflow.run
    async def run(self) -> str:
        pending = {workflow.start_local_activity(work, i, start_to_close_timeout=timedelta(seconds=10))
                   for i in range(3)}
        while pending:
            done, pending = await workflow.wait(pending, return_when=asyncio.FIRST_COMPLETED)
            for task in done:
                await task
            if pending:
                await workflow.execute_local_activity(expired, start_to_close_timeout=timedelta(seconds=10))
        return await workflow.execute_local_activity(finish, start_to_close_timeout=timedelta(seconds=10))

async def main():
    print('SDK', temporalio.__version__, flush=True)
    replayer = Replayer(workflows=[Fanout], workflow_runner=UnsandboxedWorkflowRunner())
    if len(sys.argv) > 1:
        for filename in sys.argv[1:]:
            try:
                await replayer.replay_workflow(WorkflowHistory.from_json('minimal', Path(filename).read_text()))
                print('MINIMAL_CROSS', filename, 'PASS', flush=True)
            except Exception as error:
                print('MINIMAL_CROSS', filename, 'FAIL', type(error).__name__, str(error), flush=True)
        return
    async with await WorkflowEnvironment.start_time_skipping() as server:
        async with Worker(server.client, task_queue='minimal', workflows=[Fanout],
                          activities=[work, expired, finish], workflow_runner=UnsandboxedWorkflowRunner()):
            for trial in range(10):
                handle = await server.client.start_workflow(Fanout.run, id=str(uuid.uuid4()), task_queue='minimal')
                result = await handle.result()
                history = await handle.fetch_history()
                try:
                    await replayer.replay_workflow(history)
                    print('MINIMAL', trial, 'LIVE', result, 'REPLAY PASS', flush=True)
                except Exception as error:
                    print('MINIMAL', trial, 'LIVE', result, 'REPLAY FAIL', type(error).__name__, str(error), flush=True)

if __name__ == '__main__':
    asyncio.run(main())

Run it with no arguments. It starts a time-skipping environment, runs the workflow 10 times, and replays each history immediately after the live run finishes.

Environment/Versions
1.31.0   10 REPLAY PASS,  0 REPLAY FAIL
1.32.0    0 REPLAY PASS, 10 REPLAY FAIL
1.33.0    0 REPLAY PASS, 10 REPLAY FAIL

Python 3.11 on Linux, time-skipping WorkflowEnvironment, UnsandboxedWorkflowRunner. Each version was run in its own virtualenv with nothing else installed beyond temporalio and its dependencies.

Additional context

The bisect lands on 1.32.0, which carries sdk-rust#1442, "Allow LAs to wake up workflow while another is still executing", merged 2026-08-04. That PR describes replay as preserving the live grouping. The trace above looks like a counterexample to that.

This is related to #1578 but points the other way. The shape in that issue never replayed clean, while this one replayed clean on 1.31.0 and regressed on the release carrying the change, and the symptom is a misrouted payload rather than a sequence mismatch.

Histories recorded on 1.32.0 and 1.33.0 also fail replay under the SDK version that recorded them, so this is not only a concern for histories written before an upgrade.

Lenguaje dominante
Python
Estrellas
1.2k
Forks
241
Merge medio
3 d 2 h
PR fusionados (30 d)
49

Guía de contribución

Abrir la guía de contribución

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de temporalio/sdk-python

Todos los issues de temporalio/sdk-python

Issues similares

Más issues de Python

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.