Feature Proposal: Lenient Replay Mode for Non-Determinism Handling

Chưa xác định được tệp, bài kiểm thử hoặc điểm vào nào. Hãy bắt đầu bằng cách lần theo cách framework xử lý replay và tính không xác định, sau đó xác định ngữ nghĩa cho việc cắt ngắn lịch sử, các nhánh song song, các orchestration con, bộ hẹn giờ, sự kiện bên ngoài và ContinueAsNew; để được xem là hoàn tất, cần có thiết kế đã được thống nhất, phạm vi triển khai và hành vi observability.

Feature Proposal: Lenient Replay Mode for Non-Determinism Handling

Current Behavior

When an orchestrator detects non-determinism during replay, it throws an exception and terminates execution. This is a sensible default—it enforces correctness and helps developers learn the determinism constraints early on.

The Problem

Maintaining determinism in durable orchestrations is non-trivial. Developers must carefully manage:

• Activity ordering
• Input/output formats
• Avoidance of non-deterministic APIs (e.g., DateTime.Now, Guid.NewGuid())

In practice, business processes evolve. Requirements change. When an orchestration definition must change while instances are in-flight, the current strict behavior forces developers into cumbersome workarounds: stopping running instances, implementing versioning schemes, or building custom "resume" logic to restart from a business-relevant checkpoint.

This friction is especially painful for long-running orchestrations where simply restarting from scratch isn't acceptable.

Proposed Solution

Introduce an alternative replay mode: Lenient.

Mode	Behavior on Non-Determinism
Strict (current default)	Throw exception, terminate execution
Lenient (proposed)	Truncate all events after the divergence point; execute the new activity as if it were freshly scheduled

When is Lenient Mode Appropriate?

Lenient mode offers an alternative execution model for scenarios where continuing execution is preferable to strict correctness enforcement. This trade-off makes sense when:

The cost of restarting is high

Long-running orchestrations (days, weeks, months) accumulate significant completed work. A minor change to the workflow definition—adding a logging activity, fixing a notification message, or introducing a new optional step—shouldn't invalidate weeks of progress. In strict mode, developers face an unpleasant choice: delay the change until all instances complete, or terminate instances and manually reconstruct state.

Activities are already designed for idempotency

If your activities are truly idempotent and don't depend on strict execution order relative to each other, the risks introduced by lenient mode are minimal. The framework already requires developers to design for at-least-once execution; lenient mode simply extends this to cover orchestration evolution.

You accept that posterior activities will re-execute

This is the core trade-off. When divergence is detected, all events after the divergence point are truncated—meaning those activities will execute again. This introduces a new constraint: any activity that may appear after a potential divergence point must be idempotent, or the business must tolerate duplicate execution.

This is a different mental model than strict mode. In strict mode, developers reason about "this activity runs exactly at this point in the workflow." In lenient mode, developers must reason about "this activity might run again if the workflow definition changes upstream."

Manual intervention is the alternative

In production incidents, the realistic alternative to lenient mode is often manual database manipulation—operators directly modifying orchestration state to "fix" stuck instances. Lenient mode provides a controlled, auditable path forward instead of ad-hoc state surgery.

Rapid iteration environments

In development or staging, strict mode's correctness guarantees may be less valuable than fast iteration cycles. Developers testing workflow changes shouldn't need to restart orchestrations from scratch on every code change.

When is Lenient Mode NOT Appropriate?

Lenient mode should be avoided when:

• Activities have strong ordering dependencies (e.g., "B must only run after A completes")
• Side effects are not idempotent (e.g., financial transactions, inventory decrements)
• Data consistency across activities is critical
• Regulatory or compliance requirements mandate strict execution guarantees

Trade-offs and Limitations

Lenient mode introduces certain limitations:

• No ordering guarantee after divergence: Activities scheduled after the divergence point may have already executed in a previous orchestration version. There is no guarantee that "later" activities will only execute after "earlier" ones.
• Potential re-execution of activities: Activities that were already completed may be scheduled and executed again.

These limitations are already familiar

It's worth noting that some of these trade-offs are inherent to the framework's design and developers must already account for them:

• At-least-once execution: The framework cannot guarantee that an activity will execute exactly once. Storage failures, transient errors, or replay issues can cause an activity to run multiple times. Developers are already expected to design activities with idempotency in mind.
• External side-effects are not transactional: Activities that interact with external systems (APIs, databases, etc.) operate outside the orchestration's durability guarantees.

Lenient mode extends this existing "design for resilience" mindset to cover orchestration evolution scenarios.

Observability Requirements

Since lenient mode relaxes correctness guarantees, verbose logging is essential. When divergence is detected and handled, the framework should emit detailed logs including:

• Timestamp of divergence detection
• Orchestration instance ID
• Expected event type/name vs. actual event type/name
• Position in history where divergence occurred
• Number of events truncated
• Full list of truncated event types (at DEBUG/TRACE level)

This logging allows operators to:

• Monitor how frequently divergence occurs in production
• Audit which instances were affected
• Debug unexpected behavior stemming from re-execution
• Make informed decisions about whether lenient mode is appropriate for their workload

Open Questions

Several edge cases need clarification before implementation:

• Parallel branches: If divergence occurs in one branch of a fan-out, what happens to sibling branches? Are their histories also truncated?
• Sub-orchestrations: How does divergence in a parent affect child orchestrations, and vice-versa?
• Pending timers: If a timer was scheduled before divergence but hasn't fired yet, should it be preserved or discarded?
• External events: What happens to external events that were received but appear after the divergence point in history?
• ContinueAsNew: Does lenient mode interact with ContinueAsNew semantics in any special way?