V2: triage multi-turn UI failures and make saved-memory orchestration validation reliable

Issue

Eight live-UI acceptance scenarios (FED-31–38, 2026-10-19) failed at the first turn, before their planned 10/20-turn persistence, memory-update, source-reuse, and reload checks could run. These are end-to-end UI tests, not unit tests. The evidence separates six transient-message selector failures, one response-wording oracle failure, and one unresolved saved-memory recall/clarification symptom.

Investigate and fix shared application behavior where a defect is demonstrated; do not hardcode the scenario, force incidental answer wording, preserve temporary message IDs, or treat all eight cases as a proven long-conversation defect. The admin requested that harness findings be documented separately so the harness owner can fix them.

Priority: P1. Assignee: @paullizer. Planning/investigation only; no application changes or deployment yet.

Steps to Reproduce

1. Use the V2 chat UI at https://simplechat-20260506-app.azurewebsites.net/v2 with the scenario's manual/orchestrated mode. Orchestrated reports expose Review mode; screenshots show the answer-model footer gpt-5.6-terra.
2. For memory-enabled scenarios, create and verify scenario-scoped instruction/fact text in User Settings. Ask for the usual review name/role without supplying it in the prompt.
3. Submit a baseline request containing the review identifier, training-program reference, $200,000 budget, amber readiness, and Rowan as mission owner.
4. Observe the user-message persistence transition and resulting answer/clarification. Continue the intended 10/20-turn sequence only after the observation harness correctly verifies that turn.

Report correlation

UTC on 2026-10-19; supplied screenshots supplement reports that stopped collecting evidence early.

Case	Planned turns / mode	Run window	Conversation ID	Observed stop
FED-31	10 / mixed	17:52:14–17:52:59	e2e0046d-8cc8-4ad4-b050-9a66ce525ff9	Pending-user locator missing; screenshot shows baseline answer
FED-32	20 / mixed	17:53:00–17:53:44	681ff2e8-5de9-43ce-86b1-756388bfedfe	Same locator failure; screenshot shows baseline answer
FED-33	10 / orchestrated	18:51:21–18:52:24	e9e2dcd3-4b79-469b-854a-e0ef401e9453	Literal Civic Bridge missing from otherwise fact-preserving summary
FED-34	20 / orchestrated	17:54:19–17:55:16	2696fb93-1a61-4231-a113-f7f1a979454d	Pending-user locator missing; screenshot shows baseline answer
FED-35	10 / manual	17:55:16–17:56:09	bd3ffe48-091a-4db0-9467-1c92f79a355a	Same locator failure; screenshot shows saved prefix/headings
FED-36	10 / mixed	17:56:10–17:57:02	22b26032-099f-4e60-88f6-55cfc0203d6c	Same locator failure; screenshot addresses Avery, readiness reviewer
FED-37	20 / orchestrated	18:37:25–18:38:17	4c09dd1b-2700-4153-9a65-72b0b08150c2	Planner asks for saved review name/role
FED-38	10 / mixed	17:57:57–17:58:51	2f2f3db7-5765-4a95-9d0b-b2557dddd710	Pending-user locator missing; screenshot shows prefix/headings and saved identity

Expected Behavior

• The test follows one logical user turn through optimistic rendering, server persistence, assistant completion, and reload without requiring a transient DOM ID to remain present.
• The current turn's answer is discovered independently of incidental words and optional reasoning controls. Required facts, explicit formatting instructions, identity relationships, citations, and obsolete-value exclusions remain strict acceptance checks.
• Authorized, relevant saved context is available at the stages that decide whether clarification is needed and produce the answer. Missing/ambiguous context remains a legitimate reason to ask, and memory never grants permissions or overrides current user/system constraints.
• Multi-turn success is claimed only after the actual sequence, mode transitions, memory update, provenance checks, and reload assertions complete.

Actual Behavior

Harness H1: transient user-message identity — six cases

Reported stack: ui_tests/federal/long_conversation_models.py:354 (execute_turn), called by execute_long_conversation:376, workflows.py:85, and live_evidence_plugin.py:173.

Locator expected to have count '1'
Actual value: 0
Expect "to_have_count" with timeout 5000ms
[data-tour="message-list"] [id="message-pending-user-<timestamp>"]

In current application source, chatStore.ts intentionally replaces the optimistic pending-user-* ID when onUserMessagePersisted receives the server ID; MessageList.tsx renders its wrapper from that ID. Retaining a locator keyed to the old ID is incompatible with normal persistence. Do not regress this behavior to satisfy the harness.

Harness-owner handoff: re-resolve the current message/turn after save; separate submitted, persisted, answered, and reloaded states. A fast acknowledgement may replace the optimistic bubble before the harness observes it. Scope to the conversation and current turn, handling duplicate prompt text and avoiding accidental acceptance of prior messages. If the suite is visible-UI-only, do not substitute hidden store inspection or direct API calls for UI acceptance. Do not fix this with arbitrary sleeps or a weakened duplicate-message assertion.

Harness H2: phrase/answer-discovery coupling — FED-33

Reported stack: long_conversation_models.py:348 → page_models.py:390 (facts). The locator selects the first message wrapper containing an exact Show reasoning button, then expects Civic Bridge.

Actual text: Review CB-27-372117 covers training brief FED-PROGRAM-027. The stated budget is $200,000, readiness is amber, and Rowan is the mission owner.

The required supplied facts are present. Omitting an incidental display phrase does not demonstrate failed memory or conversation handling. If that phrase is genuinely required, make the contract explicit and classify it as output noncompliance rather than a transport/memory failure.

Harness-owner handoff: discover the current answer independently from its words and optional reasoning button; assert facts/relationships separately. Keep explicit saved prefix/heading and identity contracts strict. Test paraphrases, no reasoning button, multiple prior replies, repeated prompts, wrong facts and missing facts. The current .first strategy also poses an unexercised later-turn wrong-message risk.

Application investigation A1: unnecessary saved-identity clarification — FED-37

The UI asks What review name and role should the brief be addressed to? after the harness reports creating/verifying the relevant saved profile entries. Reported stack: long_conversation_models.py:339 → review:256 → page_models.py:503 (wait_for_review) → answer_clarification:449; failed check plan.synthetic_clarification_answered.

This remains a meaningful recall symptom, not a reason to silently auto-answer and mark recall passed. Determine whether the record was unavailable, omitted by retrieval, unavailable at an earlier decision stage, or supplied but ignored. Settings visibility alone cannot distinguish those causes.

Current source investigation seams, not established causes of this particular run:

• route_backend_orchestration.py invokes resolve_conversation_request before load_orchestration_memory; a resolver clarification can return before memory loading. A first turn without history normally shortcuts that resolver, so stage correlation is essential.
• The main planner does receive memory through build_planner_context; compose reloads it at answer time. A generic claim that orchestration never loads memory is incorrect.
• functions_fact_memory_context.py selects the newest 8 instruction memories and top 4 cosine-ranked facts. Read-only planning omits unembedded/incompatible-profile facts and must not backfill or write them.
• semantic_kernel_fact_memory_store.py intentionally recalls across conversations in the authorized user/group scope. Unique scenario aliases/review markers do not establish separate storage scopes. The supplied profile screenshot contains 10 instructions and 10 facts, creating fixture-interference/selection-pressure risk, not proof of contamination.

Impact

• Six scenarios falsely stop observing a normal persistence transition.
• One stops on incidental response wording; later context checks remain unexecuted.
• One exposes a possible real memory/planning defect requiring correlated evidence.
• Test reports cannot currently distinguish these causes or substantiate the intended long-conversation guarantees.

Notes

Acceptance criteria

• Deliver the separate harness handoff with exact failing helper/line, evidence, repair guidance and regression cases. The submitted harness lives in another worktree; its ui_tests/federal/ implementation is absent from this checkout, so referenced lines are from reports, not a source review of that worktree.
• Correlate the listed windows/IDs against App Insights and App Service logs; record deployed version/image/commit where verifiable and explicitly state telemetry gaps. Do not infer a deployment version from local source.
• Reproduce any application context defect with deterministic tests of shared memory, request resolution, planner and answer boundaries before choosing its fix.
• Preserve authorized scope, shared-conversation isolation, disabled behavior, read-only planning, current-user overrides, bounded context and embedding-profile compatibility.
• Distinguish memory available/selected/omitted/unavailable without logging raw memory, credentials or prompts. Do not hardcode Civic Bridge, review IDs or identities in product logic.
• Test missing memory, competing near-duplicates, capacity pressure, unavailable/incompatible embeddings, corrections, mode transitions and reload. Do not suppress valid clarifications or treat fixture cleanup as the product fix.
• Harness reports independently record planned/submitted/answered/verified/reloaded turn counts, application build, harness commit, model selection and correlation IDs. Unexecuted checks are not run, not proof of a product defect.
• Harness fixtures use a dedicated test profile or approved run-owned lifecycle; never delete unrelated user memories.
• After harness repairs and any proven app fix, rerun all eight complete sequences with Sources/provenance and FED-38 update/stale-value checks.
• Application code changes, if approved, include focused regression tests, fix documentation and the third-segment version bump in application/single_app/config.py. Documentation/planning alone does not bump version.

Local validation so far

Source baseline: 0.261.348, commit 7c8a3177eff4ab73332738985cdb7cd7ec578c93.

• python -m pytest -q functional_tests\test_fact_memory_read_only_context.py functional_tests\test_fact_memory_streaming_context_fix.py functional_tests\test_fact_memory_streaming_retrieval_fix.py — 17 passed, 5 subtests passed, existing warnings.
• python -m pytest -q functional_tests\test_message_metadata_loading_fix.py — 4 passed.
• python -m pytest -q functional_tests\test_orchestration_memory_context.py functional_tests\test_orchestration_conversation_context.py functional_tests\test_message_metadata_loading_fix.py — collection blocked first by missing application import path; retry with application/functional-test PYTHONPATH reached a local Semantic Kernel → aiortc → pyOpenSSL failure: AttributeError: module 'lib' has no attribute 'GEN_EMAIL'. This is an environment blocker, not a failing product assertion. Do not modify a shared Python installation or production deployment to hide it.

Related, not duplicates: #1352 (memory availability/admin placement), #1153 (normal-chat memory creation), #1509 (broader Gather/Reason/Render orchestration). No exact duplicate found in memory, multi-turn, pending-user, scenario-name or FED-37 searches.