Complete the orchestration harness with Gather, Reason, Render and explicit file creation

Start by reading functions_orchestration_registry.py, functions_orchestration_schema.py, functions_saved_analysis.py, and functions_generated_file_exports.py, then trace RunContext and the executor assumptions. Review existing standalone Analyze, Compare, tabular, export, and workflow tests before choosing one milestone-sized slice. Done means the selected contract is covered by focused tests without breaking legacy behavior.

Summary

Complete SimpleChat's orchestration harness so reusable knowledge and reasoning results can feed explicit, independently recoverable file-creation tasks through the existing Generated File Export Framework.

Use Gather / Reason / Render as three capability roles, with dependency-driven execution, not a mandatory one-way sequence. Analyze, Compare, tabular computation, synthesis, and content drafting belong under Reason. A request may gather, reason, gather additional evidence, reason again, and render without inventing a fourth mandatory analysis phase.

Planning baseline: application version 0.261.124 in application/single_app/config.py.

Scope: orchestration only. Workflow integration, image/audio generation, cross-conversation memory, and new external delivery destinations are future work. Existing standalone Analyze, Compare, tabular, chat export, and workflow behavior must remain compatible.

User Value

Users should be able to ask for an analysis and several deliverables in one approved orchestration plan: for example, compare source documents, derive a structured findings dataset, draft a report, and create CSV, Word, PDF, or PowerPoint files from those retained results. Generating another format or retrying a failed file must not repeat the underlying analysis.

The harness must distinguish completed internal work from a completed downloadable file. It should deliver successful files even if another requested output fails, show exactly what remains incomplete, automatically retry eligible output failures, and offer manual retry after automatic attempts are exhausted.

Proposed Behavior

Confirmed product decisions

• Gather / Reason / Render are the visible role labels; progress labels can be Gathering / Reasoning / Rendering.
• Roles describe a task's purpose. Explicit, validated data dependencies determine execution order; the system must not drop a valid dependency simply because it points between different roles.
• Gather and Reason retain internal reusable results. They must not implicitly publish SimpleChat-generated downloadable files in the new orchestration contract.
• Internal storage blobs, complete datasets, evidence, drafts, and checkpoints are legitimate retained state; they are not user-facing exported files.
• Explicit Render tasks create requested files through one export registry/service, using authorized result readers or prepared content. Renderers do not reanalyze sources or hide new model reasoning inside serialization.
• Initial formats: CSV, XLSX, DOCX, PPTX, PDF, JSON, XML, YAML/YML, Markdown, and TXT. Supported upload extensions must not be mistaken for supported output renderers; YAML needs an explicit output adapter.
• Multiple files may use the same immutable result without rerunning its producer. Narrative reports and slide content can have distinct explicit Reason tasks when composition is actually needed.
• Successful outputs remain available when another output fails. A file is individually committed only after complete bytes and current authorization are verified.
• Each retryable failed file receives 3 total attempts: the initial attempt plus 2 automatic retries. After exhaustion, expose manual retry of that file only. Invalid input, unsupported formats, denied access, cancellation, and deterministic non-retryable failures do not receive blind retries.
• Preserve standalone tools' current file behavior, including their existing Analyze artifact contracts. Apply the internal-result policy only to the new orchestration execution contract.
• Preserve the existing memory and storage foundations. Do not add a new global memory product or workflow feature as a prerequisite.

Current implementation seams

• functions_orchestration_registry.py declares knowledge/reasoning/output but has no output-phase capability. Analyze, Compare, and tabular analysis currently occupy knowledge.
• functions_orchestration_schema.py globally orders phases, repairs away backward dependencies, and forces one final respond; the executor and React UI embed related assumptions.
• RunContext, private checkpoints, execution leases, and saved Analyze readers already provide substantial durability. Extend these rather than start another execution or memory engine.
• functions_saved_analysis.py already persists authorized orchestration results through the shared result store and reads complete saved records without relying on preview text.
• Compare currently returns useful comparison items/coverage but its orchestration adapter principally retains narrative evidence, not an equivalent general reusable result contract.
• Tabular orchestration still depends on generated-output metadata/publication-oriented native run readers. Its adapter needs a real compute-only foreground/durable execution boundary and explicit waiting/result readiness.
• run_respond currently combines synthesis with special saved-Analyze formatting. Separate content preparation, file rendering, and terminal message publication.
• functions_generated_file_exports.py has an explicit saved-record source/request interface, currently JSON-only for its exact-record profile, plus broader legacy response-based exports. Existing Word/PDF/PowerPoint and other renderer paths should be adapted, not reimplemented as an orchestration-only export stack.

Phased delivery

Each milestone includes focused tests and directly related documentation. The first user-facing release requires an end-to-end slice with durable output/retry behavior, not just a registry entry. Independent renderer adapters can be developed in parallel after their shared contracts are stable; central schema/executor integration remains coordinated.

Acceptance Criteria

• A plain orchestration analysis retains usable internal results and creates zero user-downloadable files unless the approved plan includes Render tasks. Test the storage/publication side effects, not merely whether the UI hides cards.
• Analyze, Compare, and native tabular results expose complete authoritative data or an honest partial/pending/unsupported state, with source lineage and coverage.
• Valid acyclic dependency chains may interleave Gather and Reason; cycles, missing required inputs, unauthorized references, and incompatible result kinds are rejected rather than repaired into different work.
• No plan is forced to include every role. Text-only requests do not acquire an additional reasoning/model round trip merely because rendering was separated.
• Explicit Render tasks support the ten document/data formats above with truthful source-kind compatibility and no silent format fallback.
• One retained dataset can produce multiple structured formats with preserved schema, ordering, row counts, values, and documented format-specific escaping. Internal lineage is not accidentally included in public columns.
• Word/PDF share prepared report content when appropriate; PowerPoint consumes a prepared slide representation rather than triggering hidden analysis in its renderer.
• A renderer never promotes a bounded preview, first page, or source-row passthrough into an allegedly complete derived result.
• A source-free content request can use an explicit Reason task and the same renderer without fabricating document provenance.
• Rendering/retrying does not invoke the original Analyze/Compare/tabular producers again.
• Successful files are delivered independently; every failed or waiting output is clearly represented, and run success does not follow merely from a successful chat answer.
• Retryable output failures make at most three automatic-cycle attempts, then expose manual retry of only the failed file. Attempt counts and retry scheduling survive restart and duplicate requests.
• Output publication is idempotent, individually atomic from the reader's perspective, and fenced against canceled, deleted, superseded, or stale workers.
• Current ownership, source access, screening state, and relevant revision binding are checked before consumption/publication/download/retry. Raw settings, credentials, private result blobs, and diagnostics are not exposed to the browser.
• Pending native work is not reported as a completed result or synthesized from an incomplete preview. Orchestration can resume dependent work once a validated result is ready.
• V2 shows real dependency/execution order rather than regrouping a dependency-driven plan into misleading phase buckets. Output progress, downloads, and retry state survive reload without duplication.
• Existing standalone tools, workflow behavior, legacy stored plans/checkpoints, and existing generated-file download/promotion paths remain compatible.
• Functional, route-policy, browser, format-readback, restart/race, and deterministic large-record tests cover the full path. Paid/provider-dependent validation requires separately authorized environment access.
• Related feature/admin/user docs and generated app-surface inventory are updated. Each implementation change bumps only the application version's third segment and records that version in new tests/docs; planning alone does not bump it.

Notes

• Refs #1021: broad turn-level orchestration initiative. This issue is the scoped reusable-result and explicit document/data rendering delivery tranche; it does not reopen every objective of the umbrella issue.
• Refs #1071: shared generated-file capability. Reuse and extend its framework rather than build a competing renderer stack.
• Refs #1233: standalone Analyze deliverable behavior remains intact. The new internal-result policy is explicitly orchestration-scoped, not a global removal of Markdown-first standalone artifacts.
• Refs #1031: preserve full-source tabular coverage and existing scalable execution boundaries.
• Refs #1461: new lower-level result/render services must not introduce route/config/bootstrap dependency cycles. This is not a repo-wide import refactor.

Future workflow integration should consume the same task-result/export contracts and reuse the existing workflow binding language, not automatically add a planning model call to every workflow task. Image/audio generation, cross-conversation knowledge reuse, external publication/delivery, and an unrestricted self-modifying agent loop are not in this implementation scope.

Roadmap triage requested by the admin: Priority P1, Size XL, Assignee @paullizer. Implementation has not started.