V2 workspace documents: fix public source discovery and verify metadata-analysis parity across scopes

Issue

Public documents visible in the V2 workspace cannot reliably be located in the chat Documents picker. The user also reports inconsistent manual metadata analysis across personal, group, and public workspaces, and missing automatic public-document metadata such as title, authors and abstract/summary. Metadata analysis is distinct from content ingestion.

Fix shared product behavior across all three workspace types, rather than special-casing one document.

Priority: P1. Requested assignee: @paullizer. Roadmap size: L.

Steps to Reproduce

1. Upload a Markdown document to an available public workspace and wait for content processing to complete.
2. Ensure several newer unrelated public documents exist, placing the target beyond the first six rows.
3. Open a new V2 chat, open Documents, and search for the target's filename or title.
4. Select only the target and ask a grounded question; verify the answer cites its actual passage.
5. Independently check automatic and manual metadata analysis for personal, group and public documents using the configured metadata model.

Expected Behavior

• Public source discovery honors filename/title search, sorting and pagination, with consistent visibility and availability rules.
• Explicit document selections retain the correct scope through chat and orchestration; unrelated documents are never substituted.
• When enabled and authorized, metadata analysis reaches a truthful terminal outcome and persists supported fields across all scopes.
• Confirmed behavior: if metadata analysis fails but indexing succeeds, keep the document searchable, display a separate metadata warning/failure, and permit an authorized retry.
• A missing generated title must not make a searchable filename disappear.

Actual Behavior

• A filename query leaves unrelated public documents listed; the desired public document is unavailable in the picker.
• The affected public Markdown document appears Ready but has the title Untitled Document.
• The observed interaction stopped before a chat prompt was submitted. A downstream retrieval or model failure is therefore not established by that interaction.

Impact

Users cannot reliably ground chat on public guidance. Metadata state is unclear, and successful content processing is easily mistaken for successful metadata generation.

Notes

Confirmed root cause and evidence

• application/v2_ui/src/lib/contextMentions.ts sends a search term, page 1 and a six-row page size.
• application/single_app/route_backend_public_documents.py::api_list_public_workspace_documents ignores search, page number and requested sort, then slices the unfiltered results to page size.
• A local execution of the real route/helper with eight synthetic documents reproduced the defect on both document-access-index and source-fallback paths: searching an older matching filename returned six unrelated newer documents; page 2 repeated page 1; no total count was returned.
• ui_tests/test_v2_chat_context_selection.py currently filters public rows in its fixture even though the real endpoint does not. This application regression fixture masks the API mismatch and must be aligned with the real contract.
• Default visibility resolution differs from the canonical public visibility fallback. Cover absent, legacy and explicit preferences without silently widening scope.
• Application Insights records the affected filename-search request at 2026-10-18 15:18:48 UTC returning HTTP 200. App Service logs record successful processing of the public Markdown document into one chunk on 2026-10-17 at 21:28:22 UTC. Chunk ingestion is not proof of metadata or answer success.

Metadata investigation status

• V2 group/public management uses native /api/groups/<group_id>/documents/... and /api/public-workspaces/<workspace_id>/documents/... routes.
• Source tracing confirms the native jobs preserve workspace scope. The common upload processor reads enable_extract_meta_data at execution time. Markdown deliberately suppresses its local metadata pass and relies on the final shared pass; neither this flag nor safe_errors=True alone proves a defect.
• Historical startup telemetry shows metadata extraction enabled and the legacy model setting gpt-4o; it does not prove the effective job-time model selection or a successful model call.
• The specific live metadata root cause remains unconfirmed. The current Azure principal can read telemetry but Cosmos data-plane access to the document/processing record returns 403. No permissions were changed or bypassed.
• Do not substitute guesses about adapters, model parameters or indexing for an executable reproduction and processing-stage evidence.

Validation so far

• python -B -m pytest -p no:cacheprovider -q functional_tests\test_public_chat_document_list_pending_artifacts.py functional_tests\test_v2_public_chat_scope.py functional_tests\test_v2_chat_context_picker.py — 49 passed, 11 existing return-value warnings.
• python -B -m pytest -p no:cacheprovider -q --disable-warnings --tb=short --maxfail=1 functional_tests\test_document_auto_metadata_extraction_consistency.py functional_tests\test_multiselect_metadata_extraction.py — failed at an existing structural assertion expecting process_md directly in the upload dispatcher. This does not establish a runtime metadata defect.
• python -B -m pytest -p no:cacheprovider -q --disable-warnings --tb=short --maxfail=1 functional_tests\test_public_document_management.py functional_tests\test_group_document_management.py — 1 passed, then setup error: the local Flask test client expects werkzeug.__version__. Re-run with the repository-pinned Flask/Werkzeug dependencies in an isolated test environment.
• No product code change, deployment, re-extraction or live chat submission has occurred.

Acceptance criteria

• Public listing filters before pagination, honors supported ordering, and reports accurate filtered totals on both storage read paths while preserving classic callers.
• Document discovery matches title and filename independently, including documents beyond the initial page and documents without generated metadata.
• Canonical visibility, feature gates, workspace status, document publication/review restrictions and selected-source authorization remain enforced.
• Shared picker and mention-menu errors are visible without discarding successful scopes or implying authorization denial from an empty result.
• Execute native automatic/manual metadata regression paths across all three scopes, including Markdown, model selection, persistence, retry and terminal outcomes; fix demonstrated shared defects.
• Indexed content remains searchable when metadata fails, with a separate safe warning and authorized retry.
• Selected public document/workspace IDs reach both chat and orchestration correctly; retrieval and citations are verified separately.
• Application UI fixtures and endpoint contract tests agree, and new regressions fail against the pre-fix behavior.
• Update related product documentation and increment only the application patch version with implementation. Deployment and live state-changing validation remain separately approved.

Related issues and boundaries

Closed #501 reported older public metadata symptoms. Closed #1256 addressed classic picker filename matching. Neither is an open duplicate of this V2 route-contract defect. #1542 covers broader shared-workspace work.

#1522 concerns sharing documents between public workspaces and an index-schema migration. That separate feature is out of scope.

Roadmap P1 and size L are saved. Repository label/assignee mutations were denied by the current credentials, so the requested assignment and repository P1 label still need an authorized maintainer.