Azure DevOps as a documentation source: `living-doc-collector-ad`

This is an epic: start with the spike child, which needs an operator run against a real internal project and produces only a shape inventory. The collector skeleton is under work_items/ in living-doc-collector-ad (pinned to living-doc-utilities==0.8.0), and the HTML conversion lives in authoring.html_to_markdown. Done means the spike report confirms or removes each prepared contract field, and no newcomer-level task exists until that is finished.

Background

Azure DevOps is planned. In v0.1.0 it is not a documentation source. living-doc-collector-ad on master
(7a671f3) pins living-doc-utilities==0.8.0 and holds only the collector skeleton under work_items/. The contracts
already carry the Azure DevOps fields: source_ref.area_path, source_ref.iteration_path,
metadata.source.extraction_mode and AzureDevOps in systems, unfilled. utilities ships the pieces a work item
needs: authoring.html_to_markdown, normalize and the issue-body parser collector-gh will use.
docs/tutorials/ado-workitems-to-markdown.md and docs/guides/azure-devops-collection.md are marked planned.

The design to start from:

• Two body-extraction modes; the first real project decides which is built first. markdown:
  System.Description HTML → html_to_markdown → normalize → the same issue-body parser collector-gh uses, so an
  author writes the same ## Description / ## Status / ## Business Value / ## Acceptance Criteria sections as in
  a GitHub issue. What the converter drops gives HTML_CONTENT_DROPPED. With no ## Acceptance Criteria heading,
  a non-empty Microsoft.VSTS.Common.AcceptanceCriteria field is used; with both, the description wins with
  DUPLICATE_AC_SOURCE. field-map: a committed default map plus an operator override binds each canonical section
  to a field. It changes only where the text comes from, never how it is read.
• System.State fills only source_ref.tracker_state. User Story and Functionality status is authored, a Feature's
  state is derived, and the mode is recorded in metadata.source.extraction_mode.
• Proof is golden entities, not a round trip. Each mode reproduces utilities' golden entities from synthetic
  fixtures. A round trip that drops content on both sides passes trivially.
• Synthetic fixtures only. collector-ad owns their generator (tools/make_ado_fixtures.py) and a guard that
  fails on an internal-looking URL, GUID or custom field reference. living-doc regenerates
  docs/examples/ado-work-items/ from the pinned ref and fails CI on a byte difference. Nothing from an internal project
  (organisation, project, credential, work-item content, custom field names) is ever committed.

Goals

• collector-ad work-items writes a valid doc-entities.json through write_artifact. It takes a required
  project-id and writes to <output-path>/work-items/. It follows the partial-collection rules, and a
  --local-input run needs no credential and no network.
• The mode the first real project needs reproduces utilities' golden entities; each prepared contract field is
  confirmed or removed.
• living-doc-toolkit reads doc-entities.json in normalize-docs and coverage-matrix; mixing or doubling
  documentation sources still fails.
• living-doc snapshots the ADO chain over its synthetic corpus, and a milestone runs it on a real target with a
  redacted report.
• The action has the fleet shape and the repository's documentation follows the fleet's page rules; the ADO tutorial
  carries a workflow; a release lists collector-ad.

Subtasks

In landing order. Each child is drafted when the previous one has merged.

• living-doc (Spike form): measure one real internal project in an operator run. The report holds a shape
  inventory only; no payload leaves the machine. It answers: hosting and auth, volume, the fields in use, the HTML
  constructs, the type and status mapping, the entity-id rule, which extraction mode first, and whether the second
  is needed. It confirms or removes each prepared contract field and names any missing one, which becomes a
  utilities contract change.
• living-doc-collector-ad: the composite action in the fleet shape (copy generator-markdown's venv step;
  runtime requirements.txt, dev tools in requirements-dev.txt).
• living-doc-collector-ad: REST / WIQL client with 200-id batches and paging. It retries 429 and 5xx with
  backoff and Retry-After; giving up names the id range and raises SOURCE_UNAVAILABLE, never a partial result.
  It covers the auth and hosting variants the spike found.
• living-doc-collector-ad: work-item mapping onto doc-entities (types per the spike's table, native_type,
  area_path, iteration_path, the entity_id rule with MISSING_ENTITY_ID; System.State only in
  tracker_state).
• living-doc-collector-ad: body extraction in the mode the spike selects, proven by golden entities, with every
  HTML shape from the inventory tested and the fixture generator and guard committed. A converter gap is a
  utilities patch.
• living-doc-collector-ad: relations → entity graph (UNRESOLVED_RELATION, counted in unresolved_refs).
• living-doc-collector-ad: wiring. The work-items mode in action.yml, with --local-input, the output
  layout, a required project-id, the partial-collection rules, the producer and extraction_mode, and a derived
  Feature state. The component checks run in make qa.
• living-doc-collector-ad (only if the spike recommends it): the second extraction mode.
• living-doc-collector-ad: local-integration.sh, which living-doc's runner reaches without a runner code
  change.
• living-doc-toolkit: doc-entities input for normalize-docs and coverage-matrix, behind an adapter
  registry. An unknown producer is named; MIXED_DOCUMENTATION_SOURCES and MULTIPLE_DOCUMENTATION_SOURCES are
  kept. Shared with the GitHub Issues epic: whichever epic gets there first builds it.
• living-doc: the ADO corpus regenerated from the pinned collector-ad, the snapshot chain, and both negative
  runs.
• living-doc-collector-ad: documentation by the fleet's page rules, plus the manual link-check fix.
• living-doc: the ADO tutorial and guide lose "planned" and get a workflow, then a milestone run on a real
  target with a redacted report, then the release.

Related / References

• living-doc-utilities docs/authoring/urls-and-html.md (HTML conversion), docs/contracts/pipeline-rules.md
  (output layout, one documentation source, partial collection), docs/contracts/errors.md.
• living-doc docs/guides/azure-devops-collection.md, docs/tutorials/ado-workitems-to-markdown.md,
  docs/projects/collector-ad.md, docs/_integration/README.md (internal targets, redacted reports).
• Related issues: the contract should name a scenario's repository (a second ui-tests producer); the agentic layer
  writes no issue body or work item.
• Not in this epic: the boards, pipelines, test_plans and release_notes modes. They wait until a real
  project needs one.