C7.12i Preserve DTC article metadata and fenced raw code during conversion

Read the listed process and architecture documents first, then inspect community_base/content_sync/convert/documents.py, the proposed convert modules, and tests/content_sync/test_convert_documents.py and test_convert_report_compat.py. Add focused characterization and DTC article tests in the named test module, using the proposed synthetic fixtures. Done means metadata and bounded raw-wrapper behavior are covered without changing public interfaces, reports, UUIDs, refusals, or other profiles.

C7.12i Preserve DTC article metadata and fenced raw code during conversion

Repository: community-base. Depends on: C7.12. Freeze required: no.

Status: groomed scope; implementation awaits allocation of an isolated source base and exact write set. Parent adoption: DataTalksClub/website#452, D7.2 article part (D7.2a preparation). Decisions D21, D23 and D24 remain intact.

Goal

Correct the existing dtc-articles converter so its supported records preserve authored metadata, publication and media identity, and so balanced Liquid raw wrappers around fenced code convert without changing that code. Characterize and extract the existing metadata/body concerns before changing behavior. Keep one converter, its public entry points, UUID identities, report format and other profiles stable.

This card can complete its bounded package capability while the full 55-article D7.2 rendering comparison remains open. It does not solve the other 11 include-bearing refusal candidates, change a reader or source repository, or authorize a partial corpus cutover. No visible UI, feature, route, asset or history loss is permitted.

Read first

• AGENTS.md, docs/PROCESS.md, docs/04-quality-gates.md and coding-standard.md at the allocated base.
• docs/01-decisions.md D21/D23/D24; docs/02-architecture.md public extension boundaries; docs/03-playbooks.md P15/P16.
• docs/plan/phase-7.md C7.12 and D7.2; immutable docs/plan/evidence/conversion-runs-2026-09-18.md.
• community_base/content_sync/FORMAT.md sections 3.3–3.7 and 4; kinds/article.py.
• community_base/content_sync/convert/documents.py, report.py; tests/content_sync/test_convert_documents.py and test_convert_report_compat.py.
• Retained converted-input REPORT, FIXTURE-PLAN, SOURCE-INVENTORY and REFUSAL-LEDGER; independent article-conversion scope review. These are static preparation, not a successful conversion receipt.
• DTC106c777 content/sync_parsers/articles.py publication/metadata policy and scripts/build_public_projection.py current normalization. Read committed objects only; do not copy or import site code into this package.

Frozen source and mapping

The reproducible public fixture candidate is DataTalksClub/content d3397ee2538d17d35d6473342f78329404ab7bc0, tree6c02a0211c6b6d555c59760930337bee1e272797: 55 original article paths and unique normalized targets. Original Jekyll ee43d3fa0929faf691178d79f19528e6f15a83e5 is provenance, not authority to restore currently absent iframe/script output. The historical 43-article result lacks an exact source-SHA receipt and is not new proof.

Use these concrete rules only for dtc-articles:

1. Keep title unchanged at core title. Declare subtitle, authors and faq as article keys; preserve ordered author keys, FAQ question/answer strings and answer Markdown. Do not relocate them accidentally into extra. Snapshot maxima title98/subtitle164/summary316 fit existing limits; no schema width override, clipping or placeholder title.
2. String description becomes summary without trimming or rewriting. The recognized anomaly is a mapping with exactly one string key and one string value: normalize to key: value, matching current DTC behavior. Refuse other mappings, sequences and non-string values; do not stringify them. Missing optional description remains absent. If description and summary coexist, retain both original values and refuse incompatible values rather than silently choose or overwrite.
3. Reserve extra.dtc_article_v1 as a mapping with exactly source and publication. source has path (original relative path), sha256 (original whole-file bytes), front_matter_yaml (original frontmatter text without delimiters) and image (original authored image string, or absent). publication has field (date, datepublished or filename) and value (the current DTC string-normalized published value). Record source commit/tree and Git blob in the external immutable fixture receipt, not an invented converter-detected revision. The original frontmatter text preserves types, original description/date/datepublished/layout values and lexical form; original immutable file provenance preserves delimiters/body too. Refuse a preexisting reserved namespace on an unconverted source; never overwrite it. Replaying already converted output retains that namespace and source checksum unchanged. Other authored extra keys remain unchanged; if moving a top-level site key would collide with an existing unequal extra value, refuse rather than overwrite. Equal collisions are recorded without losing either original location in source provenance.
4. Publication precedence is the current DTC rule: truthy date, then truthy datepublished, then the date prefix from the original filename, with the existing two-digit-year conversion identity rule. Preserve the selected string-normalized display value separately from derived shared chronology. Shared date is the calendar-date component of a validated ISO date or ISO datetime; validate the entire input before deriving the component, and do not shift its timezone/calendar day. YAML date/datetime values use their ISO representation. Refuse unsupported/non-ISO values rather than blindly slice ten characters. Original values remain recoverable even when date and datepublished differ. This does not change current site display or invent a new publication policy.
5. Keep layout, datepublished and all other site metadata instead of dropping them. The original text is authoritative recovery; unmapped DTC keys (math, charts, h1, related_posts and other authored fields) remain under their existing extra names, with collision checks above. These are inert data; no executable Liquid, HTML or script interpretation is added.
6. Keep the existing asset rewrite: core image is the existing format-relative or HTTPS reference. Preserve authored public image identity under source.image; retain referenced and unreferenced legacy media bytes through the existing inventory/report accounting. This card does not move/delete extra assets or alter an asset uploader.
7. Keep original slug and /blog/<slug>.html identity in the original-path/target receipt; output remains articles/<date-stripped-slug>/index.md. Existing supplied content_id stays unchanged. Missing content_id uses the existing UUIDv5 namespace uuid5(NAMESPACE_URL, 'datatalksclub:dtc-articles') and target path. Record original path → target → ID → original hash; no random IDs or claim that minted IDs were donor identifiers.

This namespace is conversion provenance consumed by a later thin DTC adapter; it is not a new generic registry or runtime parser. Package validation still treats extra as a mapping. The site must validate these fields before relying on them. Future article values exceeding current shared limits remain an explicit later compatibility question; this smaller snapshot does not establish a general long-value policy.

Bounded raw-wrapper grammar

Admit only dtc-articles wrappers outside a code fence, with standalone {% raw %} and {% endraw %} lines (optional surrounding spaces/tabs). Between a pair allow blank lines and exactly one closed fenced code block. The opening is a backtick or tilde fence of at least three matching characters, indented at most three spaces; a closing line contains the same character repeated at least the opening length and optional whitespace only. Code bytes, opening info string, line endings and surrounding blank lines are retained exactly; remove only the two wrapper lines. Code may contain literal Liquid-looking strings and ${{...}}. Wrapper-looking lines inside code are literal code, not nested grammar.

Reject a stray/end-only/open-only wrapper, a nested wrapper outside code, multiple fenced blocks inside one pair, nonblank prose outside the enclosed fence, a shorter/mismatched closing fence, malformed wrapper tokens and any unclosed fence. Run the existing common fence/body pipeline after the bounded wrapper pass. Preserve existing supported transformations elsewhere; no global Liquid deletion, arbitrary raw prose admission, include evaluation or sanitizer change. Unknown includes still refuse with file/rule diagnostics. The practical-guide-better-code candidate has two qualifying pairs; resolving it is not evidence that it or all 55 converted successfully.

Steps and proposed write ownership

1. Freeze the exact allocated package commit and source fixture receipt. Confirm actual current converter/test allocations; the current231b package subject is frozen for separate CI evidence. Do not change it. A clean path or historical issue author does not acquire/release a lease.
2. Add authoritative characterization of existing other-profile metadata/body behavior, public imports, CLI, UUID/report fields, refusals, dry-run and replay. Prove it before extraction and afterward; review the behavior-preserving extraction separately from correction.
3. Retain documents.py as the public orchestration owner with stable PROFILES, Collection, Profile, convert_documents and main imports/re-exports. Move Collection/Profile/Refused, shared naming constants and the existing fixed profile catalogue once into convert/contracts.py to break the demonstrated documents ↔ metadata/body cycle; never create duplicate contracts or profile catalogues. Extract complete metadata projection into convert/document_metadata.py and complete body transformation into convert/document_body.py. Add convert/document_collections.py as the sole existing discovery/tree/placement/target owner, and convert/document_output.py as the sole existing pending-write/copy/removal/manifest/accounting/flush owner. These are two additional complete concerns needed to reduce the materially changed 1155-line facade below 300, not a second walker, writer or serializer. Preserve shared link/asset URL state and original partial valid-file conversion/refusal behavior. Remove moved private methods rather than leave forwarding chains or mixin layers. Put the bounded dtc-articles source policy in convert/article_profile.py only in the intentional-correction stage. No edits to report.py or core schemas are assumed by this allocation; re-groom if a concrete required public change appears.
4. Implement the mapping and bounded raw pass in those owners. Add new tests/content_sync/test_convert_dtc_articles.py and small synthetic fixtures; do not grow oversized test_convert_documents.py (346 lines at231b). Add characterization in a separate cohesive test module when existing coverage is insufficient. Each new handwritten file <=300 lines and function <=30; no ternaries or filtered/nested comprehensions. documents.py currently1155 lines must shrink; do not grow oversized functions/files or invent a waiver. If necessary extraction cannot stay cohesive within these constraints, report the exact design conflict before expanding scope.
5. Update content_sync/README.md and CHANGELOG.md with the exact correction, namespace/grammar and retained refusals. Preserve original conversion evidence. Add this follow-up plan/STATUS row through the normal isolated plan flow; do not overwrite foreign plan/changelog work.
6. Produce package capability evidence. Only after accepted package delivery and immutable release/tool identity can an allocated D7.2a implementation produce the selected converted fixture and complete current-reader parity. Full corpus rendering is produced during that site work before acceptance, not demanded as a completed artifact before assigning this package correction.

Potential source write set is documents.py, convert/contracts.py, convert/document_collections.py, convert/document_metadata.py, convert/document_body.py, convert/document_output.py, convert/article_profile.py, new focused characterization/article tests and synthetic fixtures, content_sync/README.md and CHANGELOG.md. Existing foreign CHANGELOG edits remain reserved; prepare an ignored entry proposal until that exact path is coordinated. Before freezing extraction, report actual facade/function/file sizes, removed private methods, old/new total handwritten production lines, duplicate-rule occurrences and new helper/class layers. Consolidate only the identical unknown/drop/extra accounting rule while preserving page and YAML rename differences; relocation or an unmeasured size estimate is not achieved net deletion. Plan grooming is a separate docs-only change. No DTC source, source content repository, FORMAT/core width, sanitizer, migrations, protected guidance, pin/lock or released identities are in scope.

Verification to run after implementation

• uv sync --all-extras in the allocated fresh package worktree; record actual collected/passed/skipped counts against that checkout's baseline.
• uv run pytest tests/content_sync/test_convert_dtc_articles.py tests/content_sync/test_convert_report_compat.py and the added characterization module: all pass. Full values/order/Unicode/media/provenance survive; date/datepublished/filename disagreement and the mapping anomaly discriminate wrong precedence/coercion. Reserved-extra and unequal moved-key collisions refuse without overwriting input. Supplied/minted IDs and source hashes survive dry-run/replay.
• Raw positive controls include both frozen wrapper pairs and synthetic backtick/tilde, Unicode/CRLF, blank lines, Liquid-looking text and interpolation. Negative controls cover every grammar refusal above, unknown includes and script/iframe in raw prose. Prove deletion of code, global Liquid stripping or permissive prose admission reaches and fails the owning assertion; an unrelated precheck failure is not that proof.
• uv run pytest tests/content_sync tests/test_boundaries.py and uv run pytest -q: all pass with actual counts. Other profiles keep characterized before/after behavior, public imports/CLI/report fields and UUID/path replay stable. Enumeration must be nonempty and the frozen source ledger exactly55; do not hide 12 refused candidates with filtering or weakened reference checking. Tests for this slice must not assert all55 success.
• make lint, uv run ruff format --check ., uv run python testproject/manage.py check, uv run python testproject/manage.py makemigrations --check --dry-run: all pass/no changes. Fresh SQLite migrations through testproject/manage.py pass using a unique disposable database; never delete a foreign database. No migration delta expected.
• uv run python scripts/plan.py check: OK after the isolated plan amendment. Both actual consumer checks under P16, fresh exact candidate and both latest default branches, are required package completion evidence via sole OnCall; no borrowed green or self-observation. Respect each site's local-test limits. No package completion while required consumers remain unresolved.

Done when

• Existing behavior is characterized before/after cohesive extraction; public entry points, UUIDs, report schema and other profiles remain stable.
• The bounded dtc-articles metadata/date/media mapping preserves original values/provenance without truncation, collision overwrite or random IDs.
• Only balanced fenced raw wrappers are admitted; all unknown include and malformed-wrapper refusals remain truthful.
• Meaningful focused/full/quality checks and both exact-candidate consumer checks pass; independent Tester and PM review bind the actual candidate.
• Documentation states this precise capability and every retained refusal; immutable historical evidence is unchanged.

Not run here, needs: D7.2a full55 source → converted → current native projection/render comparisons, every one of the12 semantic cases, human desktop/mobile output review, routes/SEO/FAQ/charts/assets/media/history/operator/no-op/rollback parity and actual green development deployment under DTC process. #452 B–E remain open. A separate include-declaration/projection scope must cover the other11 refusal candidates without restoring absent Jekyll UI or bypassing common sanitation. D7.4 owns real editorial authoring/conversion, source cutover and eventual converter deletion. No donor caller/media deletion or partial43/44 adoption is permitted by this card; tag-before-site adoption under P15 remains mandatory.

Docs

community_base/content_sync/README.md, CHANGELOG.md; separate docs/plan/phase-7.md and generated docs/plan/STATUS.md proposal. No release is claimed or performed by grooming.