Proposal: enrich existing Pure publications with multi-source metadata

Bắt đầu bằng cách đọc backend/app/services/pure_enrichment.py, đặc biệt là _apply_publication_updates và _set_volume, sau đó kiểm tra các loại job hiện có và luồng của bảng review. Một cột mốc đầu tiên là dry run không ghi dữ liệu, báo cáo các trường còn thiếu, các giá trị nguồn và những điểm không khớp với Pure cho từng ấn phẩm; phạm vi triển khai phụ thuộc vào các quyết định về chính sách ghi, mức độ tin cậy của nguồn, việc chọn bản ghi và vị trí đặt lớp connector.

Idea

doi_resolver compares publication metadata across several scholarly sources (Crossref, OpenAlex, DataCite, Europe PMC, Unpaywall, CORE), keeps provenance for every value, and can write selected values back to Pure. Today that runs one DOI at a time through a UI.

Proposal: a BackToPure job that does the same for existing Pure publications in bulk — filling gaps in records that are already there.

Sibling of #6 (which covers full-text PDFs). Depends on #2, #3 and #5.

What doi_resolver already enriches

_apply_publication_updates (backend/app/services/pure_enrichment.py:755) maps these fields onto Pure:

title, journal, abstract, volume, issue (→ journalNumber), pages, open_access_status, license, keywords, subjects, language, published_year, published_date

It also handles identifiers, and matches persons and organizations against existing Pure entities, reporting what matched and what was skipped and why.

The important risk: this overwrites

The setters replace the value in Pure whenever it differs. _set_volume (:955) is representative — it returns early only if the incoming value is identical, otherwise it writes.

That is fine in doi_resolver, where a person looks at one DOI and picks each value deliberately. It is not fine in a bulk job: a run across a faculty could silently overwrite curated Pure data with a worse value from an external source.

This needs a deliberate decision before any implementation:

• Fill-only — write only where Pure has no value. Safe, and probably the right default.
• Overwrite with review — propose differences, require a human to accept each one via the NEEDS_REVIEW / review-table flow.
• Overwrite on trust rules — e.g. trust Crossref over OpenAlex for pages. Most useful, most work, needs agreement on the ranking.

Recommend starting fill-only, with a dry-run report showing where sources disagree with Pure. That disagreement report is valuable on its own as a data-quality signal, separately from whether anything is written.

Scope: avoid duplicating existing jobs

BackToPure already has EXTERNAL_PERSONS and EXTERNAL_ORGS job types that enrich persons and organisations. doi_resolver's person/organisation matching overlaps with those.

This issue should therefore cover publication-level metadata only — the scalar fields and identifiers listed above. Person and organisation enrichment stays with the existing jobs unless there is a clear reason to merge them, which would be its own issue.

Fit with the job architecture

As with #6: a new JobType, identity_columns=("doi",), NEEDS_REVIEW plus the review-table for the human check, and apply / rollback for the write phase. Rollback matters more here than usual, since this touches existing curated records rather than adding new ones.

Where should this functionality live? (decide later)

#6 argues for porting code out of doi_resolver rather than depending on it, because the PDF code is small and the dependency is heavy. That reasoning does not automatically carry over here: the connector layer is six sources plus provenance and merge logic, which is a lot to maintain in two places.

There is a third option worth keeping open — BackToPure may simply be the better long-term home for this functionality.

Arguments for consolidating it here:

• BackToPure is what colleagues actually run. doi_resolver is not in general use.
• The job architecture already provides what bulk enrichment needs: a job registry, NEEDS_REVIEW with a review table, apply/rollback, logs, artifacts and cancellation. doi_resolver would have to grow all of that to do this at scale.
• Multi-source comparison with provenance is useful to every job type here, not only to this one. As a shared capability inside BackToPure it could also improve the research-outputs import, which currently relies on OpenAlex alone.

Arguments against, or at least for care:

• doi_resolver's single-DOI UI is genuinely useful for interactive quality control, and that use case should not be lost.
• It is a working, tested codebase; moving it is not free.

Possible shapes, if consolidation wins: move the connector layer into BackToPure and leave doi_resolver as a thin UI over it; or move it and retire doi_resolver; or keep both and accept the duplication.

No decision needed now. Flagging it so the choice is made deliberately when this issue is picked up, rather than defaulted into by whoever writes the first line of code.

Open questions

1. Fill-only or overwrite? (see above — needs a decision before implementation)
2. Which sources are trusted for which fields?
3. Which Pure records are in scope — a faculty, a date range, or records missing specific fields?
4. Port, depend, or consolidate? See the section above — this is the significant architectural decision in this issue, and it is deliberately left open.

Suggested first step

A dry-run job that reports, per publication: which fields Pure is missing, what each source offers, and where sources contradict Pure. No writes. That gives a concrete picture of how much this would actually improve the data before anyone commits to the write path.