Report — Why CIEL Lab keeps a local OCL mirror and a local index, and what the OCL API would need to add for CIEL Lab to drop them

Author: @filiperochalopes (🤖 Deepseek 4.1 Flash) (reporter session) · Date: 2026-10-12
Repos inspected: ciel-lab-v2 (ba24368, main) and oclapi2 (local checkout)
Requested by: OCL maintainer (why the daily CIEL HEAD export download exists)
Related issues: OpenConceptLab/ocl_issues#2769, #2768, #2752; ocl_issues#2839, #2572, #2581, #2696, #2681, #2669, #2874; ciel-lab-v2#46, #62, #67, #76, #79, #82

Status of this document: evidence-backed gap analysis. Every technical claim below
cites a file path in one of the two repositories, or a committed benchmark under
docs/lab/. Items that need a human decision are collected in §7 Open questions.
No code was changed.

---

1. TL;DR

CIEL Lab does not download the CIEL HEAD export because OCL lacks an export endpoint.
It downloads it because OCL, as an API, cannot answer the four questions CIEL Lab asks
of the whole CIEL catalogue on every curation action, at the latency the curators need:

1. "Is this concept consistent with every other concept in CIEL?" — cross-concept QA
   (duplicate FSN, duplicate preferred name, duplicate SAME-AS target / merge candidate).
2. "Given this draft concept, check QA rules (cross-concept and in-concept) and give me a
   feedback if it's ok to push it to production based in our rules. (in real time / low
   latency)" — once we edit we check rules against the whole CIEL corpus and external
   sources based on debounced "onType" and "onBlur" events.
3. "What is the active/retired status of these N mapping codes across SNOMED/ICD-11/
   RxNORM/WHOATC?" — batch status lookups against terminologies partly not hosted in OCL.
4. "Give me this whole corpus, filtered, counted, sorted and paged, fast and cheap,
   without spending our OCL token and without a round-trip per keystroke."

Question 2 is the answering surface: the Concept Editor (debounced onType/onBlur) and
the MCP server (§4c) both feed the same QA engine, and both depend on the local corpus
to answer in real time. Question 2 is what makes the mirror a write-path dependency, not
just a read optimisation.

The local mirror (Valkey) and the local index (Qdrant) exist to answer those locally.
They are two different stores for two different jobs (documented in
docs/lab/bench-qdrant-vs-valkey-head.md): Valkey is the write-through working copy of
OCL HEAD that 22 modules read; Qdrant is the semantic + text index built from Valkey.

Dropping them requires OCL API features that do not exist today: a cross-concept
QA/aggregation surface, a batch external-code status surface, a server-side validation
run over a whole version, and guarantees on latency/token cost that only a local copy
gives us. A smaller, sharper version of that list is in §5.

---

2. What the daily export actually feeds (evidence)

api/src/tasks/head_refresh.py:9-33 — the nightly task ciellab.refresh_head_and_index
downloads OCL's single HEAD export ZIP and repopulates the Valkey mirror, then
enqueues index_ciel (Qdrant) and refresh_staged_changes_snapshot in a chain
(head_refresh.py:117-174, :555-562).

Three consumers read the same artifact, not one:

Consumer	Reads	Source of truth
HEAD mirror (Valkey)	every concept payload	HeadConceptCache.populate_from_export (head_refresh.py:236-244)
Semantic/text index (Qdrant)	one point per vectorizable English name	index_ciel._run_index_ciel iterates cache.all_ids() (api/src/tasks/index_ciel.py:516)
Staged-changes diff + OpenMRS release SQL	HEAD vs released version	src/tasks/build_release_sql.py, src/services/ocl_staged_changes_service.py

The Qdrant index is downstream of the mirror: it is built from Valkey, so the mirror
cannot be removed without re-plumbing the indexer, the staged-changes diff and the release
dump (docs/lab/bench-qdrant-vs-valkey-head.md:54-74). The mirror's dependency surface is
quantified there: 22 modules, 35 upsert call sites, 36 get/get_raw, 6
populate_from_export, 35 ids_for_*.

The export itself is already an OCL-side burden the maintainer should know about:
ocl_issues#2839 (HEAD exports drop and duplicate concepts when the source is edited during
the export), #2572 (optimize HEAD export generation using previous export plus diff),
#2581/#1267/#817 (export speed + old-export cleanup). CIEL Lab's daily pull is one consumer
of a pipeline that is itself under active repair.

---

3. The three jobs, and why they are local today

Job A — Cross-concept QA over the whole corpus

The QA catalogue is 32 rules, FR-01 … FR-32 (ocl_issues#2768), evaluated in two tiers
(docs/decisions/0002-qa-rules-run-in-two-tiers.md): local (decidable from one payload,
gates Save) and deferred (needs the HEAD/Qdrant/OCL corpus, runs in Bulk Validation).

The deferred / cross-concept rules are the ones with no OCL equivalent:

Rule	Catches	What it needs
FR-09	FSN collides with another active concept	corpus-wide index on (locale, FSN text)
FR-10	Preferred synonym duplicated on another active concept	corpus-wide index on (locale, preferred name)
FR-22	Mapped to a retired/inactive code in an indexed terminology	batch external-code status lookup
FR-26	Another active concept shares the same SAME-AS target (merge candidate)	corpus-wide index on mapping target

These resolve through mirror secondary indexes — HeadConceptCache.ids_for_fsn (:1062),
ids_for_preferred_name (:1068), ids_for_mapping_code (:1117),
ids_for_mapping_system (:1024), ids_for_mapping_map_type (:1113)
(api/src/adapters/secondary/cache/head_concept_cache.py). OCL exposes no query that
answers "which other concepts in this source version share this FSN / this preferred name /
this mapping target". Elasticsearch could, but no OCL endpoint composes those aggregations.

Cross-concept latency is small because it is local: FR-09/10/26 cost 0.18–0.21 s p50
on the full 58.8k corpus (docs/lab/bench-qa-bulk-validation.md:40), and each cross-concept
candidate resolves in 1.3–2.3 ms (docs/lab/bench-concept-editor-qa-latency.md:73).

Job B — Bulk code-status lookups across external terminologies

FR-22 asks, for every mapping in the corpus, whether the target code is retired/inactive in
its terminology. api/src/services/external_code_service.py:33-71 shows the routing, and it
is the clearest evidence of the external-source gap:

Terminology	Local Qdrant system	Legacy collection	OCL fallback
CIEL	—	—	(own)
SNOMED CT	snomedct, snomed	ciellab_source_snomed	none
ICD-11	icd11	ciellab_source_icd11	none
ICD-10	icd10	—	/orgs/WHO/sources/ICD-10-WHO/
LOINC	loinc	—	/orgs/Regenstrief/sources/LOINC/
WHOATC	whoatc	ciellab_source_whoatc	/orgs/WHO/sources/WHOATC/
RxNORM	rxnorm	ciellab_source_rxnorm	/orgs/NLM/sources/RxNORM/

SNOMED CT and ICD-11 — the two terminologies CIEL maps to most — have no OCL lookup path
at all; their active/retired status must come from a locally indexed copy. This is exactly
the "external sources we don't have online, and we need them for QA" point, and it is
independent of the daily export — it is a separate index (index_terminology reads
external source_* MySQL databases, not the CIEL export; index_ciel.py:2-4).
ocl_issues#2874 (Support linked mapping-only sources for CIEL SNOMED-CT and ICD-11
mappings) is the OCL-side ticket that would move this.

fetch_statuses (external_code_service.py:152-223) also asserts an important semantic:
when the index cannot answer, codes are reported unavailable — never silently read as
"no problem found". Any OCL replacement must preserve that (0003 decision, #2752).

Job C — Fast, cheap, exact corpus reads

Every benchmark that justifies the local stores, all committed under docs/lab/:

Read	Local	OCL API	Source
count(name~malaria) type-ahead	52 ms (Qdrant prefix index)	1,447 ms (before)	bench-qdrant-vs-valkey-head.md:428-435
Filter-only count(class=Diagnosis)	5.2 ms	—	bench-qdrant-vs-valkey-head.md:37
Full bulk validation of HEAD	53–64 s	(would be one HTTP round-trip per concept)	bench-qa-bulk-validation.md:29-30
Filtered CIEL scan (class=Diagnosis)	632 ms	3,522 ms	bench-ciel-scan-strategy.md:37
Concept Editor QA (local tier, gates Save)	0.1–0.3 ms	4–150 ms	bench-concept-editor-qa-latency.md:26

The maintainer's framing is right: "it is impossible to select and paginate what we need
directly through the OCL API and be fast." The committed numbers quantify the gap, and the
gap is dominated by round-trip count, not by any single OCL query being slow.

Token/rate cost is the second half of Job C: docs/decisions/0008 — anonymous reads spend
the OCL token and the index — so a local copy is also a cost-control boundary.

---

4. Correlation — OCL API capabilities vs. what CIEL Lab rebuilt locally

This is the table Filipe asked for: "what OCL API exposes vs. what we reinvent in Qdrant."

4a. Filters — OCL ↔ CIEL Lab

Need	OCL API (evidence)	CIEL Lab local equivalent
Class filter (OR, negation)	conceptClass=A,B, =!A (ocl-custom-attributes.md:40-42)	filter_dsl concept_class equals/in (filter_dsl.py:14)
Datatype filter	datatype= (concepts/models.py:298)	filter_dsl datatype (:15)
Locale filter	locale= (models.py:301)	filter_dsl locale / names_meta[].locale (index_ciel.py:137)
Retired toggle	includeRetired= (__build_concept_query_params)	retired bool payload (index_ciel.py:121)
Mapping target system	targetRepo= nested mapped_codes (client.py:1215)	mapping_systems / mappings[].system (index_ciel.py:132)
Mapping map-type	targetRepoMapType=	mapping_map_types / mappings[].map_type
Mapping target code	folded into q via same_as_map_codes (client.py:1186-1190)	mapping_codes text index (index_ciel.py:126)
Custom attributes (extras)	extras.<k>=, extras.exact.<k>=, extras.exists= (ocl-custom-attributes.md:19-23)	payload fields / filter_dsl text rules
Custom attributes (declared property)	properties__<code>= faceted, needs declaration + reindex (ocl-custom-attributes.md:24)	extras read directly from payload
Name type	nameTypes facet (concepts/search.py:51)	name_types, names_meta[].name_type
Facets (counts)	includeFacets header, ConceptFacetedSearch (concepts/search.py:35-57)	MCP list_concept_facets from OCL source summary (mcp/tools/facets.py)
Semantic / kNN	$match + semantic=true, numCandidates/kNearest (concepts/views.py:943-965)	Qdrant hybrid + retrieve_mapping_candidates
Rerank	$rerank (CrossEncoder bge-reranker-v2-m3) (common/search.py:394-425)	Qdrant scores + embeddings
Text search	q= multi_match/fuzzy (common/search.py)	Qdrant prefix tokenizer (0001 decision)

Reading: the filter vocabulary is largely already in OCL, and CIEL Lab's filter_dsl
is deliberately shaped like OCL's (filter_dsl.py:1-19 documents OCL op parity). So filters
are not the main reason for the mirror — the main reasons are §4b.

4b. What OCL does not offer (the real gaps)

CIEL Lab capability	Where it lives	OCL API equivalent
Cross-concept duplicate FSN / preferred name	ids_for_fsn, ids_for_preferred_name	none
Duplicate SAME-AS target / merge candidate (FR-26)	ids_for_mapping_code	none
Whole-corpus validation run (Bulk Validation)	qa_bulk_service._run_qa_bulk_job	none (would be N calls)
Batch external-code status (FR-22)	fetch_statuses	none; SNOMED/ICD-11 not in OCL
HEAD-vs-released diff (staged changes)	ocl_staged_changes_service	none (only the export gives both snapshots)
Write vocabulary parity check (FR-29…32)	ocl_source_directory	partially — OCL validates writes, but not against CIEL's vocab
Server-side QA rule engine	qa_bulk_rules.RULE_CATALOG	the abstraction in #2752, not shipped
Release artifact (OpenMRS 1.11 SQL)	services/openmrs_dump.py	out of scope for OCL (back-compat surface)

4c. What our MCP requires — latency and filters — that the OCL API does not satisfy

The MCP server is a demanding client of the same corpus, with requirements the OCL API
cannot meet as-is. It is the sharpest lens on the gap because its calls are interactive and
it states its own budgets. Two requirement families: latency and filter
expressiveness.

(a) Latency the MCP requires, and what OCL gives back

MCP requirement	Where it is stated	What OCL API delivers
A whole call must fit the MCP bridge/client timeout — hence the hard 32-term cap on retrieve_mapping_candidates ("keeps a single call well within the MCP bridge/client timeouts observed in production", mcp/tools/retrieve_mapping_candidates.py:11-15)	batch by design	one OCL HTTP round-trip per term; N calls cannot meet a single-call budget
Interactive type-ahead search_concepts must answer in tens of ms	CielQdrantSource two-phase scan (services/concept_search/ciel_source.py:281-325)	the OCL live fallback (OclHeadSource cold path) is the 1,397 ms full scan (bench-qdrant-vs-valkey-head.md:43)
The call cannot block the worker loop (decision 0007)	every MCP handler is async via run_blocking	a slow OCL call is exactly the blocking cost the rule exists to avoid
list_concept_facets must be instant (LLM calls it before every structured search)	mcp/tools/facets.py, cached 12 h	OCL source-summary distributions are fast but the cache is ours
propose_bulk_edit resolves uuids against the HEAD mirror	mcp/server.py:118	would need a lookup per row against OCL

The decisive number: the MCP's interactive cost is 52 ms through the local Qdrant index
versus 1,397 ms through the (Valkey) full scan that is itself already faster than a
live OCL round-trip. The MCP cannot be served by "just call OCL".

(b) Filters the MCP composes that the OCL API does not express

filter_dsl (services/concept_search/filter_dsl.py:1-19) is the structured filter the LLM
builds for search_concepts. Checking each capability against what OCL exposes
(oclapi2 core/concepts/models.py:284-323 es_fields, core/concepts/search.py,
core/common/views.py):

MCP filter capability	Exists in OCL API?	Evidence
Nested correlation: system + code + map_type on the same mapping	No	OCL has targetRepo and targetRepoMapType as separate nested facets and sends the target code through q/same_as_map_codes; it cannot AND all three on one mapping. CIEL compiles to a Qdrant Nested condition (index_ciel.py:196-210)
Text contains on a mapping code (substring)	No	OCL folds the code into q (client.py:1186-1190), whole-word; CIEL has mapping_codes text + code contains (filter_dsl {"op":"contains"})
Name filter correlated by locale + name_type + value	No (per-entry)	OCL has nameTypes and locale as independent filters; CIEL correlates them via names_meta[] (index_ciel.py:135-138)
Description value/locale filter	Partial	OCL description is a searchable field; no dedicated correlated param; CIEL descriptions_meta (index_ciel.py:238-248)
Absence query ("concepts without a pt/pt-br translation")	No	OCL locale= is a positive filter; CIEL uses must_not on the concept-level locales list (index_ciel.py:139-142)
regex on a text field	No	CIEL filter_dsl supports {"op":"regex"}; OCL has no regex param
exists on arbitrary text fields	Partial	OCL only has extras.exists= for extras (ocl-custom-attributes.md:23); CIEL exists applies to names/descriptions/mappings
Batch retrieval by an explicit concept_ids list	No	CIEL concept_ids → MatchAny (ciel_source.py:150-156); OCL needs N calls or one q
Arbitrary AND/OR filter trees	No	CIEL filter_dsl groups (`{"op":"and"
Class/datatype/locale/map-system OR + negation	Yes	conceptClass=A,B / =!A; targetRepo, targetRepoMapType (ocl-custom-attributes.md:40-42)
extras.* substring / exact / exists; declared properties__* facet	Yes	ocl-custom-attributes.md:19-24

Reading: the OCL API covers the flat filters (class, datatype, locale, map system/type,
extras/properties) but not the correlated, absence and arbitrary-tree filters the MCP
composes per search, nor the code-substring and regex text operators. Those, plus the
latency budget, are what force the MCP onto the local index. Verified against the local
oclapi2 source, not assumed.

4d. What "abstract the entities and concepts" means for a server-side QA tool

The QA engine in CIEL Lab is already a specialized rule engine over canonical facts,
not scattered validations (ocl_issues#2752): RULE_CATALOG is the single registry (each
rule declares title, severity, category, scope, tier, supported contexts, editor lock),
QaBulkEvaluator holds the facts and the cross-concept indexes, and enabled rules /
retired scope are passed into the evaluator. The natural OCL counterpart is a
canonical QA fact contract + a rule catalog + execution contexts, so the same rule can
run in the editor (real-time, question 2) and in a whole-version bulk run (question 1) —
on either side. That proposal is written up in the companion ticket
qa-abstraction-proposal.md (open a separate detailed issue), and it is the concrete
shape of F1 + F3 in §5.

---

5. Features the OCL API would need to add (so CIEL Lab can drop the mirror + index)

Ordered by how much of the local stack each would remove. F1–F3 are the load-bearing
ones; F4–F7 are supporting.

F1 — Cross-concept consistency query

A server-side way to ask "which other concepts in source version V share this FSN /
preferred name / mapping target". Concretely an aggregation or QA-search endpoint, or an
ES-backed facet with correlation, returning the participating concept ids + fields.
Replaces: ids_for_fsn / ids_for_preferred_name / ids_for_mapping_code
(mirror secondary indexes) → removes FR-09/10/26 deferred-tier local scanning.
OCL refs: #2752 (QA architecture), oclapi2 core/concepts/search.py facets.

F2 — Batch external-code status service

One endpoint that takes N (source, code) pairs and answers active/retired, spanning
terminologies OCL actually hosts — and hosting/indexing SNOMED CT and ICD-11 so their
status is answerable at all. Replaces: external_code_service.fetch_statuses + its
Qdrant/legacy-collection lookups → removes FR-22's local index.
OCL refs: #2874, core/value_sets, core/code_systems.

F3 — Server-side version validation run

A job that validates a whole source version against a rule set and returns findings
(asynchronously, like the existing resources_report / make_hierarchy tasks), so CIEL
does not have to materialize the corpus to run Bulk Validation. Replaces:
qa_bulk_service scanning the mirror → the strongest candidate to make the daily export
unnecessary for QA.
OCL refs: #2752; core/reports, core/tasks (async job pattern exists).

F4 — Version-diff endpoint

"HEAD vs released version" structured diff, so the staged-changes feature does not have to
hold two snapshots locally. Replaces: ocl_staged_changes_service +
refresh_staged_changes snapshot.
OCL refs: #2572 (export diff), docs/decisions/0011.

F5 — Exact-total, deep-offset, sorted corpus pagination with a latency SLA

totalCount consistent with nodes, arbitrary numeric offset, sort by id — at the latency
committed benchmarks show locally. OCL already returns num_found (client.py:1221-1266);
the missing piece is the latency and token-cost guarantee under CIEL-scale repeated reads.
Replaces: the head-mirror read path (0008 auth rationale, #82 slow log).

F6 — Configurable text-search semantics

Prefix/substring tokenizer + optional ngram, selectable per field, with an exact-total
guarantee for text. Replaces: CIEL's Qdrant prefix index (0001 decision). Qdrant
1.19 has no ngram tokenizer (bench-qdrant-vs-valkey-head.md:221-244), so this is a real
product decision, not a lift-and-shift.

F7 — Retired-aware surface parity

Per-query ability to include/exclude retired concepts, names and mappings separately
(CIEL needs retired concepts for the release dump but never retired names in active
content — #2681, 0003). OCL has includeRetired; the combined semantics used by CIEL are
not expressed in one query.

If OCL shipped F1–F3, CIEL Lab could delete the Qdrant QA index and the mirror's
cross-concept secondary indexes. F5–F7 are needed before the mirror itself goes.

---

6. References (online, useful)

• CIEL Lab QA overview — https://filiperochalopes.github.io/ciellab-wiki/features/quality-assurance/overview/
• CIEL Lab QA settings — https://ciellab.filipelopes.med.br/qa/settings
• OCL custom-attributes API — https://docs.openconceptlab.org/en/latest/oclapi/apireference/custom_attributes.html
  (correlation table + measured behaviour in docs/lab/ocl-custom-attributes.md)
• QA blog-post drafting issue — ocl_issues#2768; governance companion — ocl_issues#2769
• QA-rules-vs-Drools architecture issue — ocl_issues#2752
• Committed benchmarks: docs/lab/bench-qdrant-vs-valkey-head.md,
  bench-qa-bulk-validation.md, bench-ciel-scan-strategy.md,
  bench-concept-editor-qa-latency.md, bench-textual-vs-index-search.md,
  bench-concepts-head-latency.md
• Architecture decisions: docs/decisions/0001 (prefix search), 0002 (two QA tiers),
  0003 (verbatim vocabulary), 0008 (authorization/token cost), 0009 (mirror refresh),
  0011 (staged changes)