USDM rules CORE-001055, CORE-000961 and CORE-001048 treat an omitted previousId/nextId differently from null

The bug lives in the JSONata conditions of the three rule definitions CORE-001055, CORE-000961 and CORE-001048, which treat an omitted previousId/nextId as different from null. Start by running the validate command on the omitted-null and explicit-null sample files and comparing the reports. Done means both files give the same results and the omitted-key file reports no issues from these three rules. The issue's suggested $exists check is a good starting point.

Standard

CDISC USDM

Reference Rule ID(s)

DDF00204, DDF00087, DDF00088

Conformance Rule ID(s) (if published in CORE)

CORE-001055, CORE-000961, CORE-001048

JIRA Ticket

https://jira.cdisc.org/projects/CORERULES/issues/CORERULES-

CLI Command Used (if applicable)

python core.py validate -s usdm -v 4-0 -dp issue12_data/shaun_h_usdm_nulls_omitted.json -r CORE-001055 -r CORE-000961 -r CORE-001048 -of JSON

Rule YAML

core_id: CORE-001055
description: Narrative content must only reference narrative content that is specified within the same study definition document version.
rule_type: JSONata
sensitivity: Record
entities:
  Include:
  - NarrativeContent
conditions: |
  $.study.documentedBy@$sdd.
    $sdd.versions@$sddv.
      $filter(($sddv.contents),function($v)
        {
          $not(
                $type($v.previousId) = "null" or
                $v.previousId in $sddv.contents.id
              ) or
          $not(
                $type($v.nextId) = "null" or
                $v.nextId in $sddv.contents.id
              ) or
          $v.childIds[$not($ in $sddv.contents.id)]
        }
      )@$c.
        {
          "instanceType": $c.instanceType,
          "id": $c.id,
          "path": $c._path,
          "StudyDefinitionDocument.id": $sdd.id,
          "StudyDefinitionDocument.name": $sdd.name,
          "StudyDefinitionDocumentVersion.id": $sddv.id,
          "StudyDefinitionDocumentVersion.version": $sddv.version,
          "name": $c.name,
          "sectionNumber": $c.sectionNumber,
          "Invalid previousId": $not(
                                      $type($c.previousId) = "null" or
                                      $c.previousId in $sddv.contents.id
                                    )
                                ? $c.previousId,
          "Invalid nextId": $not(
                                  $type($c.nextId) = "null" or
                                  $c.nextId in $sddv.contents.id
                                )
                            ? $c.nextId,
          "Invalid childIds": $c.childIds[$not($ in $sddv.contents.id)] ~> $join("; ")
        }
actions:
- name: generate_dataset_error_objects
  params:
    message: The narrative content references a previous, next or child id value that does not match the id of any narrative content defined within the same study definition document version.

---
core_id: CORE-000961
description: Encounter ordering using previous and next attributes is expected to be consistent with the order of corresponding scheduled activity instances according to their specified default conditions.
rule_type: JSONata
sensitivity: Record
entities:
  Include:
  - StudyDesign
  - InterventionalStudyDesign
  - ObservationalStudyDesign
conditions: |
  ($.study.versions)@$sv.
      ($sv.studyDesigns)@$sd.
          ($sd.scheduleTimelines[mainTimeline = true])@$st.
              (
                  $actord := $utils.parse_refs("nextId","next",$sd.encounters)
                              [previousId = null].$append(id,**.next.id);
                  $reford := $utils.parse_refs("defaultConditionId","defaultCondition",$st.instances)
                              [id = $st.entryId].**[instanceType="ScheduledActivityInstance" and encounterId in $sd.encounters.id].encounterId
                              ~> $filter(function($v,$i,$a){$i = 0 or ($i > 0 and $v != $a[$i-1])});
                  $encounter := function($id){($e:=$sd.encounters[id=$id];$id & ": " & $e.name)};
                  {
                      "instanceType": $sd.instanceType,
                      "id": $sd.id,
                      "path": $sd._path,
                      "name": $sd.name,
                      "ScheduleTimeline.id": $st.id,
                      "ScheduleTimeline.name": $st.name,
                      "ScheduleTimeline.mainTimeline": $st.mainTimeline,
                      "Encounter order by previous/next": "[ "&($actord ~> $map($encounter) ~> $join(" > "))&" ]",
                      "Encounter order by timeline refs": "[ "&($reford ~> $map($encounter) ~> $join(" > "))&" ]"
                  }[`Encounter order by previous/next` != `Encounter order by timeline refs`]
              )
actions:
- name: generate_dataset_error_objects
  params:
    message: The order of the encounters in the study design (as defined by the next attribute) does not match the order of encounters as referenced from scheduled activity instances in the main timeline that are ordered by default condition.

---
core_id: CORE-001048
description: Epoch ordering using previous and next attributes is expected to be consistent with the order of corresponding scheduled activity instances according to their specified default conditions.
rule_type: JSONata
sensitivity: Record
entities:
  Include:
  - StudyDesign
  - InterventionalStudyDesign
  - ObservationalStudyDesign
conditions: |
  ($.study.versions)@$sv.
      ($sv.studyDesigns)@$sd.
          ($sd.scheduleTimelines[mainTimeline = true])@$st.
              (
                  $actord := $utils.parse_refs("nextId","next",$sd.epochs)
                              [previousId = null].$append(id,**.next.id);
                  $reford := $utils.parse_refs("defaultConditionId","defaultCondition",$st.instances)
                              [id = $st.entryId].**[instanceType="ScheduledActivityInstance" and epochId in $sd.epochs.id].epochId
                              ~> $filter(function($v,$i,$a){$i = 0 or ($i > 0 and $v != $a[$i-1])});
                  $epoch := function($id){($e:=$sd.epochs[id=$id];$id & ": " & $e.name)};
                  {
                      "instanceType": $sd.instanceType,
                      "id": $sd.id,
                      "path": $sd._path,
                      "name": $sd.name,
                      "ScheduleTimeline.id": $st.id,
                      "ScheduleTimeline.name": $st.name,
                      "ScheduleTimeline.mainTimeline": $st.mainTimeline,
                      "Epoch order by previous/next": "[ "&($actord ~> $map($epoch) ~> $join(" > "))&" ]",
                      "Epoch order by timeline refs": "[ "&($reford ~> $map($epoch) ~> $join(" > "))&" ]"
                  }[`Epoch order by previous/next` != `Epoch order by timeline refs`]
              )
actions:
- name: generate_dataset_error_objects
  params:
    message: The order of the epochs in the study design (as defined by the next attribute) does not match the order of epochs as referenced from scheduled activity instances in the main timeline that are ordered by default condition.

Attach any sample test data file(s)

issue12_data.zip: two USDM 4.0 JSON files for the same fictional study. The only difference is that shaun_h_usdm_nulls_omitted.json leaves out previousId/nextId where they are null (on epochs, encounters and narrative content), and shaun_h_usdm_nulls_explicit.json writes them as null. Both pass the USDM 4.0 schema with no errors.

Second command, for the explicit-null file:
python core.py validate -s usdm -v 4-0 -dp issue12_data/shaun_h_usdm_nulls_explicit.json -r CORE-001055 -r CORE-000961 -r CORE-001048 -of JSON

issue12_data.zip

Attach any output report and/or log file(s)

With previousId/nextId omitted, all three rules report issues (CORE-001055 on the first and last NarrativeContent; CORE-000961 and CORE-001048 on the study design, with an empty order from previous/next). With explicit null, all three pass.

issue12_output.txt

Expected output

The same result for both files: no issues from CORE-001055, CORE-000961 or CORE-001048, since omitting an optional previousId/nextId means the same as null.

Any Additional Information

The USDM 4.0 API schema does not require previousId or nextId, so they can be omitted. CORE-001055 tests $type($v.previousId) = "null"; an omitted key is undefined, not null, so the ends of the list are reported as pointing to content that does not exist. CORE-000961 and CORE-001048 build the encounter and epoch order from the item where previousId = null; with the key omitted nothing matches and the order comes out empty.

Suggested fix: treat a missing key as null, e.g. $not($exists($v.previousId)) or $type($v.previousId) = "null" in CORE-001055, and [$not($exists(previousId)) or previousId = null] when finding the first item in CORE-000961 and CORE-001048.

Environment: CORE 0.17.1 (GitHub main, commit 22ce7a7, 29 October 2026), Python 3.12, Linux.