Non-deterministic input/output mapping: overlapping destination paths silently drop data
I maintainer di solito rispondono entro 1 giorno
@lokewate ci sta già lavorando.
Dal 14/6/2026.
Valutazione
Questa issue non è ancora stata valutata.
Descrizione
Describe the Bug
mapTaskInputs (and mapTaskOutputs) apply a node's input_mapping / output_mapping by ranging over a Go map and calling setNestedKey for each entry. When two mapping entries write to overlapping destination paths — one a leaf (submission) and one a nested child of it (submission.reference_number) — the final result depends on the iteration order of a Go map, which is randomized. Because setNestedKey overwrites on a leaf assignment but merges on a nested assignment, the two possible orders produce different results, and one of them silently drops data.
This is a latent, ~50/50 flaky data-loss bug for any workflow whose mapping has a destination path that is a prefix of another destination path.
Where (code references)
workflow/dsl.go—InputMapping map[string]stringis documented asWorkflowVariables Key -> Task Input Key, i.e. the map value is the destination path.workflow/workflow.go—mapTaskInputsiterates the mapping nondeterministically and applies each entry:for rawGlobalKey, localKey := range inputMapping { // <-- Go map range: randomized order globalKey, optional := parseMappingKey(rawGlobalKey) val, exists := getNestedKey(g.instance.WorkflowVariables, globalKey) // ... setNestedKey(inputs, localKey, val) // <-- destination path }workflow/utils.go—setNestedKeyoverwrites a leaf but merges a nested path:func setNestedKey(m map[string]any, dotPath string, value any) { // ... for a path WITH a dot: descends, REUSES/creates the intermediate map (merge) // setNestedKey(subMap, rest, value); m[key] = subMap; return // ... for a leaf path (no dot): m[dotPath] = value // <-- WHOLESALE OVERWRITE }workflow/mapTaskOutputshas the identical pattern overoutput_mapping, so it is affected too.
Mechanism
Given this (real) mapping, where both entries resolve to destination key submission:
"input_mapping": {
"reference_number": "submission.reference_number",
"userform": "submission"
}
| Iteration order | setNestedKey effect |
Result |
|---|---|---|
submission.reference_number then submission |
nested write creates {submission:{reference_number}}, then leaf write overwrites the whole submission |
{submission: <userform>} — reference_number lost |
submission then submission.reference_number |
leaf write sets {submission:<userform>}, then nested write merges into it |
{submission: {<userform fields>, reference_number}} — correct |
Because range over a map randomizes order, which result you get varies per process/run.
To Reproduce
A minimal unit test in workflow/utils_test.go demonstrates the order-dependence directly (this is exactly what mapTaskInputs does, with the two possible orderings):
func TestSetNestedKey_OverlappingDestinationsAreOrderDependent(t *testing.T) {
userform := map[string]any{"exporter_name": "Acme"}
// Order A: nested first, leaf second -> reference_number is DROPPED
a := map[string]any{}
setNestedKey(a, "submission.reference_number", "REF-123")
setNestedKey(a, "submission", userform)
// a == {"submission": {"exporter_name": "Acme"}} (reference_number gone)
// Order B: leaf first, nested second -> both present
b := map[string]any{}
setNestedKey(b, "submission", userform)
setNestedKey(b, "submission.reference_number", "REF-123")
// b == {"submission": {"exporter_name": "Acme", "reference_number": "REF-123"}}
// a and b differ -> result depends on iteration order, which range() randomizes.
}
To see the flakiness end-to-end:
- Define a TASK node with the
input_mappingabove (a leaf destinationsubmissionand a nested destinationsubmission.reference_number). - Activate the node repeatedly across multiple process runs.
- Observe that
inputs["submission"]intermittently is missingreference_number, with no error.
Expected Behavior
- Expected: applying a node's mapping is deterministic; overlapping/parent-child destination paths either merge predictably or are rejected as a misconfiguration.
- Actual: the outcome depends on Go's randomized map iteration order; a parent-leaf assignment can silently clobber a sibling nested assignment (or vice-versa).
Version
github.com/OpenNSW/core at commit 66c100d4aa8a (line numbers in the references above are from this commit). Consuming project currently pins v0.0.0-20260612190859-9c090fdca33c; the logic is unchanged there.
Environment Details (with versions)
Go (workflow engine github.com/OpenNSW/core). Language-level behavior (randomized map iteration order) means this reproduces on any Go version / OS.
Additional Context
Impact / severity
- Silent data loss in task inputs (and outputs) — e.g. a payload dispatched to an external government agency intermittently omits a field.
- Flaky / hard to diagnose: passes in most runs, fails in others, with no error.
- Triggered by an ordinary, reasonable-looking config; nothing warns the author.
Secondary observation
In the "good" ordering, getNestedKey returns the live source value (not a copy), so the subsequent nested setNestedKey mutates the original WorkflowVariables entry as a side effect (aliasing). This is independently undesirable under deterministic replay and worth addressing alongside the ordering fix.
Suggested fixes
- Deterministic application order (primary). Before applying, sort mapping entries by destination-path depth ascending (fewer dots first) so a parent/leaf assignment is always applied before its nested children. This makes the parent-then-child merge the defined behavior and removes the randomness.
- Validate overlapping destinations (guard). At definition load/validation time, detect when one destination path is a prefix of another (or two entries share a leaf destination) and reject/warn, so misconfigurations fail loudly instead of silently.
- Copy-on-read for mapped values. Deep-copy the value returned by
getNestedKeybefore inserting it into the target bag, to remove the aliasing side effect noted above.
Recommend (1) + (2) together; (3) closes the related mutation hazard. The same fix should be applied to both mapTaskInputs and mapTaskOutputs.
Notes
Surfaced while authoring FCAU sub-workflow configs in OpenNSW/nsw-srilanka (six configs/fcau/**/workflow.json nodes use the overlapping submission / submission.reference_number mapping to flatten the OGA payload envelope). The configs are a legitimate use of nested destination paths; the defect is the engine's nondeterministic, overwrite-vs-merge handling.
- Lingua principale
- Go
- Stelle
- 6
- Fork
- 8
- Merge medio
- 21h 36m
- PR unite (30g)
- 42
Preparare l'ambiente
- Nessun Dockerfile né file Docker Compose
- Ha un modello di pull request
- Leggi la guida per i contributori
Come iniziare
- Leggi tutta la issue e poi la guida ai contributi del progetto.
- Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
- Fai un fork del repository e lavora su un branch.
- Apri una pull request che faccia riferimento al numero della issue.
Altre issue di OpenNSW/core
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 74/100
I maintainer di solito rispondono entro 1 giorno
-
payment: test fixtures hardcode real gateway names (govpay/lankapay) instead of generic placeholdersAperta
Difficoltà 2/5 1-3 ore Idoneità per principianti 88/100
I maintainer di solito rispondono entro 1 giorno
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 72/100
I maintainer di solito rispondono entro 1 giorno
-
nsw-sl cleanup Type/Improvement
Difficoltà 3/5 1-2 giorni Idoneità per principianti 48/100
I maintainer di solito rispondono entro 1 giorno
-
remote: let the host inject a logger, and log with one consistent policyForse già presa @mushrafmim l’ha presa 5 giorni fa. ApertaType/Improvement
Difficoltà 4/5 3-5 giorni Idoneità per principianti 22/100
I maintainer di solito rispondono entro 1 giorno
Tutte le issue di OpenNSW/core
Issue simili
-
agent-research-finding chore
Difficoltà 2/5 1-3 ore Idoneità per principianti 78/100
jordansmall/spindrift#7370 ·
I maintainer di solito rispondono entro 1 giorno
-
☢️ Bug v3
Difficoltà 2/5 1-3 ore Idoneità per principianti 82/100
I maintainer di solito rispondono entro 1 giorno
-
The repo's gates check is clean and runs need no swapsForse già presa @turbot-agent-fleet l’ha presa oggi. Apertaagent:ready
Difficoltà 2/5 1-3 ore Idoneità per principianti 64/100
turbot/tailpipe#677 · 2 commenti ·
I maintainer di solito rispondono entro 1 giorno
-
needs-kind needs-triage
Difficoltà 2/5 1-3 ore Idoneità per principianti 72/100
I maintainer di solito rispondono entro 2 giorni
-
enhancement
Difficoltà 2/5 1-3 ore Idoneità per principianti 70/100
crossplane-contrib/provider-keycloak#806 ·
I maintainer di solito rispondono entro 1 giorno