Multiple rune_deserialize / rune_serialize failures block CDM TradeState roundtrip

Start with BaseDataClass.rune_deserialize(), resolve_references(), and ComplexTypeMetaDataMixin.serialise()/deserialize(); reproduce the five cases using a standard FpML IRS TradeState. Done means a valid deserialize-to-serialize roundtrip with no unresolved references or validation errors, plus a documented decision for flat versus nested @type input.

After PR #38 / rune-runtime 2.2.1 fixed one class of resolve_references() failures,
we identified 4 additional bugs that together prevent a complete roundtrip of a
standard CDM TradeState (FpML IRS fixed/float). All bugs are in rune.runtime and
affect all finos-cdm versions since they all require rune.runtime>=2.2.0.

Bug 1 — resolve_references() does not resolve list-TYPED reference fields
AncillaryParty.partyReference is typed as list[Party | BaseReference | None].
After resolve_references(), the list still contains UnresolvedReference instead
of the resolved Party. PR #38 fixed the case where a list contains objects that
have reference fields; it did not fix the case where the field itself is a list
of references.

Bug 2 — rune_deserialize() silently drops payout data when @type uses flat CDM format
The FINXIS/FpML reference JSON uses a flat @type discriminator on choice fields:
"rateSpecification": {"@type": "cdm.product.asset.FixedRateSpecification", ...}
The Python model expects a nested form:
"rateSpecification": {"FixedRateSpecification": {...}}
When loaded via rune_deserialize() or model_validate(), the flat form is silently
accepted but the contents are dropped entirely — the Payout wrapper ends up with
all 8 sub-fields set to None, failing the oneOf condition on rune_serialize().

Bug 3 — rune_serialize() rejects None on optional reference-capable fields
Even with strict=False, the serializer raises a ValidationError when an optional
reference-capable field (e.g. quantityReference, priceSchedule,
dateAdjustmentsReference) is None. These fields are optional by definition in the
model but the serializer's reference validator does not accept None for them.

Bug 4 — rune_serialize() crashes with AttributeError on list-typed reference fields
When validate_model=False is passed to bypass Bug 3, the serializer crashes:
AttributeError: 'list' object has no attribute 'serialise_meta'
This occurs on list-typed reference fields (same category as Bug 1) — the
serializer calls .serialise_meta() directly on the list without guarding for
list or None values.

Bug 5 — Scoped key registration ordering: @ref resolves before @data registers the key
In the observable block, a rateOption @ref:scoped references InterestRateIndex-1,
but the observable @data that defines that key is processed after the reference
during rune_deserialize(). This causes a KeyError at resolution time:
KeyError: 'InterestRateIndex-1'
This is an ordering dependency in the key registration pass that cannot be worked
around externally.

Environment:
finos-cdm: any version (all require rune.runtime>=2.2.0 where the bugs live)
rune-runtime == 2.2.1
pydantic == 2.13.5
Python 3.11, Windows

Use any standard FpML IRS fixed/float TradeState JSON (e.g. from the FINXIS CDM
viewer at https://finxissolutions.com/viewer/).

--- Bug 1: list-typed reference field not resolved ---

from rune.runtime.base_data_class import BaseDataClass
ts = BaseDataClass.rune_deserialize(raw, namespace_prefix='finos',
validate_model=False)
ts.resolve_references()
print(ts.trade.ancillaryParty[0].partyReference)

[] <-- should be a Party object

--- Bug 2: flat @type payout data silently dropped ---

ts = BaseDataClass.rune_deserialize(raw, namespace_prefix='finos',
validate_model=False)
p0 = ts.trade.product.economicTerms.payout[0]
print({k: v for k, v in p0.dict.items() if v is not None
and not k.startswith('__')})

{} <-- all payout fields are None; data was silently dropped

--- Bug 3: serializer rejects None on optional reference fields ---

After rewriting payouts into nested form and applying Bug 1 workaround:

ts.rune_serialize(strict=False)

ValidationError: trade.product.economicTerms.payout.0.InterestRatePayout

.priceQuantity.quantityReference

Expected either ResolvablePriceQuantity or dict but got NoneType

--- Bug 4: serializer crashes on list-typed reference fields ---

ts.rune_serialize(validate_model=False)

AttributeError: 'list' object has no attribute 'serialise_meta'

in ComplexTypeMetaDataMixin.serialise()

--- Bug 5: scoped key registration ordering (KeyError) ---

After rewriting payouts into nested form:

ts2 = BaseDataClass.rune_deserialize(nested_data, namespace_prefix='finos',
validate_model=False)

KeyError: 'InterestRateIndex-1'

rateOption @ref:scoped resolves before observable @data registers the key

--- Workarounds we implemented (for Bugs 1, 3, 4) ---

For Bug 1: walk ancillaryParty[].partyReference[] after resolve_references()
and replace UnresolvedReference using __rune_metadata['@key:external'] as
the lookup key.

For Bugs 3+4: monkeypatch ComplexTypeMetaDataMixin.serialise() and
.deserialize() BEFORE importing finos (validators are captured at import time):
- serialise: guard for obj is None -> return None; obj is list -> recurse
- deserialize: guard for obj is None -> return None

Bug 2 and Bug 5 have no external workaround — they require fixes inside
rune_deserialize() itself.

--- Net result ---
A complete roundtrip of a standard CDM TradeState is not currently possible
using rune_deserialize() + rune_serialize(). We have worked around this by
building CDM output as plain Python dicts in the reference file shape,
bypassing the library entirely for the build path.

Expected Result:

A complete roundtrip should be possible for any valid CDM TradeState JSON:

ts = BaseDataClass.rune_deserialize(raw, namespace_prefix='finos')
result = ts.rune_serialize()

Bug 1 — resolve_references() should resolve all reference fields regardless of
whether the field is a scalar reference or a list of references. After calling
resolve_references(), no UnresolvedReference objects should remain anywhere in
the object graph for keys that are defined in the same document.

Bug 2 — rune_deserialize() should correctly route the flat @type discriminator
form into the appropriate choice field. A Payout loaded from:
{"@type": "cdm.product.asset.InterestRatePayout", "payerReceiver": {...}}
should result in payout.InterestRatePayout being populated, not silently dropped.
Alternatively, the documentation should explicitly state which JSON form the
deserializer expects (flat vs nested) so callers can pre-process accordingly.

Bug 3 — rune_serialize() should accept None for optional reference-capable fields.
A field declared as optional in the model should not raise a ValidationError
when it is absent, regardless of whether strict=True or strict=False is passed.

Bug 4 — ComplexTypeMetaDataMixin.serialise() should guard against being called
with a list or None value. When a reference field is list-typed, serialise()
should iterate the list and serialise each element individually.

Bug 5 — rune_deserialize() should register all scoped key definitions before
attempting to resolve any @ref:scoped references, regardless of document order.
A two-pass approach (collect all keys first, then resolve all references) would
fix the ordering dependency.

Actual Result:

Bug 1 — AncillaryParty.partyReference remains as UnresolvedReference after
resolve_references(). rune_serialize() then fails with:
ValidationError: trade.ancillaryParty.0.partyReference
Expected either Party or dict but got UnresolvedReference

Bug 2 — Payout data is silently dropped when loaded from flat @type JSON.
All 8 sub-fields of the Payout choice wrapper are None after loading.
rune_serialize() then fails with:
ConditionViolationError: One and only one of ('AssetPayout',
'InterestRatePayout', ...) should be set!

Bug 3 — rune_serialize(strict=False) raises:
ValidationError: trade.product.economicTerms.payout.0.InterestRatePayout
.priceQuantity.quantityReference
Expected either ResolvablePriceQuantity or dict but got NoneType
This field is optional in the model; None should be accepted.

Bug 4 — rune_serialize(validate_model=False) crashes with:
AttributeError: 'list' object has no attribute 'serialise_meta'
in ComplexTypeMetaDataMixin.serialise() when processing a list-typed
reference field.

Bug 5 — rune_deserialize() raises during reference resolution:
KeyError: 'InterestRateIndex-1'
The rateOption @ref:scoped is resolved before the observable @data block
has registered its InterestRateIndex-1 key, due to document traversal order.

Net result: no combination of rune_deserialize() / resolve_references() /
rune_serialize() parameters produces a successful roundtrip for a standard
CDM IRS TradeState that includes an ancillaryParty block and observable
references.

###Compatibility
All 5 fixes are internal to rune.runtime and should be non-breaking for
callers that currently work around these bugs or avoid the affected code paths.

Bug 1 fix: purely additive — resolving more references than before cannot
break existing code that only used resolved references.

Bug 2 fix: if the deserializer is updated to accept the flat @type form,
existing code that already pre-processes JSON into the nested form will
continue to work. However, the expected JSON input format should be
clearly documented to avoid ambiguity.

Bug 3 fix: accepting None on optional fields is consistent with the model
definition. No existing valid usage should be affected.

Bug 4 fix: adding None/list guards to serialise() is purely defensive.
Existing code that does not encounter list-typed reference fields is
unaffected.

Bug 5 fix: a two-pass key registration approach changes the internal
traversal order but the observable output (resolved references) should
be identical for all well-formed documents.

Release:

The fix should be released in rune.runtime (https://github.com/finos/rune-python-runtime),
not in finos-cdm. All finos-cdm versions depend on rune.runtime>=2.2.0, so a fix
in rune.runtime will benefit all finos-cdm versions simultaneously without requiring
a finos-cdm release.

Suggested priority order for rune.runtime releases:

High priority (blocks any TradeState with ancillaryParty or observable references):
Bug 1 — resolve_references() missing list-typed reference fields
Bug 5 — scoped key registration ordering (KeyError on InterestRateIndex)

Medium priority (blocks serialization of loaded TradeState):
Bug 4 — serialise() AttributeError on list-typed reference fields
Bug 3 — serialise() rejects None on optional reference-capable fields

Requires design decision on supported JSON input format before fixing:
Bug 2 — flat @type payout data silently dropped on rune_deserialize()
(fix depends on whether flat or nested @type form is the intended
input contract, which should also be documented)

Additional Context:

These bugs were discovered while attempting to validate CDM output for an
Interest Rate Swap mapping pipeline using FINXIS reference files as the
ground truth (https://finxissolutions.com/viewer/).

Related: rune-python-runtime issue #37 and PR #38, which fixed the
list-member reference resolution case. The bugs reported here are
follow-ons discovered after upgrading to rune-runtime 2.2.1.

Workarounds implemented for Bugs 1, 3, and 4:

Bug 1: post-resolve_references() walk of ancillaryParty[].partyReference[],
replacing UnresolvedReference using __rune_metadata['@key:external'] as
the lookup key against trade.party[].

Bugs 3 + 4: monkeypatching ComplexTypeMetaDataMixin.serialise() and
.deserialize() before importing finos (validators are captured at class
creation time, so the patch must precede any finos import):
serialise: guard obj is None -> return None; obj is list -> recurse
deserialize: guard obj is None -> return None

Bugs 2 and 5 have no external workaround and require fixes inside
rune_deserialize() itself.

Due to these blockers, our pipeline currently builds CDM output as plain
Python dicts matching the reference file shape, bypassing the library for
the build path entirely. We intend to reintroduce rune_serialize() as an
optional validation pass once these issues are resolved.