build_from_populations leaves each holder bound to the simulation that made it

Start by reading Simulation.build_from_populations, Holder._set, Holder._is_input, and the holder handling in Simulation.clone and branch creation; the issue gives a core-only reproduction for each case. Trace how holders and input records are rebound during a rebuild, then run the reproductions to check that the receiver, its clone, and its branch all retain and export supplied inputs correctly.

Simulation.build_from_populations(populations) points each population at the new simulation (link_to_entities_instances), but each holder keeps holder.simulation, the simulation that made it. The holder uses that simulation to decide whether a store is an input and to record it (Holder._set reads its _user_input_contexts and adds to its _user_input_keys), to answer Holder._is_input, and in delete_arrays and fast-cache eviction. So after a rebuild from another simulation's populations, an input set on the new simulation goes into the other simulation's record. The new simulation and its clones then neither export it nor count it as an input.

Reproduction (core only, 3.33.12; build_from_populations reads the same on master at 04cda03b7, 3.33.13)

from policyengine_core.country_template import CountryTaxBenefitSystem
from policyengine_core.simulations import SimulationBuilder

tbs = CountryTaxBenefitSystem()
households = {"h": {"parents": ["a"]}}
def build(salary):
    return SimulationBuilder().build_from_entities(
        tbs, {"persons": {"a": {"salary": salary}}, "households": households}
    )

donor = build({"2018-01": 1000})
receiver = build({"2018-01": 500})
receiver.build_from_populations(donor.populations)
holder = receiver.populations["person"].get_holder("salary")
print("population bound to receiver:", receiver.populations["person"].simulation is receiver)
print("holder bound to receiver:", holder.simulation is receiver, "| to donor:", holder.simulation is donor)
receiver.set_input("salary", "2018-02", [700])
key = ("salary", "default", __import__("policyengine_core").periods.period("2018-02"))
print("receiver records the input set on it:", key in receiver._user_input_keys)
print("donor records it instead:", key in donor._user_input_keys)
print("receiver exports:", sorted(c for c in receiver.to_input_dataframe().columns if c.startswith("salary")))
clone = receiver.clone()
print("clone exports:", sorted(c for c in clone.to_input_dataframe().columns if c.startswith("salary")))

Output:

population bound to receiver: True
holder bound to receiver: False | to donor: True
receiver records the input set on it: False
donor records it instead: True
receiver exports: ['salary__2018-01']
clone exports: ['salary__2018-01']

The receiver still exports salary__2018-01 because its own record of the input it held before the rebuild (#605) now reads the donor's holder, which stores 1,000 there.

A clone or branch of the receiver then loses the donor's inputs too. Simulation.clone copies the receiver's _user_input_keys, and Holder.clone binds each copied holder to the clone, so the copy reads a record that never held the donor's keys:

"""Core only: a clone of a rebuilt simulation loses the donor's inputs."""
from policyengine_core.country_template import CountryTaxBenefitSystem
from policyengine_core.periods import period
from policyengine_core.simulations import SimulationBuilder

tbs = CountryTaxBenefitSystem()
households = {"h": {"parents": ["a"]}}


def build(persons):
    return SimulationBuilder().build_from_entities(
        tbs, {"persons": {"a": persons}, "households": households}
    )


donor = build({"salary": {"2018-01": 1000}})
receiver = build({})
receiver.build_from_populations(donor.populations)
january = period("2018-01")
for label, simulation in (("receiver", receiver), ("clone", receiver.clone()), ("branch", receiver.get_branch("b"))):
    holder = simulation.populations["person"].get_holder("salary")
    print(
        label,
        "| stores:", holder.get_array(january, simulation.branch_name),
        "| counts it as an input:", holder._is_input(january, simulation.branch_name),
        "| exports:", sorted(c for c in simulation.to_input_dataframe().columns if c.startswith("salary")),
    )

receiver | stores: [1000.] | counts it as an input: True | exports: []
clone | stores: [1000.] | counts it as an input: False | exports: []
branch | stores: [1000.] | counts it as an input: False | exports: []

The receiver counts the value as an input only because its holder still reads the donor's record, and it does not export it; the clone and the branch treat it as calculated, so code that defers to a supplied value (policyengine-uk's housing benefit income override does) recalculates it.

Related

• #605: the rebuild keeps the record of the replaced holders' inputs.
• PolicyEngine/policyengine-uk#2234 works around these in its Simulation.build_from_populations (adopt_population_records in policyengine_uk/utils/supplied_inputs.py): it binds each holder from another simulation to the new one, carries that simulation's input records for the values the holder still stores, drops the values it calculated, and forgets the records of values the rebuild removed.

A review of that PR found the clone and branch case.