Repeated SimPaths runs with the same random seed can produce different benefit-receipt flags

Start with DonorTaxImputation.evaluate() and its receipt draw, then trace the parallel labour calculations in LabourMarket and how BenefitUnit applies the tax wrapper. Run the DonorReceiptOrderProbe reproducer described in the issue to confirm that reversing subject order changes receipt flags. Before implementing a fix, resolve the listed design questions about per-unit random draws, repeated evaluations and which option's flags are retained; done means receipt flags remain reproducible without changing unrelated model results.

Summary

With the same prepared inputs, model options and seeds, repeated runs using SimPaths’ Java MultiRun launcher can produce different yBenUCReceivedFlag and yBenLegacyReceivedFlag values in BenefitUnit.csv.

DonorTaxImputation.evaluate() uses a shared seeded random generator to assign UC receipt. It is reached from parallel benefit-unit labour calculations. The seed fixes the sequence of draws, but thread scheduling can change which unit receives each draw. Legacy-benefit receipt also depends on the UC flag.

This appears to be a remaining issue after commit 7e8a3a71e9, which replaced Math.random() with SimulationEngine.getRnd().nextDouble() to make these flags reproducible. The shared seeded draw still depends on evaluation order.

Source checked

Checked both upstream branches after a fresh fetch from https://github.com/simpaths/SimPaths.git on 25 September 2026:

The shared receipt draw and parallel labour calculations remain in both branches. main is also tagged 2026.09.01. The newer develop revision has not changed the affected calculation path.

• Shared receipt draw and dependent legacy flag.
• Parallel labour calculations, including updateLabourSupplyAndIncome at line 262.
• Benefit-unit tax wrapper.
• The same receipt draw and parallel labour update in the fetched develop revision.

I confirmed that the relevant source files (DonorTaxImputation, DonorKeys, TaxEvaluation, BenefitUnit, LabourMarket, Innovations and Expectations) are byte-for-byte identical in both upstream branches and my tested build. I have not separately run a full simulation using the latest develop branch. The full simulation was run on a development build with other changes; it was not a clean upstream build. To narrow the result, I also compiled the upstream DonorTaxImputation.java directly and ran the synthetic reproducer below with that class first on the classpath. Its remaining dependencies came from the development build. Both versions give the same result.

Observed full-run differences

Two runs using SimPaths’ Java MultiRun launcher used copies of the same prepared UK public training inputs, a population target of 2,000, years 2019–2020, and seeds 606, 607 and 608. The emitted options.txt files were identical for each seed. CSV comparisons ignore only the run field and row ordering.

All these differences occurred in 2020. Benefit-unit entity/time keys and every other benefit-unit value matched. The other eight simulation output CSV files matched for all three seeds, after ignoring row order and the run field.

In the configuration I tested, differences were limited to the two benefit-receipt flags. Other configurations and longer simulations have not been checked. I did not record which benefit unit received each random draw during the full simulations. However, the small test below confirms that changing the calculation order can change receipt flags even when the seed stays the same.

Small reproducer: no database or input files needed

The program below calls the real DonorTaxImputation.evaluate() using ten synthetic, equally weighted donors. All donors have identical income; half receive UC and half receive legacy benefits. It evaluates 16 labelled targets with seed 606, repeats them in the same order, then reverses their evaluation order. Labels identify results only; they do not change tax inputs.

This is a deterministic demonstration of scheduling sensitivity, so it does not need to wait for a particular thread interleaving. Same-seed target results should remain stable when evaluation order changes under a reproducible draw assignment. This test demonstrates the existing problem: it finishes normally when changing the calculation order produces different receipt flags. It is not intended to verify a fix.

Using Java 25 and a packaged SimPaths multirun.jar, save the source below as DonorReceiptOrderProbe.java in the same directory as the JAR, then run:

java -Djava.awt.headless=true --class-path multirun.jar DonorReceiptOrderProbe.java

Observed with OpenJDK 25.0.4.1:

Same seed and same order repeat exactly: true
Same seed with reversed subject order: 10/16 receipt flags differ
Disposable income remains 120/week for all subjects in both orders

The source sets static model data and should be run in its own JVM, as above. It performs no simulation, database preparation or web operations.

/* 
 * Synthetic in-memory diagnostic of SimPaths donor receipt ordering; no database or model inputs.
 */
import java.util.*;
import java.util.stream.IntStream;
import microsim.engine.SimulationEngine;
import org.apache.commons.math3.util.Pair;
import org.apache.commons.lang3.tuple.Triple;
import simpaths.data.Parameters;
import simpaths.model.taxes.*;

public class DonorReceiptOrderProbe {
    static class Keys extends DonorKeys {
        public int getKey(int regime) { return 0; }
        public Integer getSimYear() { return 2015; }
        public Integer getPriceYear() { return 2015; }
        public Double getOriginalIncomePerWeek() { return 100.0; }
        public Double getSecondIncomePerWeek() { return 0.0; }
        public Double getChildcareCostPerWeek() { return 0.0; }
        public boolean isLowIncome(int regime) { return true; }
    }
    static class Policy extends DonorTaxUnitPolicy {
        final int uc;
        Policy(int uc) { this.uc = uc; }
        public double getOriginalIncomePerMonth() { return 100.0 * Parameters.WEEKS_PER_MONTH; }
        public double getNormalisedOriginalIncomePerMonth(int year) { return Parameters.asinh(getOriginalIncomePerMonth()); }
        public double getDisposableIncomePerMonth() { return 120.0 * Parameters.WEEKS_PER_MONTH; }
        public double getBenMeansTestPerMonth() { return 20.0 * Parameters.WEEKS_PER_MONTH; }
        public double getBenNonMeansTestPerMonth() { return 0.0; }
        public Integer getReceivesUC() { return uc; }
        public Integer getReceivesLegacyBenefit() { return 1 - uc; }
    }
    static int[][] evaluate(boolean reverse) {
        SimulationEngine.getRnd().setSeed(606);
        int[][] result = new int[16][2];
        for (int i = 0; i < 16; i++) {
            int subject = reverse ? 15-i : i;
            DonorTaxImputation imputation = new DonorTaxImputation(new Keys());
            imputation.evaluate();
            if (Math.abs(imputation.getDisposableIncomePerWeek()-120.0)>1e-8)
                throw new AssertionError("Unexpected imputed income");
            result[subject][0] = imputation.getReceivedUC();
            result[subject][1] = imputation.getReceivedLegacyBenefit();
        }
        return result;
    }
    public static void main(String[] args) {
        SimulationEngine.getInstance();
        Parameters.BASE_PRICE_YEAR = 2015;
        Parameters.taxDonorUpratingByWage = false;
        Parameters.EUROMODpolicyScheduleSystemYearMap.clear();
        Parameters.EUROMODpolicyScheduleSystemYearMap.put(2015, new Pair<>("synthetic",2015));
        Map<MatchFeature,Map<Integer,Integer>> counters = new EnumMap<>(MatchFeature.class);
        for (MatchFeature feature: MatchFeature.values()) {
            Map<Integer,Integer> regimes = new HashMap<>();
            for (int r=0;r<Parameters.TAXDB_REGIMES;r++) regimes.put(r,1);
            counters.put(feature, regimes);
        }
        Parameters.setTaxdbCounter(counters);
        List<DonorTaxUnit> donors = new ArrayList<>();
        for (int i=0;i<10;i++) {
            final Policy policy = new Policy(i<5 ? 1:0);
            DonorTaxUnit donor = new DonorTaxUnit() {
                public DonorTaxUnitPolicy getPolicyBySystemYear(int year) { return policy; }
            };
            donor.setId(i); donor.setWeight(1.0); donors.add(donor);
        }
        Parameters.setDonorPool(donors);
        Parameters.setTaxdbReferences(Map.of(Triple.of(2015,0,0), IntStream.range(0,10).boxed().toList()));
        int[][] forward = evaluate(false), repeated = evaluate(false), reversed = evaluate(true);
        if (!Arrays.deepEquals(forward,repeated)) throw new AssertionError("Fixed order was not repeatable");
        int changed=0;
        for (int i=0;i<16;i++) if (!Arrays.equals(forward[i],reversed[i])) changed++;
        System.out.println("Same seed and same order repeat exactly: true");
        System.out.println("Same seed with reversed subject order: " + changed + "/16 receipt flags differ");
        System.out.println("Disposable income remains 120/week for all subjects in both orders");
        if (changed==0) throw new AssertionError("Ordering defect not reproduced");
    }
}

Proposed direction requiring maintainer agreement

One possible fix is to assign each benefit unit a reproducible random number for each simulation year and supply it to the tax-and-benefit calculation. This would make receipt decisions independent of the order in which units are processed. This fits the existing per-unit Innovations design, but the following choices need confirmation:

1. Should the same random number be reused when evaluating alternative labour choices, employment alignment adjustments and repeated tax calculations for the same benefit unit in the same year? How should it evolve across years and relate to the donor-selection draw?
2. Some optimisation calculations use only the calculated income, but still generate benefit-receipt flags and consume a random number. Should they skip generating these unused flags, or use a reproducibly assigned random number? For example, Expectations calls TaxEvaluation and uses the calculated income and donor-match information, but does not use the generated receipt flags in that calculation.
3. When SimPaths selects a working-hour option, does it also keep the benefit-receipt flags calculated for that option? The code assigns these flags while evaluating the different options. After selecting an option, it restores that option’s income and donor-match information, but I did not see equivalent code restoring its receipt flags. Could the flags therefore belong to a different option? This should be checked both when results are calculated directly and when previously calculated results are reused. I have not confirmed this as a separate bug.

The value currently used for donor selection cannot always be used to decide benefit receipt: when donor averaging is enabled, it is -1, meaning ‘average over donors’, rather than a random number. Also, generating an additional random number each year through Innovations would change the numbers subsequently used by other calculations. The fix should avoid changing unrelated model results where possible.

Allowing only one thread at a time to take a random number (i.e. synchronising) would not solve the problem: the order in which benefit units reach the generator could still vary. Running the labour calculations one unit at a time (i.e. serialising), in a fixed order, could help confirm the cause, but may slow the simulation. I have not checked whether that would resolve the problem everywhere it occurs.

When checking a possible fix, it may be useful to compare results with benefit units processed in different orders and in parallel, as well as repeat full simulations with the same seed. Testing both donor averaging and individual donor selection could also help check that the fix works in both modes.