H2 — Pandemic disruption accelerated a decline that began earlier | Student decline: More than pandemic learning loss alone

Step 2 — Define exactly what will count as evidence

Decide which concepts in the research question need measurable definitions.

For this hypothesis, the major concepts are:

Achievement

• Reading achievement;
• Mathematics achievement;
• potentially grade-specific achievement;
• potentially subgroup or distributional measures if available.

Pre-pandemic decline

• A negative achievement trend before 2020;
• potentially a change in slope between an earlier period and approximately 2013–2019.

Pandemic disruption

• A departure from the expected pre-pandemic trajectory around 2020–2022.

Recovery

• The trajectory after testing resumes;
• whether districts return toward their previous trajectory, remain below it, or continue deteriorating.

At this stage, don't require every possible outcome. Start with the variables actually needed to answer H2.

Step 3 — Understand and document the data

This should happen before statistical modeling.

Create a data dictionary for each primary dataset containing at minimum:

Also document the unit of observation.

For example:

One row = district × year × subject × grade

or whatever the source actually provides.

Then answer basic feasibility questions:

• Which years overlap?
• Is 2013 directly observed?
• Is 2019 directly observed?
• How are 2020 and 2021 represented?
• When does post-pandemic testing resume?
• Are scores on a common scale?
• Are district identifiers stable?
• Are there districts entering/leaving the dataset?
• Are there changes in assessment methodology?

This step prevents a huge amount of bad analysis later.

Step 4 — Develop testable statistical hypotheses

Rather than treating the original H2 statement as one giant hypothesis, break it into several tests.

Test A — Was there a pre-pandemic decline?

Null hypothesis H0A​:

Achievement did not decline during the pre-pandemic period.

Conceptually:
βpre-COVID​≥0
Alternative HAA​:

Achievement declined during the pre-pandemic period.

βpre-COVID​<0

Test B — Did the trajectory change before COVID?

This addresses the proposed 2013 change point.

H0B​:

The achievement trend after approximately 2013 was not meaningfully different from the earlier trend.

β2003−2012​=β2013−2019​
HAB​:

The 2013–2019 trend was more negative than the earlier trend.

β2013−2019​<β2003−2012​
The exact breakpoint should ultimately be tested rather than assumed.

Test C — Was there an additional pandemic shock?

H0C​:

Achievement after the pandemic remained consistent with the pre-pandemic trajectory.

HAC​:

Achievement after the pandemic was significantly below what would have been expected from the pre-pandemic trajectory.

This is where an interrupted-time-series or segmented-regression framework becomes useful.

Test D — Were pre-pandemic declines associated with post-pandemic outcomes?

At the district level:

H0D​:

A district's pre-pandemic achievement trend has no relationship with its pandemic or post-pandemic performance.

HAD​:

Districts experiencing greater pre-pandemic declines tend to experience poorer post-pandemic achievement or recovery.

That doesn't establish causation, but it directly tests the “accelerated rather than created” idea.

Step 5 — Conduct exploratory data analysis

Before fitting formal models, learn what the data look like.

A useful first-pass sequence would be:

A. Coverage

Count observations by:

• Year;
• state;
• district;
• subject;
• grade.

Look for gaps and changes in coverage.

B. Trend plots

Plot:

• National/pooled reading achievement over time;
• National/pooled mathematics achievement over time;
• Individual state trajectories;
• representative district trajectories.

Mark important periods visually:

• earlier baseline;
• proposed pre-pandemic decline;
• pandemic disruption;
• recovery.

C. Distribution of district trends

Calculate preliminary district-level slopes and examine their distribution.

For example:
Slope2013−2019​
This helps answer:

Was the apparent decline widespread, or driven by a relatively small number of districts?

D. Missingness and unusual observations

Investigate:

• Missing years;
• sudden district jumps;
• district consolidations;
• changes in sample composition;
• unusually large year-to-year changes.

EDA should be treated as question-generating, not hypothesis confirmation.

Step 6 — Choose analysis methods based on what the data support

Only after Steps 1–5 should the researcher settle on the formal methodology.

For H2, a sensible progression might be:

Analysis 1 — Simple period trends

Estimate separate trends for:
Early period2013–2019Post-pandemic
This provides an understandable baseline.

Analysis 2 — Segmented regression

Fit a model allowing the slope to change over time.

Conceptually:
Yt​=β0​+β1​Time+β2​PostBreak+β3​TimeAfterBreak+ϵt​
Potential breakpoints could include:

• a data-derived pre-pandemic breakpoint;
• the pandemic disruption.

This lets you distinguish:

underlying trend

from

level shock

from

change in trajectory.

Analysis 3 — Change-point detection

Rather than simply declaring 2013 the start of the decline, ask:

Where does the data indicate that the trajectory changed?

If an estimated breakpoint occurs around 2013, that strengthens the original proposition.

If it occurs in 2015, 2011, or nowhere at all, that is scientifically important too.

Analysis 4 — District-level persistence

Estimate each district's pre-pandemic trajectory and compare it with later outcomes.

For example:
PostPandemicOutcomei​=α+βPrePandemicSlopei​+ϵi​
Later models could introduce relevant controls if justified.

Step 7 — Compare competing explanations

This is one of the most important parts of H2.

Instead of merely asking whether COVID hurt achievement, compare models representing different explanations.

Model 1 — Pandemic-only explanation

Achievement is relatively stable beforehand, followed by a COVID-era disruption.

Model 2 — Pre-existing decline

Achievement begins declining before COVID, without a distinct additional pandemic effect.

Model 3 — Pre-existing decline + pandemic shock

Achievement begins declining earlier and experiences an additional pandemic-era disruption.

Then ask:

Which explanation fits the observed data best?

That gets much closer to the scientific question than simply testing whether 2022 scores were below 2019 scores.

Step 8 — Perform targeted robustness checks

Robustness checks should follow the core analysis rather than becoming another enormous research project.

Repeat the central tests where practical:

• Reading vs. mathematics;
• Different grade levels;
• States vs. districts;
• Alternative pre-pandemic starting dates;
• Alternative breakpoint specifications;
• Balanced panel only vs. all available districts;
• SEDA vs. Education Scorecard estimates;
• NAEP as an independent national/state comparison.

If the conclusion disappears whenever one modeling choice changes, that is important evidence about its uncertainty.

Step 9 — Interpret the results against the original research question

At the end, return explicitly to H2.

Possible conclusions might be:

Supported

Evidence indicates a meaningful pre-pandemic deterioration, followed by an additional pandemic shock.

Partially supported

There is evidence of pre-pandemic deterioration, but the pandemic effect or persistence relationship is weak or inconsistent.

Not supported

Achievement appears broadly stable before COVID, with the primary deterioration occurring during the pandemic.

Inconclusive

Data coverage, measurement changes, uncertainty, or conflicting sources prevent distinguishing the explanations.

That last category matters. A research issue should not force every analysis into “confirmed/rejected.”

Step 10 — Document what was learned and what should happen next

The subissue should end with a concise record of:

• Datasets used;
• variables used;
• exclusions;
• data dictionary;
• analysis code/notebooks;
• assumptions;
• statistical results;
• visualizations;
• limitations;
• conclusion;
• unanswered questions;
• recommended follow-up analysis.

This creates a reproducible research trail rather than just a collection of charts.