Backfill requests omit the first logical date
Nobody has claimed this yet.
Assessment
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Newbie friendliness
- 82/100
Research direction
Start in plugins/linkedin/airflow/backfill/utils/request_handling.py around the validator at lines 102-106, then reproduce the one-date and three-date requests through the API creation service with a fresh SQLite store. Ensure the stored and returned logical_dates include the complete window while empty windows remain rejected.
Written by the indexing model from the issue text.
Description
Backfill requests omit the first date from the stored and returned logical_dates. A one-date request records an empty list, while a three-date request records only its last two dates.
I reproduced this with Airflow 2.10.1, Python 3.9.6 and the plugin's 1.0.4 validator, which is unchanged in current main.
For a daily DAG starting on January 1:
from airflow import DAG
from airflow.operators.empty import EmptyOperator
from airflow.utils import timezone
with DAG(
dag_id="date_retention_multiple",
start_date=timezone.parse("2024-01-01"),
schedule="@daily",
catchup=False,
) as dag:
EmptyOperator(task_id="noop")
The creation request was:
{
"dag_id": "date_retention_multiple",
"start_date": "2024-01-01",
"end_date": "2024-01-03",
"delete_duration_days": 30
}
The API response and persisted row both contained:
["2024-01-02T00:00:00+00:00", "2024-01-03T00:00:00+00:00"]
They should also contain 2024-01-01T00:00:00+00:00. Restricting the request to January 1 instead records [].
The validator calls any() on Airflow's timetable iterator before collecting its dates, so the emptiness check consumes the first item. Checking the existing materialized date list instead preserves the complete window. I have a small correction that also keeps empty-window rejection.
The reproducer used real DAG parsing, the API creation service and fresh SQLite stores initialized through the normal migration/plugin setup. This establishes the request-metadata defect; it does not claim that the scheduler skips execution of the first date.
- Dominant language
- Python
- Stars
- 10
- Forks
- 4
- PR merge metrics
- No merged PRs in 30d
Getting set up
- Ships a Dockerfile or Docker Compose file
- No pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from linkedin/Li-Airflow-Backfill-Plugin
-
Difficulty 4/5 3-5 days Newbie friendliness 25/100
linkedin/Li-Airflow-Backfill-Plugin#5 · 7 comments ·
All issues in linkedin/Li-Airflow-Backfill-Plugin
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
GoogleCloudPlatform/professional-services-data-validator#1899 ·
Maintainers usually reply within 4 days
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
mohanraj00/entelos#200 ·
Maintainers usually reply within 1 day
-
Difficulty 1/5 Under an hour Newbie friendliness 68/100
-
sweep
Difficulty 2/5 1-3 hours Newbie friendliness 85/100
bketelsen/truenas-mcp#77 ·
Maintainers usually reply within 1 day
-
Console writes URL credentials to the local log, though AGENTS.md says secrets never reach diskOpen
Difficulty 1/5 Under an hour Newbie friendliness 86/100
sandbox-quantum/switch#2163 ·
Maintainers usually reply within 1 day