Backfill requests omit the first logical date

Open Beginner friendly
#7 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
82/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
python
Domain
api, backend

Research direction

Start in plugins/linkedin/airflow/backfill/utils/request_handling.py around the validator at lines 102-106, then reproduce the one-date and three-date requests through the API creation service with a fresh SQLite store. Ensure the stored and returned logical_dates include the complete window while empty windows remain rejected.

Written by the indexing model from the issue text.

Description

Backfill requests omit the first date from the stored and returned logical_dates. A one-date request records an empty list, while a three-date request records only its last two dates.

I reproduced this with Airflow 2.10.1, Python 3.9.6 and the plugin's 1.0.4 validator, which is unchanged in current main.

For a daily DAG starting on January 1:

from airflow import DAG
from airflow.operators.empty import EmptyOperator
from airflow.utils import timezone

with DAG(
    dag_id="date_retention_multiple",
    start_date=timezone.parse("2024-01-01"),
    schedule="@daily",
    catchup=False,
) as dag:
    EmptyOperator(task_id="noop")

The creation request was:

{
  "dag_id": "date_retention_multiple",
  "start_date": "2024-01-01",
  "end_date": "2024-01-03",
  "delete_duration_days": 30
}

The API response and persisted row both contained:

["2024-01-02T00:00:00+00:00", "2024-01-03T00:00:00+00:00"]

They should also contain 2024-01-01T00:00:00+00:00. Restricting the request to January 1 instead records [].

The validator calls any() on Airflow's timetable iterator before collecting its dates, so the emptiness check consumes the first item. Checking the existing materialized date list instead preserves the complete window. I have a small correction that also keeps empty-window rejection.

The reproducer used real DAG parsing, the API creation service and fresh SQLite stores initialized through the normal migration/plugin setup. This establishes the request-metadata defect; it does not claim that the scheduler skips execution of the first date.

Dominant language
Python
Stars
10
Forks
4
PR merge metrics
No merged PRs in 30d

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from linkedin/Li-Airflow-Backfill-Plugin

All issues in linkedin/Li-Airflow-Backfill-Plugin

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.