Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

`global_substream_cursor`: configured `lookback_window` not applied to `should_be_synced` filter, causing silent data loss

Closed
  • Is something wrong?
#1,014 2 comments 2 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

@bazarnov is already working on this.

Since Sep 28, 2026.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
55/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Quiet
Tech stack
python
Domain
tooling

Research direction

Trace ConcurrentPerPartitionCursor._get_cursor() and _create_cursor() in airbyte_cdk/sources/declarative/incremental/concurrent_partition_cursor.py, then follow ConcurrentCursor.init() and should_be_synced() in airbyte_cdk/sources/streams/concurrent/cursor.py. Compare the configured DatetimeBasedCursor lookback with the runtime lookback, and verify that a one-day window prevents records just behind the global cursor from being dropped.

Written by the indexing model from the issue text.

Description

bug

Community report

Reported by a user in the Airbyte community Slack: https://airbytehq.slack.com/archives/C021JANJ6TY/p1776936626673869

The user has an Intercom connector with hourly syncs, lookback_window = 1 day, and conversation_parts in incremental append+deduped mode. They found that statistics.count_conversation_parts on conversations consistently exceeds the number of parts in their destination table. Manual verification against the Intercom API confirmed the API returns the correct number of parts but Airbyte drops some during sync.

Summary

When a substream uses global_substream_cursor: true + is_client_side_incremental: true, the configured lookback_window on the DatetimeBasedCursor is not applied to the should_be_synced client-side filter. Instead, the filter uses the runtime_lookback_window (previous sync's duration in seconds), which is typically only a few minutes. This causes parts whose updated_at is even slightly behind the global cursor to be silently dropped.

Affected code path

  1. ConcurrentPerPartitionCursor._get_cursor() creates a cursor using self._lookback_window, which is the runtime lookback (previous sync duration in seconds), loaded from state at airbyte_cdk/sources/declarative/incremental/concurrent_partition_cursor.py line 474:

    self._lookback_window = int(stream_state.get("lookback_window", 0))

  2. This value is passed to _create_cursor() at line 596-600:

    if self._use_global_cursor:
    return self._create_cursor(
        self._global_cursor,
        self._lookback_window if self._global_cursor else 0,
    )

  3. In model_to_component_factory.py lines 1416-1426, the runtime lookback is subtracted from the cursor value:

    stream_state_value = stream_state.get(cursor_field.cursor_field_key)
if runtime_lookback_window and stream_state_value:
    new_stream_state = (
        connector_state_converter.parse_timestamp(stream_state_value)
        - runtime_lookback_window
    )

  4. This becomes self.start in ConcurrentCursor.__init__() at airbyte_cdk/sources/streams/concurrent/cursor.py line 225.

  5. should_be_synced() at line 561-573 uses self.start as the lower bound:

    return self.start <= record_cursor_value <= self._end_provider()

The configured lookback_window (e.g., P1D) from the manifest's DatetimeBasedCursor is passed to ConcurrentCursor as self._lookback_window (line 226) but is only used in stream_slices() for generating API request time ranges. It is never referenced by should_be_synced().

Reproduction

Connector: source-intercom
Stream: conversation_parts (substream of conversations)
Config: lookback_window = 1, hourly syncs, incremental append+deduped
Manifest settings: global_substream_cursor: true, is_client_side_incremental: true

Observed behavior:

  • Intercom API returns 44 parts for a conversation
  • Airbyte destination table has 38 parts (6 missing)
  • The 6 missing parts were created seconds/minutes before other parts on the same conversation that were successfully synced
  • Example: On April 28, parts created at 15:33:10-16 were dropped, while parts created at 15:33:18 (2 seconds later) on the same conversation were synced. The should_be_synced cutoff landed between them.

Expected behavior:

  • With a 1-day lookback_window, self.start should be global_cursor - 1 day, not global_cursor - previous_sync_duration_seconds
  • All 44 parts should be synced

Root cause

The global_substream_cursor tracks one cursor across all partitions. When the cursor advances (from processing parts on other conversations), parts from individual conversations that have updated_at slightly behind the global max are dropped by should_be_synced. The runtime lookback (a few minutes) is far too small to cover the spread of updated_at values across conversations in a workspace.

Suggested fix

Apply the configured lookback_window (from the manifest's DatetimeBasedCursor) to the should_be_synced filter when using global_substream_cursor, either instead of or in addition to the runtime lookback. The effective lookback should be max(configured_lookback_window, runtime_lookback_window).

Devin session — requested by @mark-grivnin

Dominant language
Python
Stars
27
Forks
56
Avg merge
1d 20h
Merged PRs (30d)
26

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from airbytehq/airbyte-python-cdk

All issues in airbytehq/airbyte-python-cdk

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.