SubstreamPartitionRouter emits partitions for explicitly null parent_key values, producing literal 'None' in request paths

Open Beginner friendly
#1,127 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
82/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
python
Domain
api, backend

Research direction

Start in airbyte_cdk/sources/declarative/partition_routers/substream_partition_router.py around lines 215-239, then use the minimal reproduction with an explicitly null parent_key to trace the emitted StreamSlice. Done means null partition values are skipped like missing paths, preventing a request path containing literal None; preserve the existing behavior for valid values.

Written by the indexing model from the issue text.

Description

community

Symptom

SubstreamPartitionRouter handles a missing parent_key path and an explicitly null parent_key value inconsistently:

  • parent record LACKS the parent_key path >> dpath.get raises KeyError >> the record is silently skipped (emit_slice = False), no partition is produced;
  • parent record HAS the path but its value is explicitly null >> dpath.get returns None >> a partition IS produced with a None partition value.

A downstream requester that interpolates the partition value into its path then issues a garbage request with the literal string None in the URL, e.g. GET https://api.linkedin.com/rest/posts/None. Most APIs answer 400, which maps to FAIL via DEFAULT_ERROR_MAPPING and kills the whole sync.

Root cause

At tag v7.23.8, airbyte_cdk/sources/declarative/partition_routers/substream_partition_router.py:

  • lines 232-239: only the KeyError branch sets emit_slice = False (with the in-code FIXME noting the missing log);
  • line 215-216: the guard only checks parent_record is not None, not the extracted partition_value.

So partition_value = None flows through to the yielded StreamSlice, and Jinja stringifies it during request interpolation ('None').

Minimal reproduction

Parent stream returns {"id": "c1", "content": {"reference": null}}; child stream config:

partition_router:
  type: SubstreamPartitionRouter
  parent_stream_configs:
    - type: ParentStreamConfig
      parent_key: content/reference
      partition_field: post_urn
      stream: { $ref: "#/definitions/streams/parent" }

with requester path: posts/{{ stream_slice.get('post_urn') }}. Observed request: GET <url_base>/posts/None. A parent record with no content key at all is correctly skipped.

Expected

A null partition value should be treated like a missing one (skip the record, ideally with a log line), or at minimum be skippable via configuration. Emitting a partition whose value is None is never useful: it cannot address a real resource.

Impact

Any substream whose parent API emits explicit null for the parent key field fails the sync with a confusing 400 on a .../None URL. Connectors have to work around it with response-filter IGNOREs on the child requester, which also masks genuinely malformed requests.

Precedent

Found while reviewing https://github.com/airbytehq/airbyte/pull/81509 (source-linkedin-ads videos stream: creatives.content.reference >> GET /rest/posts/{urn}). The connector-side mitigation there is an IGNORE response filter matching URN-related 400s.

Dominant language
Python
Stars
26
Forks
53
Avg merge
5d 12h
Merged PRs (30d)
7

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from airbytehq/airbyte-python-cdk

All issues in airbytehq/airbyte-python-cdk

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.