SubstreamPartitionRouter emits partitions for explicitly null parent_key values, producing literal 'None' in request paths
Nobody has claimed this yet.
Assessment
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Newbie friendliness
- 82/100
Research direction
Start in airbyte_cdk/sources/declarative/partition_routers/substream_partition_router.py around lines 215-239, then use the minimal reproduction with an explicitly null parent_key to trace the emitted StreamSlice. Done means null partition values are skipped like missing paths, preventing a request path containing literal None; preserve the existing behavior for valid values.
Written by the indexing model from the issue text.
Description
Symptom
SubstreamPartitionRouter handles a missing parent_key path and an explicitly null parent_key value inconsistently:
- parent record LACKS the
parent_keypath >>dpath.getraisesKeyError>> the record is silently skipped (emit_slice = False), no partition is produced; - parent record HAS the path but its value is explicitly
null>>dpath.getreturnsNone>> a partition IS produced with aNonepartition value.
A downstream requester that interpolates the partition value into its path then issues a garbage request with the literal string None in the URL, e.g. GET https://api.linkedin.com/rest/posts/None. Most APIs answer 400, which maps to FAIL via DEFAULT_ERROR_MAPPING and kills the whole sync.
Root cause
At tag v7.23.8, airbyte_cdk/sources/declarative/partition_routers/substream_partition_router.py:
- lines 232-239: only the
KeyErrorbranch setsemit_slice = False(with the in-code FIXME noting the missing log); - line 215-216: the guard only checks
parent_record is not None, not the extractedpartition_value.
So partition_value = None flows through to the yielded StreamSlice, and Jinja stringifies it during request interpolation ('None').
Minimal reproduction
Parent stream returns {"id": "c1", "content": {"reference": null}}; child stream config:
partition_router:
type: SubstreamPartitionRouter
parent_stream_configs:
- type: ParentStreamConfig
parent_key: content/reference
partition_field: post_urn
stream: { $ref: "#/definitions/streams/parent" }
with requester path: posts/{{ stream_slice.get('post_urn') }}. Observed request: GET <url_base>/posts/None. A parent record with no content key at all is correctly skipped.
Expected
A null partition value should be treated like a missing one (skip the record, ideally with a log line), or at minimum be skippable via configuration. Emitting a partition whose value is None is never useful: it cannot address a real resource.
Impact
Any substream whose parent API emits explicit null for the parent key field fails the sync with a confusing 400 on a .../None URL. Connectors have to work around it with response-filter IGNOREs on the child requester, which also masks genuinely malformed requests.
Precedent
Found while reviewing https://github.com/airbytehq/airbyte/pull/81509 (source-linkedin-ads videos stream: creatives.content.reference >> GET /rest/posts/{urn}). The connector-side mitigation there is an IGNORE response filter matching URN-related 400s.
- Dominant language
- Python
- Stars
- 26
- Forks
- 53
- Avg merge
- 5d 12h
- Merged PRs (30d)
- 7
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from airbytehq/airbyte-python-cdk
-
airbyte-cdk connector test: --no-creds crashes when combined with --pytest-arg (tuple has no extend) Opencommunity
Difficulty 1/5 Under an hour Newbie friendliness 88/100
airbytehq/airbyte-python-cdk#1153 ·
-
community
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
airbytehq/airbyte-python-cdk#1147 ·
-
community
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
airbytehq/airbyte-python-cdk#1137 ·
-
community
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
airbytehq/airbyte-python-cdk#1118 · 2 comments ·
-
community
Difficulty 2/5 1-3 hours Newbie friendliness 85/100
airbytehq/airbyte-python-cdk#1110 · 1 comment ·
All issues in airbytehq/airbyte-python-cdk
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
-
enhancement
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 74/100