dlt-hub/dlt

Add 'retry' response action for generic rest api source

已关闭

#1,786 创建于 2024年9月4日

 (5 条评论) (0 个反应) (1 位负责人)Python (589 个派生)github user discovery
QoLenhancementhelp wanted

仓库指标

星标
 (5,767 个星标)
PR 合并指标
 (平均合并 7天 7小时) (30 天内合并 54 个 PR)

描述

Feature description

Extend the default response actions with request retry based on response content.

Are you a dlt user?

I'd consider using dlt, but it's lacking a feature I need.

Use case

I am considering dlt with declarative rest api source as a replacement for custom ingestion script. The API I am using can return incomplete pages of data having response status 200, but 'error' key in response content. Currently this leads to missing data at destination. In my case incremental cursor value is read from last page so previous incomplete pages are never updated.

Proposed solution

Add 'retry' response action based on response content which will trigger retry of rest api request. The error key can be looked up as content substring, but it would be great to have an ability to check for actual keys. As an example, in my case it is "status":"failure" and "error":{"message": ..., "details": ...}. I understand that I use custom Client instance for this use case, but it seems a good addition to declarative source.

Related issues

No response

贡献者指南