Repository metrics
- Stars
- (5,441 stars)
- PR merge metrics
- (PR metrics pending)
説明
Feature description
Extend the default response actions with request retry based on response content.
Are you a dlt user?
I'd consider using dlt, but it's lacking a feature I need.
Use case
I am considering dlt with declarative rest api source as a replacement for custom ingestion script. The API I am using can return incomplete pages of data having response status 200, but 'error' key in response content. Currently this leads to missing data at destination. In my case incremental cursor value is read from last page so previous incomplete pages are never updated.
Proposed solution
Add 'retry' response action based on response content which will trigger retry of rest api request. The error key can be looked up as content substring, but it would be great to have an ability to check for actual keys. As an example, in my case it is "status":"failure" and "error":{"message": ..., "details": ...}. I understand that I use custom Client instance for this use case, but it seems a good addition to declarative source.
Related issues
No response