dlt-hub/dlt

make the use of `loader_file_format` parameter consistent

開放

#815 建立於 2023年12月11日

 (4 則留言) (0 個反應) (0 位負責人)Python (590 個分叉)github user discovery
QoLgood first issue

倉庫指標

星標
 (5,774 顆星)
PR 合併指標
 (平均合併 7天 7小時) (30 天內合併 54 個 PR)

描述

Background The loader_file_format can be configured on the pipeline level and this is fine. But technically it is a normalize setting and we should be able to configure it there as well.

Also not all data types are supported by all formats on all destinations. We should detect those and fail early.

Tasks

    • add loader_file_format to normalize configuration and use it to set the file format. If the explicit value is passed from the pipeline, it has the precedence.
    • add the exceptions to destination capabilities where we declare unsupported types per file format. Take the current cases from particular destination code (ie Bigquery and Snowflake have plenty of exceptions)
    • detect such exceptions in normalize (preferably in worker: before updating a schema check if any type is on exception list and raise exception with an explanation)
    • the behavior above must be tested

貢獻者指南