dlt-hub/dlt

make the use of `loader_file_format` parameter consistent

Open

#815 opened on Dec 11, 2023

View on GitHubContributor guide
 (4 comments) (0 reactions) (0 assignees)Python (589 forks)github user discovery
QoLgood first issue

Repository metrics

Stars
 (5,765 stars)
PR merge metrics
 (Avg merge 7d 7h) (54 merged PRs in 30d)

Description

Background The loader_file_format can be configured on the pipeline level and this is fine. But technically it is a normalize setting and we should be able to configure it there as well.

Also not all data types are supported by all formats on all destinations. We should detect those and fail early.

Tasks

    • add loader_file_format to normalize configuration and use it to set the file format. If the explicit value is passed from the pipeline, it has the precedence.
    • add the exceptions to destination capabilities where we declare unsupported types per file format. Take the current cases from particular destination code (ie Bigquery and Snowflake have plenty of exceptions)
    • detect such exceptions in normalize (preferably in worker: before updating a schema check if any type is on exception list and raise exception with an explanation)
    • the behavior above must be tested

Contributor guide

Research direction
Examine how loader file format is currently used in pipeline and normalize configurations. Identify where to add the normalize setting and precedence logic. Review destination capabilities for unsupported types per format. Implement detection in normalize worker and write tests.
Tech stack
python
Domain
backend
Issue type
Feature
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
Half day
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
Active
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
Mostly clear
Prerequisites
Pythondlt
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
60