dlt-hub/dlt

implement include and exclude filters in Schema as data item transformations

開放

#64 建立於 2022年9月14日

 (2 則留言) (0 個反應) (0 位負責人)Python (589 個分叉)github user discovery
good first issue

倉庫指標

星標
 (5,767 顆星)
PR 合併指標
 (平均合併 7天 7小時) (30 天內合併 54 個 PR)

描述

row filtering can be done in item transform and be added optionally to any resource. implementation in Schema is slow and only solves problems it has created :)

here's a GPT-4 prompt that writes correct function

Write me a function in python that takes a nested dictionary as input. the dictionary can contain dictionaries, lists and basic types as values. the keys are string. the function takes two more arguments: a list of exclude regexes and a list of include regexes. the regex is matching paths in the dictionary. the paths are similar to json path but a separator is __ if given element has a path matching exclude regex it is removed from dictionary. however if any of the nested (child) elements of that element matches include path it should stay but other elements should be removed

testing: normalize tests are using various advanced modes of filtering. they must pass with new function

貢獻者指南