apache/beam

BigQueryBatchFileLoads dynamic destination

Open

#20,242 opened on 2022年6月4日

GitHub で見る
 (6 comments) (0 reactions) (1 assignee)Java (4,097 forks)batch import
P3bugdocumentationgcpgood first issueiopython

Repository metrics

Stars
 (7,313 stars)
PR merge metrics
 (平均マージ 6d) (30d で 243 merged PRs)

説明

I am trying to use BigQueryBatchFileLoads to upload data from pubsub. It works fine for a single table, but when I try to use a dynamic destination such as destination=lambda elem: "my_project:my_dataset." **** elem["sensor_key"], it just makes a new table for each time the triggering_frequency procs. I know it makes temporary tables before loading it all into one, but it never loads them. It just creates more and more tables.

Imported from Jira BEAM-9917. Original Jira may contain additional context. Reported by: tordaut.

コントリビューターガイド