influxdata/telegraf

Auto create table in BigQuery

Open

#13,033 opened on Apr 4, 2023

View on GitHub
 (2 comments) (0 reactions) (0 assignees)Go (4,161 forks)batch import
area/gcpfeature requesthelp wantedplugin/output

Repository metrics

Stars
 (9,892 stars)
PR merge metrics
 (Avg merge 2d 12h) (136 merged PRs in 30d)

Description

Use Case

With some input plugins the metric names and formats are unknown on beforehand, the number of tags and fields can vary over time. Therefore it would be nice if the outputs.bigquery plugin could handle this in a way similar to the outputs.sql plugin.

Expected behavior

Telegraf creates the required table in BigQuery with all the corresponding metric fields and tags just as the sql

Actual behavior

Tables for metrics need to be created on beforehand in BigQuery.

Additional info

Here is a PoC which adds a new BigQuery field on the fly, proving it should be possible.

Contributor guide

Research direction
Examine the provided PoC gist and understand how to dynamically add fields to BigQuery table. Then adapt the existing outputs.bigquery plugin to auto create tables with all metric fields and tags, similar to outputs.sql plugin.
Tech stack
go
Domain
backendapidatabase
Issue type
Feature
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
Half day
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
Active
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
Needs investigation
Prerequisites
GoBigQuery
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
50