描述
Tell us about the problem you're trying to solve
The Klaviyo Events stream hits an endpoint that returns events for all metrics, which can be quite large (we have one sync that was just shy of 100GB on its initial run with a start date of 2021.01.01).
Describe the solution you’d like
I want to be able to only pull events for specific metrics. See https://developers.klaviyo.com/en/reference/metric-timeline
Describe the alternative you’ve considered or used
Currently we have no alternative. We separate the events out into per-metric tables in dbt transforms, but we still incur the cost of potentially huge Events syncs and then ignore a lot of them.
Additional context
I don't think this is super straightforward due to how the Klaviyo API works so I wanted to detail my implementation idea here:
The endpoint whose docs I linked above requires a metric_id, but even native Klaviyo metrics appear to have different ids across different accounts, so the source config would need to accept an optional list of either ids or metric names that are mapped to the proper ids by the connector (by hitting the metrics list endpoint for the given account). Our use case involves pulling data from multiple accounts, so being able to specify something like ["Bounced Email", "Clicked Email"] in the config for each connection would be preferable to needing to find out the ids for each metric for each account. I also feel like this would be more friendly even for users who are only worried about a single account, but I don't want to force this approach if it seems that the step to map them would be too much of a headache to maintain, or if it seems like it wouldn't work at all.
As for the effects on the streams, I was imagining producing a metric_name_events (or metric_id_events) stream per metric, which is similar to how I've noticed the facebook marketing connector handles custom insights config.
Are you willing to submit a PR?
Yes