cube-js/cube

Cubestore - ADD Snowflake Storage Integration to Access Amazon S3

Open

#5,472 opened on Oct 17, 2022

View on GitHub
 (1 comment) (0 reactions) (0 assignees)Rust (19,563 stars) (1,965 forks)batch import
driver:snowflakehelp wanted

Description

Is your feature request related to a problem? Please describe. This is barely documented at all, but it seems Snowflake driver's S3 Export Bucket functionality currently works by:

  1. Creating temp pre-agg tables in Snowflake (with CREATE commands)
  2. Using a Snowflake COPY command to move this temp data into the S3 bucket you've specified in the CUBEJS_EXPORT_ and CUBESTORE env variables, including the access key/secret.

This is not the ideal way to integrate Snowflake with S3. Snowflake has a concept of an S3 Storage Integration.

Describe the solution you'd like Users should be allowed to set this up and specify their integration, rather than need to pass the access key/secret from Cube to Snowflake. There are many preferred ways to access an S3 bucket from a certain place (for example, EKS where Cube may be running), and the underlying Rust-AWS-auth lib seems to already support them.

Matter of fact, this Snowflake integration seems to have already been added for GCP buckets.

Describe alternatives you've considered No workarounds that are secure.

Contributor guide

Tech stack
rustaws
Domain
databasecloudinfrastructure
Issue type
feature
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
3-5 days
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
stale
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
mostly clear
Prerequisites
Rust programmingSnowflake SQLAWS S3 permissions
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
35
Research direction
Review the existing GCP storage integration implementation in the Cube repository, particularly the Snowflake driver code. Examine the Snowflake documentation on S3 Storage Integrations. Identify where the CUBEJS EXPORT environment variables are used in the Rust codebase and propose modifications to accept a storage integration name instead. The implementation should follow the pattern already established for GCP buckets.