New Integration: Google BigQuery

Description

Add Google BigQuery integration to sync data warehouse queries, datasets, tables, and job metadata into CORE.

Reference Implementations

Existing Integrations (use as templates)

• integrations/google-sheets/ - Similar Google API with OAuth 2.0
• integrations/google-docs/ - Google Workspace integration pattern
• integrations/google-calendar/ - Google OAuth and scheduling reference

Required Files Structure

integrations/google-bigquery/
├── src/
│   ├── index.ts          # Main entry, OAuth spec
│   ├── schedule.ts       # Sync logic
│   ├── utils.ts          # BigQuery API utilities
│   ├── account-create.ts # Google OAuth setup
│   └── create-activity.ts # Activity formatting
├── package.json
├── tsup.config.ts
└── README.md

Google BigQuery API Integration

OAuth Setup

• Use Google OAuth 2.0 (same pattern as other Google integrations)
• Required scopes:
  • https://www.googleapis.com/auth/bigquery.readonly - View BigQuery data
  • https://www.googleapis.com/auth/bigquery - Full BigQuery access
  • https://www.googleapis.com/auth/cloud-platform - Cloud platform access
• Base URL: https://bigquery.googleapis.com/bigquery/v2

Key Endpoints

• GET /projects/{projectId}/datasets - List all datasets
• GET /projects/{projectId}/datasets/{datasetId} - Get dataset details
• GET /projects/{projectId}/datasets/{datasetId}/tables - List tables in dataset
• GET /projects/{projectId}/datasets/{datasetId}/tables/{tableId} - Get table details
• GET /projects/{projectId}/jobs - List all jobs
• GET /projects/{projectId}/jobs/{jobId} - Get job details
• POST /projects/{projectId}/queries - Run a query
• GET /projects/{projectId}/datasets/{datasetId}/tables/{tableId}/data - Read table data

Events to Track

1. Jobs - Query jobs executed, load jobs, export jobs, copy jobs
2. Datasets - Created, modified, deleted datasets
3. Tables - Created, modified, deleted tables, schema changes
4. Queries - Query executions, results metadata

Implementation Tasks

• Set up basic integration structure following integrations/google-sheets/src/index.ts pattern
• Implement Google OAuth 2.0 flow in account-create.ts (reuse existing Google OAuth pattern)
• Create API utilities in utils.ts with BigQuery REST API v2 client
• Implement sync logic in schedule.ts for jobs, datasets, tables
• Convert BigQuery events to CORE activity format
• Add error handling and rate limiting
• Create integration documentation
• Add to integrations/README.md

Technical Notes

• BigQuery API v2 is the current REST API version
• Can reuse Google OAuth flow from existing Google integrations (google-sheets, google-docs, google-calendar)
• Discovery document available at https://bigquery.googleapis.com/$discovery/rest?version=v2
• Client libraries available for Java, Python, C#, Go, Node.js, PHP, Ruby
• Jobs API is most useful for tracking query activity and usage
• Consider syncing job history for audit/activity tracking
• Pagination via pageToken parameter
• Large result sets use maxResults for page size control

Resources

• BigQuery REST API Reference
• BigQuery API Documentation
• BigQuery Client Libraries
• Google APIs Explorer - BigQuery

Labels

enhancement, integration, new-feature