delta-io/delta

[Feature Request] LogStore integration test framework

Open

#1124 aperta il 11 mag 2022

Vedi su GitHub
 (7 commenti) (0 reazioni) (1 assegnatario)Scala (2100 fork)batch import
enhancementgood first issue

Metriche repository

Star
 (8807 star)
Metriche merge PR
 (Merge medio 7g 1h) (142 PR mergiate in 30 g)

Descrizione

Feature request

Overview

It would be great to have an automated integration test framework for all of our LogStores that includes a) integration-testing (using real cloud buckets/tables) and b) stress-testing (using multiple concurrent readers + writers) c) validation that the reads/writes were successful

Motivation

Further details

There is an existing example integration test for S3DynamoDBLogStore at here.

It would be great to get similar integration tests for each of the production log stores in the delta-storage artifact here.

Ideally it would be one python file that took command line arguments for the

  • log store to use
  • cloud/scheme
  • tablePath/bucket
  • credentials
  • number of current readers / writers
  • per-cloud specific arguments

that could be invoked from a specific run-logstore-integration-tests.py file that ran for each of our log stores.

Guida contributor