Aborted query notification in Redshift

Open
#239 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
25/100
Issue type
Bug
Clarity
Needs clarification
Activity status
Stale
Tech stack
aws, javascript
Domain
backend, cloud, database

Research direction

Start with the LambdaRedshiftLoader batch-processing and retry paths, focusing on how COPY results are reported when Redshift aborts and rewrites a query. Reproduce the multi-cluster case if possible and determine whether reprocessing an error batch retries both clusters or only the failed one; the issue is complete when that behavior and the correct batch status are established.

Written by the indexing model from the issue text.

Description

I faced a weird behavior of the Redshift cluster (the support ticket is still open with service team: Case ID 9386087231) where COPY command query sent by the LambdaRedshiftLoader getting aborted first, and then re-written, and competed successfully by the Redshift engine. The problem is that the only Aborted result reported back to the Lambda function and batch status marked as 'error'. I verified in the Redshift that data was loaded successfully in this scenario.
Also, the complexity of this issue is that there is multi cluster scenario, where data loaded to two Redshift clusters and only one is failing with the above scenario. This leads to a question I can't find an answer to in this repo: If I reprocess the 'error' batch does the data loaded to both clusters, or solution is smart enough to load failed batches only for failed Redshift clusters?

Dominant language
JavaScript
Stars
595
Forks
161
No merged PRs in 30d

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from awslabs/aws-lambda-redshift-loader

All issues in awslabs/aws-lambda-redshift-loader

Similar issues

More JavaScript issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.