Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

How to correctly handle locked and reprocessed batches?

Open
  • Is something wrong?
#119 5 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
25/100
Issue type
Bug
Clarity
Needs clarification
Activity status
Stale
Tech stack
aws, javascript
Domain
backend, cloud, databases

Research direction

Start with unlockBatch.js and reprocessBatch.js, then inspect the LambdaRedshiftBatches and LambdaRedshiftBatchLoadConfig records alongside the reported CloudWatch log. Trace how locked, reprocessed, and open batches update currentBatch, and document the expected recovery path and whether all affected S3 files reach Redshift.

Written by the indexing model from the issue text.

Description

Hi,

We are using the loader function to load JSON files (generated from an Apache Spark application running in an EMR cluster) to Redshift.

We ran into an issue where many batches became "locked". We think this was related to a problem where we had many ProvisionedThroughputExceededException on the DynamoDB tables backing the loader function. We increased the provisioning for the tables, but we are noticing that for some of our s3Prefix configurations, we are not seeing any new data loaded to Redshift. We are also seeing many of these logs in Cloudwatch:

"2017-03-16T20:00:20.166Z 2e92425c-0a83-11e7-8d55-69fee54689ef Reload of Configuration Complete after attempting to write to Locked Batch a0d645fd-5dbb-4386-b2c5-4674426851a9. Attempt 1"

(This log was for batchId a0d645fd-5dbb-4386-b2c5-4674426851a9 in particular, but we see them for many batches.)

I noticed that the batches for which we are seeing these messages were marked as "locked" in the LambdaRedshiftBatches table. I used the included unlockBatch.js script, and the reprocessBatch.js script, to attempt to "reprocess" these batches. This did not work - the batch status in LambdaRedshiftBatches became marked as "reprocessed", but we continued to see the aforementioned logs in Cloudwatch.

Next, I noticed that for the rows in LambdaRedshiftBatchLoadConfig where the s3Prefix corresponded to a locked/reprocessed batchId, the currentBatch property referred to the locked/reprocessed batchId. I queried LambdaRedshiftBatches for any open batches for that s3Prefix, then manually updated the currentBatch property in LambdaRedshiftBatchLoadConfig with the batchId of the open batch.

This seems to have resolved the issue, but I'm unsure now if I've missed loading the files in the locked/reprocessed batches into Redshift. I'm left with a number of other questions:

  1. What does it mean for a batch to have the status "locked" or "reprocessed"?
  2. How can batches get into the "locked" state?
  3. I noticed that there were many "open" batches for a particular s3Prefix in LambdaRedshiftBatches. Is this normal? The documentation states "There will always be one open batch, and may be multiple closed batches per S3 input prefix from LambdaRedshiftBatchLoadConfig", but does this mean only one open batch per s3Prefix?
  4. If it is true that there should be only one open batch per s3Prefix, how could the system get into a state where there are multiple?
  5. Was there a better way for me to handle my situation?

Hope you can help, and please let me know if any more info from me would be of use.

Regards,
Justin

Dominant language
JavaScript
Stars
595
Forks
161
No merged PRs in 30d

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from awslabs/aws-lambda-redshift-loader

All issues in awslabs/aws-lambda-redshift-loader

Similar issues

More JavaScript issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.