Option to disable hive partitioning wild cards
Nobody has claimed this yet.
Assessment
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Newbie friendliness
- 35/100
- Issue type
- Feature
- Clarity
- Mostly clear
- Activity status
- Stale
- Tech stack
- javascript
- Domain
- cloud
Research direction
Start in index.js at line 78, where transformHiveStylePrefix() is defined, and line 1584, where it transforms inputInfo.prefix. Trace how the prefix is configured and determine how an option could preserve folder names containing '='; done means the supplied prefix is used exactly when the option disables hive-style wildcard transformation.
Written by the indexing model from the issue text.
Description
The avro files we are trying to load into RedShift are stored in folders with "=" in their names, i.e.
event_type=users.behaviors.app.FirstSession/.
When loading data from the following S3 prefix,
com.hoopladigital.brazecurrentsstaging/StagingCurrentFull/dataexport.prod-03.S3.integration.60d3692fcab9ca5f83919aab/event_type%3Dusers.behaviors.app.FirstSession
The lambda failed with this error:
error: No Configuration Found for com.hoopladigital.brazecurrentsstaging/StagingCurrentFull/dataexport.prod-03.S3.integration.60d3692fcab9ca5f83919aab/event_type=*/date=*/399/prod-03
As shown in the error message above, the"event_type=/date=" portion of the error message was transformed assuming that we are taking advantage of the hive partitioning wildcards (https://github.com/awslabs/aws-lambda-redshift-loader#hive-partitioning-style-wildcards) and replaces the event_type value with *.
We don't want to use this feature- I need the lambda to use the exact folder name that I provided in the prefix. Is there a way for me to configure the lambda to not use hive partitioning wild cards?
line 1584 of index.js:
inputInfo.prefix = inputInfo.bucket + '/' + searchKey.transformHiveStylePrefix();
line 78 of index.js
transformHiveStylePrefix()
- Dominant language
- JavaScript
- Stars
- 595
- Forks
- 161
- PR merge metrics
- No merged PRs in 30d
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from awslabs/aws-lambda-redshift-loader
-
Difficulty 1/5 Under an hour Newbie friendliness 68/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 32/100
awslabs/aws-lambda-redshift-loader#247 · 3 comments ·
-
awslabs/aws-lambda-redshift-loader#244 · 4 comments · 1 assignee ·
-
Difficulty 4/5 3-5 days Newbie friendliness 25/100
awslabs/aws-lambda-redshift-loader#239 · 4 comments ·
-
Difficulty 4/5 3-5 days Newbie friendliness 20/100
awslabs/aws-lambda-redshift-loader#238 · 1 comment ·
All issues in awslabs/aws-lambda-redshift-loader
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 70/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
mksglu/context-mode#1200 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
neondatabase/website#5944 ·
-
module: core
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
bigbluebutton/bigbluebutton#25849 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
jaegertracing/jaeger-ui#4506 ·