good first issuehelp wantedingestion
Repository metrics
- Stars
- (4,937 stars)
- PR merge metrics
- (Avg merge 6d 7h) (186 merged PRs in 30d)
Description
- search for ingestion file is hard-coded to recursive, no option to do non-recursive scan
- glob setting is matching absolute path, which should obviously exclude part from the root directory
- ingestion config is sometimes required to be escape but sometimes it is not - depending on the plugin it is using
- ingestion job returns empty list of segments being pushed when segment generation fails - it should error out if # of files is different from # of segment generated.
- some of the 2-stage segment generation job doesn't populate exception/throwable back to the main thread and thus executions like SEGMENT_CREATE --> SEGMENT_UPLOAD will end up in a success state with an empty TAR file upload result;
TODO:
- provide recursive flag in commandline utils
- remove glob matcher prefix for inputFileDir
- unifies plugin config parsing
- pipe-back segment job runner / task runner exception back to main thread.