Add option to not apply split filter if new batch size approaches memory limit
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 35/100
- Issue type
- Feature
- Clarity
- Needs clarification
- Activity status
- Stale
- Tech stack
- ruby
- Domain
- backend, performance
Research direction
Start by reviewing the split filter implementation and the linked Logstash change introducing pipeline.batch.max_output_size. Clarify how event memory is estimated, the memory threshold and configuration behavior before choosing an approach. Done should include a defined safeguard for oversized split batches and tests covering the relevant memory-limit cases.
Written by the indexing model from the issue text.
Description
Split filters can sometimes cause Out of Memory errors if they significantly increase the size of a pipeline batch. This is either due to a filter multiplying the number of events or to the filter increasing the size of individual events.
We've added a config option pipeline.batch.max_output_size to Logstash that, when set, will divide a batch into chunks to be sent to the outputs. This mitigates some OOM errors but work could also be done in the split filter itself to mitigate the errors.
One idea is to call estimateMemory on each event and then multiply that by the split factor, and compare the number to memory size. We could refuse to split if event_size*factor > 50% heap size
This idea would add an overhead but it might be worth it.
- Dominant language
- Ruby
- Stars
- 6
- Forks
- 29
- PR merge metrics
- No merged PRs in 30d
Getting set up
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from logstash-plugins/logstash-filter-split
-
enhancement
Difficulty 5/5 Over a week Newbie friendliness 25/100
-
bug
Difficulty 3/5 1-2 days Newbie friendliness 45/100
logstash-plugins/logstash-filter-split#47 · 1 comment · 1 reaction ·
-
Difficulty 4/5 3-5 days Newbie friendliness 35/100
logstash-plugins/logstash-filter-split#42 · 1 comment ·
-
Difficulty 3/5 1-2 days Newbie friendliness 35/100
logstash-plugins/logstash-filter-split#38 · 1 reaction ·
-
Split Filter is not multi thread, how to fix this?May be free again @colinsurprenant claimed this 2783 days ago, and no pull request is open. Open
logstash-plugins/logstash-filter-split#37 · 1 comment · 1 assignee ·
All issues in logstash-plugins/logstash-filter-split
Similar issues
-
security
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
Maintainers usually reply within 1 day
-
Mend: dependency security vulnerability
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
Maintainers usually reply within 1 day
-
Mend: dependency security vulnerability
Difficulty 2/5 1-3 hours Newbie friendliness 62/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
Maintainers usually reply within 1 day
-
homebrew warningsOpen
Difficulty 1/5 Under an hour Newbie friendliness 88/100
bradygaster/squad#2082 ·
Maintainers usually reply within 1 day