Split Filter is not multi thread, how to fix this?
@colinsurprenant is already working on this.
Since Feb 12, 2019.
Assessment
This issue has not been assessed yet.
Description
i am using a conf file whole input flush 80K records after read
input {
http_poller{
urls => {
http_request => {
method => get
url => "http://xyzfmt=CSV"
}
}
codec => "json"
}
}
filter {
split {
field => "message"
}
csv {
columns => ["Org", "Network", ......................................"Brand"]
separator => ","
}
mutate {
remove_field => ["message","tags"]
}
}
output {
stdout { codec => rubydebug }
elasticsearch
{
}
}
After reading the input i get 80 K records and it does split operations and i have to wait 4 hrs
Why i cant i do filter operations on 20 records --> output to 20 records
then in second batch
filter operations on another 20 records --> output another 20 records to output
Currently it does split operation 80 K records which takes around 4 hrs and then start posting records to output, which is very slow process.
For this i have to wait 4 hrs to see records in output, cant we start seeing it just after minutes at least records in batches of 20?
Cant we start seeing 20 records in output in couple of minutes, else i have to wait 4 hrs to check the output
I have tried keeping the pipeline worker more that 2, 4,6, 12 but nothing works out. Looks like its a bug on split filter
pipeline.workers: 2
pipeline.output.workers: 2
- Dominant language
- Ruby
- Stars
- 6
- Forks
- 29
- PR merge metrics
- No merged PRs in 30d
Getting set up
- No Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from logstash-plugins/logstash-filter-split
-
enhancement status:needs-triage
Difficulty 5/5 Over a week Newbie friendliness 35/100
-
enhancement
Difficulty 5/5 Over a week Newbie friendliness 25/100
-
bug
Difficulty 3/5 1-2 days Newbie friendliness 45/100
logstash-plugins/logstash-filter-split#47 · 1 comment · 1 reaction ·
-
Difficulty 4/5 3-5 days Newbie friendliness 35/100
logstash-plugins/logstash-filter-split#42 · 1 comment ·
-
Difficulty 3/5 1-2 days Newbie friendliness 35/100
logstash-plugins/logstash-filter-split#38 · 1 reaction ·
All issues in logstash-plugins/logstash-filter-split
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 90/100
Maintainers usually reply within 3 days
-
Difficulty 2/5 1-3 hours Newbie friendliness 86/100
mastodon/mastodon#40755 · 1 comment · 1 reaction ·
Maintainers usually reply within 1 day
-
L: docker L: elm L: github:actions L: helm L: ruby:bundler
Difficulty 2/5 1-3 hours Newbie friendliness 85/100
dependabot/dependabot-core#16425 ·
Maintainers usually reply within 2 days