Fallback to plain text does not work when the message can not be unpacked
Nobody has claimed this yet.
Assessment
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Newbie friendliness
- 52/100
Research direction
Start with lib/logstash/codecs/msgpack.rb around the rescue path at line 36 and compare it with the UDP input backtrace in the report. Reproduce the malformed msgpack input shown in the logs and verify that fallback produces a plain-text event tagged with _msgpackparsefailure while Logstash continues reading subsequent messages.
Written by the indexing model from the issue text.
Description
Technical context
OS : linux (in the docker container for logstash)
Gem Version : 3.0.2
Config file
input {
udp {
port => 1518
type => udp
codec => "msgpack"
}
}
filter {
geoip {
source => "ip"
}
}
output {
elasticsearch {
hosts => ["es1", "es2"]
index => "logstash-activities-%{+YYYY.MM.dd}"
document_id => "%{id}"
}
stdout {
codec => "json"
}
}
The problem
sometimes, my client send wrong messages to my logstash server probably due to UDP.
The message can not be unpacked correctly by msgpack and the failover to plain text does not work and crash logstash reader.
Logstash stop reading messages from this point but does not crash completely.
Logs
WARN logstash.codecs.msgpack - Trouble parsing msgpack input, falling back to plain text {:input=>"\x00", :exception=>#}
09:10:47.013 ["\x00", :exception=>#}
09:10:47.013 [#, "backtrace"=>["/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-codec-msgpack-3.0.2-java/lib/logstash/codecs/msgpack.rb:36:in `decode'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-codec-msgpack-3.0.2-java/lib/logstash/codecs/msgpack.rb:29:in `decode'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-udp-3.1.0/lib/logstash/inputs/udp.rb:118:in `inputworker'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-udp-3.1.0/lib/logstash/inputs/udp.rb:89:in `udp_listener'"]}
09:10:47.014 [#, "backtrace"=>["/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-codec-msgpack-3.0.2-java/lib/logstash/codecs/msgpack.rb:36:in `decode'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-codec-msgpack-3.0.2-java/lib/logstash/codecs/msgpack.rb:29:in `decode'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-udp-3.1.0/lib/logstash/inputs/udp.rb:118:in `inputworker'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-udp-3.1.0/lib/logstash/inputs/udp.rb:89:in `udp_listener'"]}
From this point, logstash stops
The way to solve it
In the rescue part of the plain text failover, event is not set. It should be initialized if not present this way :
rescue => e
# Treat as plain text and try to do the best we can with it?
@logger.warn("Trouble parsing msgpack input, falling back to plain text",
:input => data, :exception => e)
event ||= LogStash::Event.new()
event.set("message", data)
tags = event.get("tags").nil? ? [] : event.get("tags")
tags << "_msgpackparsefailure"
event.set("tags", tags)
end
I can handle it if you wish.
- Dominant language
- Ruby
- Stars
- 5
- Forks
- 10
- PR merge metrics
- No merged PRs in 30d
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from logstash-plugins/logstash-codec-msgpack
-
Difficulty 4/5 3-5 days Newbie friendliness 35/100
-
logstash-plugins/logstash-codec-msgpack#11 · 1 assignee ·
-
Difficulty 4/5 3-5 days Newbie friendliness 30/100
logstash-plugins/logstash-codec-msgpack#5 · 3 comments ·
All issues in logstash-plugins/logstash-codec-msgpack
Similar issues
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
hanami/hanami-cli#449 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
resque/resque-scheduler#826 ·
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100