Logstash 1.5.0, Unable to remove field with nil value if it comes from redis input
#3310 opened on May 25, 2015
Description
Running logstash 1.5.0, I have a file with csv data that I read into logstash and parse using the csv plugin. Some fields will always be empty and gets a value of null. I then do some checking and remove these fields before shipping to elasticsearch. This all works fine, unless I first ship the csv data to redis, and then parse it, then I'm unable to remove the fields with value null.
Here are two small config files that I can use to recreate this issue:
redis-send.conf
input {
stdin {}
}
output {
redis {
data_type => "list"
key => "logs"
}
}
redis-receive.conf
input {
stdin {}
redis {
data_type => "list"
key => "logs"
}
}
filter {
csv {
columns => ["a", "b", "c", "d", "e"]
}
mutate {
remove_field => "a"
}
}
output {
stdout {
codec => "rubydebug"
}
}
On the sender, I'm typing the string "1,2,3,4,5" to stdin. On the receiver agent, I get this output:
{
"message" => [
[0] "1,2,3,4,5"
],
"@version" => "1",
"@timestamp" => "2015-05-22T14:12:44.173Z",
"host" => "starscream",
"b" => "2",
"c" => "3",
"d" => "4",
"e" => "5"
}
Just like I expect.
However, leaving the first field empty:
{
"message" => [
[0] ",2,3,4,5"
],
"@version" => "1",
"@timestamp" => "2015-05-22T14:17:54.859Z",
"host" => "starscream",
"a" => nil,
"b" => "2",
"c" => "3",
"d" => "4",
"e" => "5"
}
It does not remove the field "a".
But if I type the same string in stdin on the receiver agent, it removes the field:
{
"message" => [
[0] ",2,3,4,5"
],
"@version" => "1",
"@timestamp" => "2015-05-22T14:18:24.907Z",
"host" => "starscream",
"b" => "2",
"c" => "3",
"d" => "4",
"e" => "5"
}
I've tried the above in logstash 1.4.2 as well but didn't encounter the issue in that version.