elastic/logstash

Logstash 1.5.0, Unable to remove field with nil value if it comes from redis input

Open

#3310 opened on May 25, 2015

View on GitHub
 (1 comment) (0 reactions) (0 assignees)Ruby (14,197 stars) (3,496 forks)batch import
bughelp wanted

Description

Running logstash 1.5.0, I have a file with csv data that I read into logstash and parse using the csv plugin. Some fields will always be empty and gets a value of null. I then do some checking and remove these fields before shipping to elasticsearch. This all works fine, unless I first ship the csv data to redis, and then parse it, then I'm unable to remove the fields with value null.

Here are two small config files that I can use to recreate this issue:

redis-send.conf

input {
    stdin {}
}

output {
    redis {
        data_type => "list"
        key => "logs"
    }
}

redis-receive.conf

input {
    stdin {}
    redis {
        data_type => "list"
        key => "logs"
    }
}

filter {
    csv {
        columns => ["a", "b", "c", "d", "e"]
    }
    mutate {
        remove_field => "a"
    }
}

output {
    stdout {
        codec => "rubydebug"
    }
}

On the sender, I'm typing the string "1,2,3,4,5" to stdin. On the receiver agent, I get this output:


{
       "message" => [
        [0] "1,2,3,4,5"
    ],
      "@version" => "1",
    "@timestamp" => "2015-05-22T14:12:44.173Z",
          "host" => "starscream",
             "b" => "2",
             "c" => "3",
             "d" => "4",
             "e" => "5"
}

Just like I expect.

However, leaving the first field empty:

{
       "message" => [
        [0] ",2,3,4,5"
    ],
      "@version" => "1",
    "@timestamp" => "2015-05-22T14:17:54.859Z",
          "host" => "starscream",
             "a" => nil,
             "b" => "2",
             "c" => "3",
             "d" => "4",
             "e" => "5"
}

It does not remove the field "a".

But if I type the same string in stdin on the receiver agent, it removes the field:

{
       "message" => [
        [0] ",2,3,4,5"
    ],
      "@version" => "1",
    "@timestamp" => "2015-05-22T14:18:24.907Z",
          "host" => "starscream",
             "b" => "2",
             "c" => "3",
             "d" => "4",
             "e" => "5"
}

I've tried the above in logstash 1.4.2 as well but didn't encounter the issue in that version.

Contributor guide