elastic/logstash

Logstash 1.5.0, Unable to remove field with nil value if it comes from redis input

Open

Aperta il 25 mag 2015

Vedi su GitHub
 (1 commento) (0 reazioni) (0 assegnatari)Ruby (14.197 star) (3496 fork)batch import
bughelp wanted

Descrizione

Running logstash 1.5.0, I have a file with csv data that I read into logstash and parse using the csv plugin. Some fields will always be empty and gets a value of null. I then do some checking and remove these fields before shipping to elasticsearch. This all works fine, unless I first ship the csv data to redis, and then parse it, then I'm unable to remove the fields with value null.

Here are two small config files that I can use to recreate this issue:

redis-send.conf

input {
    stdin {}
}

output {
    redis {
        data_type => "list"
        key => "logs"
    }
}

redis-receive.conf

input {
    stdin {}
    redis {
        data_type => "list"
        key => "logs"
    }
}

filter {
    csv {
        columns => ["a", "b", "c", "d", "e"]
    }
    mutate {
        remove_field => "a"
    }
}

output {
    stdout {
        codec => "rubydebug"
    }
}

On the sender, I'm typing the string "1,2,3,4,5" to stdin. On the receiver agent, I get this output:


{
       "message" => [
        [0] "1,2,3,4,5"
    ],
      "@version" => "1",
    "@timestamp" => "2015-05-22T14:12:44.173Z",
          "host" => "starscream",
             "b" => "2",
             "c" => "3",
             "d" => "4",
             "e" => "5"
}

Just like I expect.

However, leaving the first field empty:

{
       "message" => [
        [0] ",2,3,4,5"
    ],
      "@version" => "1",
    "@timestamp" => "2015-05-22T14:17:54.859Z",
          "host" => "starscream",
             "a" => nil,
             "b" => "2",
             "c" => "3",
             "d" => "4",
             "e" => "5"
}

It does not remove the field "a".

But if I type the same string in stdin on the receiver agent, it removes the field:

{
       "message" => [
        [0] ",2,3,4,5"
    ],
      "@version" => "1",
    "@timestamp" => "2015-05-22T14:18:24.907Z",
          "host" => "starscream",
             "b" => "2",
             "c" => "3",
             "d" => "4",
             "e" => "5"
}

I've tried the above in logstash 1.4.2 as well but didn't encounter the issue in that version.

Guida contributor