elastic/logstash

Logstash 1.5.0, Unable to remove field with nil value if it comes from redis input

Open

#3310 aperta il 25 mag 2015

Vedi su GitHub
 (1 commento) (0 reazioni) (0 assegnatari)Ruby (3496 fork)batch import
bughelp wanted

Metriche repository

Star
 (14.197 star)
Metriche merge PR
 (Merge medio 4g 19h) (50 PR mergiate in 30 g)

Descrizione

Running logstash 1.5.0, I have a file with csv data that I read into logstash and parse using the csv plugin. Some fields will always be empty and gets a value of null. I then do some checking and remove these fields before shipping to elasticsearch. This all works fine, unless I first ship the csv data to redis, and then parse it, then I'm unable to remove the fields with value null.

Here are two small config files that I can use to recreate this issue:

redis-send.conf

input {
    stdin {}
}

output {
    redis {
        data_type => "list"
        key => "logs"
    }
}

redis-receive.conf

input {
    stdin {}
    redis {
        data_type => "list"
        key => "logs"
    }
}

filter {
    csv {
        columns => ["a", "b", "c", "d", "e"]
    }
    mutate {
        remove_field => "a"
    }
}

output {
    stdout {
        codec => "rubydebug"
    }
}

On the sender, I'm typing the string "1,2,3,4,5" to stdin. On the receiver agent, I get this output:


{
       "message" => [
        [0] "1,2,3,4,5"
    ],
      "@version" => "1",
    "@timestamp" => "2015-05-22T14:12:44.173Z",
          "host" => "starscream",
             "b" => "2",
             "c" => "3",
             "d" => "4",
             "e" => "5"
}

Just like I expect.

However, leaving the first field empty:

{
       "message" => [
        [0] ",2,3,4,5"
    ],
      "@version" => "1",
    "@timestamp" => "2015-05-22T14:17:54.859Z",
          "host" => "starscream",
             "a" => nil,
             "b" => "2",
             "c" => "3",
             "d" => "4",
             "e" => "5"
}

It does not remove the field "a".

But if I type the same string in stdin on the receiver agent, it removes the field:

{
       "message" => [
        [0] ",2,3,4,5"
    ],
      "@version" => "1",
    "@timestamp" => "2015-05-22T14:18:24.907Z",
          "host" => "starscream",
             "b" => "2",
             "c" => "3",
             "d" => "4",
             "e" => "5"
}

I've tried the above in logstash 1.4.2 as well but didn't encounter the issue in that version.

Guida contributor