record_accessor raises ArgumentError instead of ConfigError for an array index that is not an integer

Open Beginner friendly
  • Is something wrong?
#5,595 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
82/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
ruby
Domain
backend

Research direction

Start with plugin_helper/record_accessor.rb, especially parse_dot_array_op and parse_bracket_notation, where Integer(index_value) raises the reported error. Read the neighboring syntax error handling and run test/plugin_helper/test_record_accessor.rb. Done means non-integer array indices raise Fluent::ConfigError with the invalid key in the message.

Written by the indexing model from the issue text.

Description

Describe the bug

A record_accessor key with an array index that is not an integer, such as $.a[x] or $['a'][x], makes Fluentd stop with ArgumentError instead of Fluent::ConfigError. Fluentd stops either way, but the error is not reported as a config error: on 1.20.1 it is logged as "unexpected error" without the file of the configuration, and on 1.16.11 Fluentd exits with a Ruby backtrace.

The cause is Integer(index_value) in plugin_helper/record_accessor.rb (line 143 in parse_dot_array_op and line 183 in parse_bracket_notation, on master af25743e1 and v1.20.1). The other syntax errors next to them, such as a missing ] or an empty [], raise Fluent::ConfigError, but the ArgumentError of Integer() is not converted.

This affects every plugin that uses record_accessor_create for a key from the configuration, for example filter_grep.

To Reproduce

With this configuration:

<source>
  @type sample
  tag t
</source>
<filter t>
  @type grep
  <regexp>
    key $.a[x]
    pattern /x/
  </regexp>
</filter>
<match t>
  @type stdout
</match>

run fluentd --dry-run -c fluent.conf.

The same happens without a configuration file:

require "fluent/version"
require "fluent/plugin_helper/record_accessor"

puts "Fluentd #{Fluent::VERSION}"
["$.a[x]", "$['a'][x]", "$.[bad"].each do |key|
  Fluent::PluginHelper::RecordAccessor::Accessor.new(key)
  puts "#{key}: no error"
rescue => e
  puts "#{key}: #{e.class}: #{e.message}"
end

On master (af25743e1) with Ruby 3.4.12, it prints:

Fluentd 1.20.1
$.a[x]: ArgumentError: invalid value for Integer(): "x"
$['a'][x]: ArgumentError: invalid value for Integer(): "x"
$.[bad: Fluent::ConfigError: '[' found but ']' not found. Invalid syntax: $.[bad

With fluentd --dry-run and the configuration above, with each key in <regexp>:

key master (af25743e1) and 1.20.1 1.16.11
$.a[x] [error]: unexpected error error_class=ArgumentError error="invalid value for Integer(): \"x\"" (exit 1) No log line. Exits with a Ruby backtrace: record_accessor.rb:143:in `Integer': invalid value for Integer(): "x" (ArgumentError) (exit 1)
$['a'][x] (bracket notation) The same ArgumentError The same backtrace, from line 183
$.[bad (for comparison) [error]: config error file="fluent.conf" error_class=Fluent::ConfigError error="'[' found but ']' not found. Invalid syntax: $.[bad" The same config error
Expected behavior

Fluent::ConfigError, like the other syntax errors of a record_accessor key, so that Fluentd reports it as a config error with the file and the key, for example "array index must be an integer. Invalid syntax: $.a[x]".

Your Environment
  • Fluentd version: master (af25743e1), 1.20.1 and 1.16.11
  • Ruby version: 3.4.12 (master, 1.20.1), 3.2.11 (1.16.11)
  • Operating system: Linux (Docker, ruby:3.4 and ruby:3.2)
Your Configuration

The configuration is in "To Reproduce" above.

Your Error Log

Fluentd 1.20.1 and master:

[info]: starting fluentd-1.20.1 as dry run mode ruby="3.4.12"
[error]: unexpected error error_class=ArgumentError error="invalid value for Integer(): \"x\""

Fluentd 1.16.11:

[info]: starting fluentd-1.16.11 as dry run mode ruby="3.2.11"
/usr/local/bundle/gems/fluentd-1.16.11/lib/fluent/plugin_helper/record_accessor.rb:143:in `Integer': invalid value for Integer(): "x" (ArgumentError)

For comparison, key $.[bad gives:

[error]: config error file="fluent.conf" error_class=Fluent::ConfigError error="'[' found but ']' not found. Invalid syntax: $.[bad"
Additional context
  • I tried rescuing the ArgumentError of the two Integer(index_value) calls and raising Fluent::ConfigError on master. The keys above then raise Fluent::ConfigError, and test/plugin_helper/test_record_accessor.rb passes (63 tests). In parse_bracket_notation, the message should use orig_param, as the other errors there do, because param is only the rest of the key at that point.
  • A related but different case: a key that ends with [ or . is accepted without an error, and the end is ignored. $.a[, $.a., $.a.. and $['a'][ all read the key "a", on master, 1.20.1 and 1.16.11. $.a[1 and $.[bad raise "'[' found but ']' not found". In parse_dot_array_op, the loop does not run when nothing follows [, and String#split('.') drops the empty strings at the end. These configurations start today, so making them an error could stop Fluentd after an update; a warning may fit better. I am not asking to change this here, but mention it, as it is next to the code above.
  • This was found in fluent/fluent-plugin-rewrite-tag-filter, which reports the broken settings of its rules together at startup. To report this key with the others, it would have to rescue this ArgumentError by its message.
Dominant language
Ruby
Stars
13.6k
Forks
1.4k
Avg merge
1d 4h
Merged PRs (30d)
56

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from fluent/fluentd

All issues in fluent/fluentd

Similar issues

More Ruby issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.