parser_syslog, in_syslog: text before a PRI on a later line is silently dropped

Open
  • Is something wrong?
#5,593 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
3/5
Estimated time
1-2 days
Newbie friendliness
72/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
ruby
Domain
backend

Research direction

Read lib/fluent/plugin/parser_syslog.rb and lib/fluent/plugin/in_syslog.rb, focusing on the listed PRI and detection regexps and how both parser engines use them. Check the existing parser and input tests for multiline messages, then reproduce the examples with the regexp engine. Done means headers are recognized only at the start of the message, malformed leading text is emitted as unmatched_line with a warning, and auto detection retains an initial RFC3164 header.

Written by the indexing model from the issue text.

Description

bug
Describe the bug

With the default parser_engine regexp, parser_syslog looks for the PRI with a regexp that starts with ^. ^ matches at the start of any line. When a message contains a newline and its first line is not a syslog header, the parser starts from a later line that looks like one. The text before that line is dropped without a warning.

in_syslog has the same problem in its own SYSLOG_REGEXP. It uses this regexp when the parser does not parse the priority, for example with <parse> @type none.

A message with a newline can reach the parser through in_syslog over UDP, or over TCP with frame_type octet_count. It can also come through filter_parser.

Input Configuration Result
first line\n<14>1 2026-04-25T16:43:29Z host app 123 ID - second line message_format rfc5424 A record from the second line. first line is dropped.
first line\n<14>Apr 25 16:43:29 host app: second line message_format rfc3164 A record from the second line with pri 0, so in_syslog tags it kern.emerg. first line is dropped.
<13>Apr 25 16:43:29 web01 nginx: request failed:\n<14>1 2026-04-25T16:43:29Z host app 123 ID - second line message_format auto Detected as RFC5424 and parsed from the second line. The RFC3164 header with web01 and nginx is lost.
first line\n<14>second line in_syslog with <parse> @type none {"message":"second line"}. first line is dropped.
To Reproduce

  1. Run Fluentd with the configuration below.

  2. Send the datagrams:

    require 'socket'
s = UDPSocket.new
s.send("first line\n<14>1 2026-04-25T16:43:29Z host app 123 ID - second line\n", 0, '127.0.0.1', 5141)
s.send("first line\n<14>Apr 25 16:43:29 host app: second line\n", 0, '127.0.0.1', 5142)
s.send("first line\n<14>second line\n", 0, '127.0.0.1', 5143)
Expected behavior

None of these datagrams starts with a syslog header. Each one is emitted as unmatched_line, and a warning is logged. parser_engine string already does this for the first two. in_syslog with @type none already does this for first line alone.

For message_format auto, a message that starts with an RFC3164 header is detected as RFC3164.

Your Environment
- Fluentd version: 1.20.1 (master at bd36a9ee)
- Package version: N/A (from source)
- Operating system: CachyOS
- Kernel version: 7.3.1-1-cachyos
- Ruby: 4.0.9
Your Configuration
<source>
  @type syslog
  port 5141
  bind 127.0.0.1
  tag rfc5424
  emit_unmatched_lines true
  <parse>
    @type syslog
    message_format rfc5424
  </parse>
</source>

<source>
  @type syslog
  port 5142
  bind 127.0.0.1
  tag rfc3164
  emit_unmatched_lines true
</source>

<source>
  @type syslog
  port 5143
  bind 127.0.0.1
  tag none
  emit_unmatched_lines true
  <parse>
    @type none
  </parse>
</source>

<match **>
  @type stdout
</match>
Your Error Log
No warning or error is logged. The output is:

2026-04-26 01:43:29.000000000 +0900 rfc5424.user.info: {"host":"host","ident":"app","pid":"123","msgid":"ID","extradata":"-","message":"second line"}
2026-04-25 16:43:29.000000000 +0900 rfc3164.kern.emerg: {"host":"host","ident":"app","message":"second line"}
2026-11-05 17:45:19.501588606 +0900 none.user.info: {"message":"second line"}
Additional context

These regexps start with ^:

  • RFC5424_PRI_REGEXP, RFC3164_PRI_REGEXP and REGEXP_DETECT_RFC5424 in lib/fluent/plugin/parser_syslog.rb
  • SYSLOG_REGEXP in lib/fluent/plugin/in_syslog.rb

REGEXP_DETECT_RFC5424 is used by both engines. With parser_engine string, the third input is also detected as RFC5424, and the parser raises TimeParseError.

Using \A would anchor them to the start of the message. Note that a message with a leading empty line would then be unmatched too. The string parser already treats it as unmatched.

SYSLOG_REGEXP has used ^ since in_syslog was added. In parser_syslog, the results changed in v1.10.0. v1.9.3 yields nil, nil for the first and third inputs, because it matched RFC5424 with \A. For the second input, v1.9.3 also parses from the second line, but with pri 14.

#2873 asks to support messages that contain newlines. This issue is only about where the header is looked for.

Dominant language
Ruby
Stars
13.6k
Forks
1.4k
Avg merge
1d 4h
Merged PRs (30d)
56

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from fluent/fluentd

All issues in fluent/fluentd

Similar issues

More Ruby issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.