vectordotdev/vector

Update Syslog source to accept non UTF-8 encoding in syslog message

Open

#20,462 opened on May 8, 2024

View on GitHub
聽(9 comments)聽(3 reactions)聽(0 assignees)Rust聽(21,837 stars)聽(2,126 forks)batch import
domain: codecsgood first issue

Description

A note for the community

  • Please vote on this issue by adding a 馃憤 reaction to the original issue to help the community and maintainers prioritize this request
  • If you are interested in working on this issue or have submitted a pull request, please leave a comment

Problem

Vector drops logs when encounter a syslog message with non UTF-8 characters. Can you add an option to replace non utf-8 characters with U+FFFD or allow passing non-UTF8 text as-is like a Promtail do.

Configuration

No response

Version

0.37.1-distroless-static

Debug Output

No response

Example Data

2024-05-08T07:35:43.209847Z DEBUG source{component_kind="source" component_id=rsyslog component_type=syslog}:connection{peer_addr=172.22.0.4:44600}: vector::sources::util::net::tcp: Accepted a new connection. peer_addr=172.22.0.4:44600

2024-05-08T07:35:44.293974Z ERROR source{component_kind="source" component_id=rsyslog component_type=syslog}:connection{peer_addr=172.22.0.4:44594}: vector::internal_events::codecs: Failed framing bytes. error=Unable to decode input as UTF8 error_code="decoder_frame" error_type="parser_failed" stage="processing" internal_log_rate_limit=true

2024-05-08T07:35:44.294029Z ERROR source{component_kind="source" component_id=rsyslog component_type=syslog}:connection{peer_addr=172.22.0.4:44594}: vector::internal_events::codecs: Internal log [Failed framing bytes.] is being suppressed to avoid flooding.

Additional Context

No response

References

No response

Contributor guide

Tech stack
rustshell
Domain
backendobservability
Issue type
feature
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
1-3 hours
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
active
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
clear
Prerequisites
Rust basicsUnderstanding of syslog protocol
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
50
Research direction
Investigate the syslog source implementation in src/sources/syslog/ and the codec handling in src/codecs/. The issue mentions an option to replace non UTF 8 characters with U+FFFD or pass them as is. Look for existing configuration for encoding behavior and consider adding a new option like 'charset' or 'encoding' to the syslog source config. Review how Promtail handles this for reference.
Update Syslog source to accept non UTF-8 encoding in syslog message 路 vectordotdev/vector#20462 | Good First Issue