JournalHandler: Sanitize extra fields to be valid journald fields
Nobody has claimed this yet.
Assessment
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Newbie friendliness
- 38/100
- Issue type
- Bug
- Clarity
- Mostly clear
- Activity status
- Stale
- Tech stack
- python
- Domain
- observability
Research direction
Start at the Python systemd.journal JournalHandler entry point and compare its handling of logging extra fields with the sanitation code linked from cysystemd/_journal.pyx. Verify the behavior using the structured_data logging example; done means valid extra fields are no longer silently omitted by journald.
Written by the indexing model from the issue text.
Description
I noticed an incontinence with JournalHandler when using it in an existing code base. I already use the Python logging module and all my logging already includes structured data like this:
LOG.info('test', extra={'structured_data': 23})
The structured_data field does not make it into journald and is silently omitted. I checked https://github.com/mosquito/cysystemd and it has sanitation code which works for me:
As the status of this project and relationship with https://github.com/mosquito/cysystemd is not clear I just wanted to document this here.
Edit:
I ended up using python-systemd instead of cysystemd mainly because of the less cluttered fields it outputs so I needed to work around the issue described here. I needed to workaround another limitation which is that journald cannot handle nested fields so I combined the two workarounds into this (also I want to log sets so it includes a workaround for this as well):
import logging
import json
from systemd.journal import JournalHandler
class SetJSONEncoder(json.JSONEncoder):
def default(self, obj):
if isinstance(obj, set):
return list(obj)
return json.JSONEncoder.default(self, obj)
class LogExtraAsJsonDataAdapter(logging.LoggerAdapter):
def process(self, msg, kwargs):
if 'extra' in kwargs:
kwargs['extra'] = {'JSON_SD': json.dumps(kwargs['extra'], cls=SetJSONEncoder)}
return msg, kwargs
_LOG = logging.getLogger(__name__)
_LOG.addHandler(JournalHandler())
LOG = LogExtraAsJsonDataAdapter(_LOG, {})
The idea here is that Journalbeat sends those logs to Logstash where JSON_SD is JSON decoded and included into the log event.
- Dominant language
- C
- Stars
- 522
- Forks
- 79
- PR merge metrics
- No merged PRs in 30d
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from systemd/python-systemd
-
Difficulty 3/5 1-2 days Newbie friendliness 48/100
systemd/python-systemd#177 · 1 comment · 1 reaction ·
-
Difficulty 3/5 1-2 days Newbie friendliness 42/100
systemd/python-systemd#169 · 1 comment ·
-
Difficulty 4/5 3-5 days Newbie friendliness 48/100
systemd/python-systemd#167 · 19 comments ·
-
Difficulty 5/5 Over a week Newbie friendliness 35/100
systemd/python-systemd#152 · 1 comment · 13 reactions ·
-
Difficulty 4/5 3-5 days Newbie friendliness 45/100
systemd/python-systemd#146 · 2 comments ·
All issues in systemd/python-systemd
Similar issues
-
internal.h中，漏掉了1个定义。 Open
Difficulty 1/5 Under an hour Newbie friendliness 95/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
-
Difficulty 1/5 Under an hour Newbie friendliness 90/100
kovidgoyal/kitty#10516 ·
-
Difficulty 1/5 Under an hour Newbie friendliness 80/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
zephyrproject-rtos/zephyr#120011 ·