JournalHandler: Sanitize extra fields to be valid journald fields

Aperta
#69 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
3/5
Tempo stimato
1-2 giorni
Idoneità per principianti
38/100
Tipo di issue
Bug
Chiarezza
Abbastanza chiara
Stato di attività
Ferma
Stack tecnologico
python
Ambito
observability

Direzione di ricerca

Start at the Python systemd.journal JournalHandler entry point and compare its handling of logging extra fields with the sanitation code linked from cysystemd/_journal.pyx. Verify the behavior using the structured_data logging example; done means valid extra fields are no longer silently omitted by journald.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

I noticed an incontinence with JournalHandler when using it in an existing code base. I already use the Python logging module and all my logging already includes structured data like this:

LOG.info('test', extra={'structured_data': 23})

The structured_data field does not make it into journald and is silently omitted. I checked https://github.com/mosquito/cysystemd and it has sanitation code which works for me:

https://github.com/mosquito/cysystemd/blob/e0acede93387d51e4d6b20fdc278b2675052958d/cysystemd/_journal.pyx#L26-L34

As the status of this project and relationship with https://github.com/mosquito/cysystemd is not clear I just wanted to document this here.

Edit:

I ended up using python-systemd instead of cysystemd mainly because of the less cluttered fields it outputs so I needed to work around the issue described here. I needed to workaround another limitation which is that journald cannot handle nested fields so I combined the two workarounds into this (also I want to log sets so it includes a workaround for this as well):

import logging
import json

from systemd.journal import JournalHandler


class SetJSONEncoder(json.JSONEncoder):
    def default(self, obj):
        if isinstance(obj, set):
            return list(obj)
        return json.JSONEncoder.default(self, obj)


class LogExtraAsJsonDataAdapter(logging.LoggerAdapter):
    def process(self, msg, kwargs):
        if 'extra' in kwargs:
            kwargs['extra'] = {'JSON_SD': json.dumps(kwargs['extra'], cls=SetJSONEncoder)}
        return msg, kwargs


_LOG = logging.getLogger(__name__)
_LOG.addHandler(JournalHandler())
LOG = LogExtraAsJsonDataAdapter(_LOG, {})

The idea here is that Journalbeat sends those logs to Logstash where JSON_SD is JSON decoded and included into the log event.

Lingua principale
C
Stelle
522
Fork
79
Nessuna PR unita negli ultimi 30g

Guida per i contributori

Nessuna guida per i contributori indicizzata per questo repository

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di systemd/python-systemd

Tutte le issue di systemd/python-systemd

Issue simili

Altre issue su C

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.