Add structured metadata to log records for programmatic consumption

未关闭
  • 哪里不对？
#386 1 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

维护者通常 1 天内回复

还没有人认领这个 Issue。

评估

难度
4/5
预计耗时
3-5 天
新手友好度
48/100
Issue 类型
功能
描述清晰度
基本清楚
活跃度
冷清
技术栈
python
领域
observability

调研方向

定位用于单次 LLM 调用的 Python logging 调用点，记录失败情况以及工作流结束时的使用情况摘要。首先跟踪这些调用如何获取模型、令牌、延迟、重试、状态和失败数据，然后验证自定义 handler 能够收集带命名空间的 dd_event 字典，同时保持现有日志输出不变。

由索引模型根据 Issue 内容生成。

描述

enhancement

Priority Level

Medium (Nice to have)

Is your feature request related to a problem? Please describe.

Downstream tools and pipelines that build on Data Designer currently have no programmatic way to access per-call telemetry (model used, token counts, latency, retries, failure reasons). The only signals available are human-readable log messages, which means consumers must either:

  • Parse log strings with brittle regex to extract stats
  • Set DD's log level to INFO/DEBUG and hope the right messages propagate
  • Infer failures indirectly (e.g. by diffing input vs output record IDs)

This makes it difficult to build observability dashboards, compute cost estimates, track model usage across runs, or diagnose slow pipelines — all without resorting to fragile log parsing.

Describe the solution you'd like

Attach structured metadata to Python LogRecord objects via the extra parameter at key call sites. This requires no new dependencies and no changes to existing log output — the human-readable messages stay exactly as they are.

Concretely, add a namespaced extra dict (e.g. dd_event) to log calls at these sites:

  1. Per-LLM call — model alias, input/output tokens, latency, retries, status (success/failed/filtered), error message if any
  2. Record failure — record ID, column, failure reason, attempt number
  3. Model usage summary — aggregate token counts and request counts per model at workflow end

Example:

# Existing log line stays unchanged for human readers
logger.debug(
    "LLM call to %s completed in %dms",
    model_alias, latency_ms,
    extra={"dd_event": {
        "type": "llm_call",
        "model": model_alias,
        "column": column_name,
        "record_id": record_id,
        "input_tokens": 285,
        "output_tokens": 130,
        "latency_ms": 340,
        "retries": 0,
        "status": "success",
    }},
)

Consumers attach a lightweight handler to collect events:

class EventCollector(logging.Handler):
    def __init__(self):
        super().__init__()
        self.events = []

    def emit(self, record):
        event = getattr(record, "dd_event", None)
        if event:
            self.events.append(event)

This enables several use cases without DD needing to expose new public API:

  • Live monitoring — attach a handler during create()/preview() to stream events as they happen
  • Offline analytics — write events to JSONL for post-hoc cost analysis, latency profiling, or model comparison
  • Observability integration — forward events to Prometheus, OpenTelemetry, or custom dashboards
  • Debugging — filter events by record ID to trace a single record's journey through the pipeline

Describe alternatives you've considered

  • Structured fields on CreateResult/PreviewResult (e.g. result.model_usage, result.failed_records). This would be cleaner for aggregate stats but doesn't support live streaming and requires new public API surface. Could complement the logging approach for summary data.

  • Callback / event hook parameter on create()/preview() (e.g. on_event=my_handler). More explicit contract with typed event dataclasses, but a larger API change and less idiomatic Python than logging.

  • Adopting structlog for structured logging throughout DD. Powerful but adds a dependency and is a bigger architectural change. The extra approach is a stepping stone that's compatible with a future structlog migration.

Additional context

Using a single namespaced key (dd_event) rather than flat extra keys avoids collisions with LogRecord's built-in attributes and with any other libraries that use extra. The convention is simple: if dd_event is present on a record, it contains a typed dict with a "type" discriminator field.

This is a non-breaking, additive change — existing users who don't attach a custom handler see zero difference in behavior or output.

主要语言
Python
星标
2.4k
派生
243
平均合并
1 天 3 小时
30 天内合并 PR
90

环境准备

从这里开始

  1. 先读完整个 Issue，再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库，在一个分支上完成修改。
  4. 提交 Pull Request，并在描述里引用这个 Issue 编号。

NVIDIA-NeMo/DataDesigner 的其他 Issue

查看 NVIDIA-NeMo/DataDesigner 的全部 Issue

相似的 Issue

更多 Python Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。