Tracing crashes with UnicodeEncodeError on surrogate escapes in hook arguments/results

Open Beginner friendly
#681 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
85/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
python
Domain
tooling

Research direction

Start with _format_message in _tracing.py around line 38 and the hook result tracing call in _manager.py around line 506. Verify tracing with surrogate-containing hook arguments and results no longer raises UnicodeEncodeError, while structural labels remain readable and values are safely represented.

Written by the indexing model from the issue text.

Description

Problem

When pluggy tracing is enabled (e.g. pytest's --debug flag), hook arguments or return values containing surrogate escape characters (like \ud800) cause a UnicodeEncodeError crash. This is because _format_message in _tracing.py uses str() to format values, and the resulting string with literal surrogates cannot be written to most output targets.

Reported originally as pytest-dev/pytest#13750.

Analysis

_format_message currently formats two kinds of data identically using str():

  1. Structural labels passed as positional args — e.g. "finish", "-->", hook names — which are always safe ASCII strings
  2. Python values — hook kwargs values (rendered via the extra dict) and hook return values — which may contain arbitrary data including surrogates

The fix needs to apply repr() only to the value positions, not to structural labels.

Intended Solution

  1. In _tracing.py line 38 — use {value!r} for the extra/kwargs dict values:

    lines.append(f"{indent}    {name}: {value!r}\n")
    

    This makes kwargs values in trace output show their type (e.g. 'lfplugin' instead of lfplugin, PosixPath('/foo') instead of /foo) and safely escapes surrogates.

  2. In _manager.py line 506 — use repr() on the hook result before passing it as a trace arg:

    hooktrace("finish", hook_name, "-->", repr(outcome.get_result()))
    

    This ensures the result value is safely formatted, while "finish", hook_name, and "-->" remain plain str()-formatted (via _format_message's existing map(str, args)).

  3. Do NOT change the content = " ".join(map(str, args)) line in _format_message. Keeping str() there preserves readable structural output without quoting labels. The caller (_manager.py) is responsible for pre-formatting any unsafe values with repr().

This avoids the double-repr problem and keeps trace output readable:

  finish pytest_runtest_call --> '\ud800' [hook]
      config: <Config object at 0x...>
      plugin_name: 'lfplugin'

Rather than the over-quoted version that blanket repr() in _format_message would produce:

  'finish' 'pytest_runtest_call' '-->' "'\\ ud800'" [hook]

References

  • pytest-dev/pytest#13750
  • #627 (PR with discussion leading to this distilled approach)
Dominant language
Python
Stars
1.7k
Forks
160
Avg merge
21h 4m
Merged PRs (30d)
6

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from pytest-dev/pluggy

All issues in pytest-dev/pluggy

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.