perf: Source context re-reads the whole file via loader.get_source() for every frame

Abierto Apto para principiantes
  • ¿Algo no está bien?
#8,089 1 comentario 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

@alexon1234 ya está trabajando en esto.

Desde el 20/10/2026.

  • #8092 de @alexon1234 — abierto

Evaluación

Dificultad
2/5
Tiempo estimado
1-3 horas
Aptitud para principiantes
72/100
Tipo de issue
Error
Claridad
Bien especificado
Estado de actividad
Activo
Stack tecnológico
python
Área
backend, performance

Línea de trabajo

Empieza leyendo get_lines_from_file() en sentry_sdk.utils y compara cómo usa loader.get_source() con su alternativa linecache. El reproductor del issue mide las llamadas al loader y comprueba que el payload de la excepción siga siendo idéntico; usa esas comprobaciones para confirmar que el cambio está hecho.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

Improvement Python
Problem Statement

When include_source_context=True (the default), get_lines_from_file() calls loader.get_source(module) and then .splitlines() for every frame of every event, whenever the frame's module has a __loader__ with get_source (i.e. any normally imported module). SourceFileLoader.get_source() re-reads and decodes the whole file each time; linecache, which is the fallback a few lines below, keeps it in memory.

All of this runs synchronously in the thread that raises/logs the error, before the event reaches the transport worker. In asyncio apps that log many handled exceptions in a burst, this cost is paid on the event loop (related: #2824). sample_rate / before_send don't avoid it, since the frames are built before they run (#4365).

Reproducer (only sentry_sdk + stdlib, events go to a no-op transport):

repro_source_context.py 
"""Source context re-reads the whole source file through loader.get_source() for every frame of every event."""
import importlib.machinery, linecache, logging, os, sys, tempfile, time

import sentry_sdk
import sentry_sdk.utils
from sentry_sdk.transport import Transport

SENT = []


class NullTransport(Transport):
    def capture_envelope(self, envelope):
        SENT.append(envelope.get_event())


sentry_sdk.init(dsn="http://[email protected]/1", transport=NullTransport())

# A ~1,500-line module; the exception passes through 10 frames in it.
code = "def f0():\n    raise TimeoutError('boom')\n"
code += "".join(f"def f{i}():\n    return f{i - 1}()\n" for i in range(1, 10))
code += "".join(f"# filler line {j}\n" for j in range(1500))
tmp = tempfile.mkdtemp()
with open(os.path.join(tmp, "bigmod.py"), "w") as f:
    f.write(code)
sys.path.insert(0, tmp)
import bigmod  # noqa: E402

calls = {"get_source": 0}
_get_source = importlib.machinery.SourceFileLoader.get_source


def counting_get_source(self, name):
    calls["get_source"] += 1
    return _get_source(self, name)


importlib.machinery.SourceFileLoader.get_source = counting_get_source

log = logging.getLogger("repro")
log.addHandler(logging.NullHandler())
log.propagate = False


def emit():
    try:
        bigmod.f9()
    except TimeoutError:
        log.exception("something failed")


def bench(n=1000):
    SENT.clear()
    calls["get_source"] = 0
    t = time.perf_counter()
    for _ in range(n):
        emit()
    return 1000 * (time.perf_counter() - t) / n, calls["get_source"] / n, SENT[-1]


_orig = sentry_sdk.utils.get_lines_from_file


def linecache_first(filename, lineno, max_length=None, loader=None, module=None):
    # Prefer the cached linecache copy; only fall back to the loader (zipimport,
    # frozen apps, ...) when linecache has nothing for this file.
    if linecache.getlines(filename):
        loader = None
    return _orig(filename, lineno, max_length, loader, module)


emit()  # warm up
ms_a, gs_a, ev_a = bench()
sentry_sdk.utils.get_lines_from_file = linecache_first
ms_b, gs_b, ev_b = bench()

print(f"sentry-sdk {sentry_sdk.VERSION}, Python {sys.version.split()[0]}")
print(f"stock:            {ms_a:.3f} ms/event, loader.get_source() calls/event: {gs_a:.0f}")
print(f"linecache first:  {ms_b:.3f} ms/event, loader.get_source() calls/event: {gs_b:.0f}")
print("identical exception payload:", ev_a["exception"] == ev_b["exception"])

Results with sentry-sdk==2.72.0 (same code on master), Linux x86_64:

Python 3.14.8
stock:            3.693 ms/event, loader.get_source() calls/event: 11
linecache first:  2.166 ms/event, loader.get_source() calls/event: 0
identical exception payload: True

Python 3.13.16
stock:            2.937 ms/event, loader.get_source() calls/event: 11
linecache first:  2.466 ms/event, loader.get_source() calls/event: 0
identical exception payload: True
Solution Brainstorm

Read source from linecache first and only fall back to the loader when linecache has nothing for that file (zipimport, PyInstaller/frozen apps, see #812). Alternatively linecache.getlines(filename, frame.f_globals) gives the loader fallback and caching in one call. The output payload is unchanged (checked in the script).

Lenguaje dominante
Python
Estrellas
2.2k
Forks
698
Merge medio
1 d 3 h
PR fusionados (30 d)
158

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de getsentry/sentry-python

Todos los issues de getsentry/sentry-python

Issues similares

Más issues de Python

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.