[gRPC aio] Prior RPC error logs leak into later RPC breadcrumbs despite isolation_scope

Abierto
  • ¿Algo no está bien?
#8,152 3 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
4/5
Tiempo estimado
3-5 días
Aptitud para principiantes
48/100
Tipo de issue
Error
Claridad
Bastante claro
Estado de actividad
Activo
Stack tecnológico
grpc, python
Área
backend, observability

Línea de trabajo

Start with the gRPC aio server integration in sentry_sdk/integrations/grpc/aio/server.py, then trace how its isolation scope interacts with gRPC's post-handler error logging and LoggingIntegration breadcrumbs. Reproduce the two-call case described in the issue and inspect the second event's breadcrumbs. Done means RPC 2 contains its own breadcrumbs without inheriting RPC 1's error log, while gRPC still logs the exception normally.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

Bug Python

How do you use Sentry?

Self-hosted/on-premise. The reproducer captures events in memory and requires no Sentry server.

Version

sentry-sdk 2.68.1; grpcio 1.83.1; protobuf 7.36.1; Python 3.12.

Steps to Reproduce

Install grpcio==1.83.1, protobuf==7.36.1, and sentry-sdk==2.68.1. Save this as issue_repro.py and run python issue_repro.py. It starts a local gRPC server, makes two failing unary-unary calls, and captures Sentry events in memory. The localhost DSN is never contacted.

"""Reproduce cross-RPC breadcrumbs with no Sentry server or generated protobuf files."""

import asyncio
import logging

import grpc
import sentry_sdk
from sentry_sdk.integrations.grpc import GRPCIntegration
from sentry_sdk.integrations.logging import LoggingIntegration
from sentry_sdk.transport import Transport


class MemoryTransport(Transport):
    def __init__(self, options):
        super().__init__(options)
        self.events = []

    def capture_envelope(self, envelope):
        event = envelope.get_event()
        if event and any(
            value.get('mechanism', {}).get('type') == 'grpc'
            for value in event.get('exception', {}).get('values', [])
        ):
            self.events.append(event)


class GrpcLogSignal(logging.Handler):
    def __init__(self):
        super().__init__()
        self.arrived = asyncio.Event()

    def emit(self, record):
        if record.name == 'grpc._cython.cygrpc' and record.getMessage().startswith('Unexpected ['):
            self.arrived.set()


async def fail(request, context):
    sentry_sdk.add_breadcrumb(message=f'rpc:{request.decode()}')
    raise RuntimeError(f'rpc:{request.decode()}')


async def main():
    sink = MemoryTransport({'dsn': 'http://public@localhost/1'})
    sentry_sdk.init(
        dsn='http://public@localhost/1',
        transport=sink,
        default_integrations=False,
        integrations=[
            GRPCIntegration(),
            LoggingIntegration(level=logging.INFO, event_level=logging.ERROR),
        ],
    )
    signal = GrpcLogSignal()
    logger = logging.getLogger('grpc._cython.cygrpc')
    logger.addHandler(signal)
    server = grpc.aio.server()
    server.add_generic_rpc_handlers((
        grpc.method_handlers_generic_handler('repro.Service', {
            'Fail': grpc.unary_unary_rpc_method_handler(
                fail, request_deserializer=lambda data: data,
                response_serializer=lambda data: data,
            ),
        }),
    ))
    port = server.add_insecure_port('127.0.0.1:0')
    await server.start()
    try:
        async with grpc.aio.insecure_channel(f'127.0.0.1:{port}') as channel:
            call = channel.unary_unary('/repro.Service/Fail')
            for number in (1, 2):
                signal.arrived.clear()
                try:
                    await call(str(number).encode())
                except grpc.aio.AioRpcError:
                    pass
                await asyncio.wait_for(signal.arrived.wait(), 5)
                event = sink.events[number - 1]
                messages = [b['message'] for b in event.get('breadcrumbs', {}).get('values', [])]
                print(f'RPC {number} event breadcrumbs: {messages}')
    finally:
        await server.stop(0)
        logger.removeHandler(signal)
        sentry_sdk.get_client().close()

    second = [b['message'] for b in sink.events[1]['breadcrumbs']['values']]
    assert any('Unexpected [RuntimeError]' in message for message in second)
    print('Reproduced: second RPC inherited first RPC error log')


if __name__ == '__main__':
    asyncio.run(main())

Expected Result

The Sentry error event for RPC 2 should contain only breadcrumbs generated during RPC 2. It must not contain the gRPC error log from RPC 1. gRPC can still log the exception normally.

Actual Result

Output from the script above with the pinned versions:

RPC 1 event breadcrumbs: ['rpc:1']
RPC 2 event breadcrumbs: ['Unexpected [RuntimeError] raised by servicer method [/repro.Service/Fail]', 'rpc:2']
Reproduced: second RPC inherited first RPC error log

This differs from #5894, which fixed the absence of an async gRPC isolation_scope() in 2.59.0. In 2.68.1, Sentry captures the exception inside its per-handler scope and re-raises. After that scope exits, gRPC's outer _handle_exceptions logs Unexpected [...]. LoggingIntegration adds the record to the restored outer scope. A subsequent request forks that scope and inherits the breadcrumb.

A local application workaround clears the active Sentry request scope at the start of the RPC handler. It prevents the stale log from reaching the next RPC event, but also removes legitimate breadcrumbs added by middleware earlier in the current RPC. Is there a way for the integration to isolate the entire gRPC RPC task, including the framework's post-handler exception log, or to start the request scope without inherited breadcrumbs?

Lenguaje dominante
Python
Estrellas
2.2k
Forks
698
Merge medio
1 d 3 h
PR fusionados (30 d)
158

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de getsentry/sentry-python

Todos los issues de getsentry/sentry-python

Issues similares

Más issues de Python

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.