[gRPC aio] Prior RPC error logs leak into later RPC breadcrumbs despite isolation_scope

Aperta
  • Qualcosa non va?
#8,152 3 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
48/100
Tipo di issue
Bug
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
grpc, python
Ambito
backend, observability

Direzione di ricerca

Start with the gRPC aio server integration in sentry_sdk/integrations/grpc/aio/server.py, then trace how its isolation scope interacts with gRPC's post-handler error logging and LoggingIntegration breadcrumbs. Reproduce the two-call case described in the issue and inspect the second event's breadcrumbs. Done means RPC 2 contains its own breadcrumbs without inheriting RPC 1's error log, while gRPC still logs the exception normally.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Bug Python

How do you use Sentry?

Self-hosted/on-premise. The reproducer captures events in memory and requires no Sentry server.

Version

sentry-sdk 2.68.1; grpcio 1.83.1; protobuf 7.36.1; Python 3.12.

Steps to Reproduce

Install grpcio==1.83.1, protobuf==7.36.1, and sentry-sdk==2.68.1. Save this as issue_repro.py and run python issue_repro.py. It starts a local gRPC server, makes two failing unary-unary calls, and captures Sentry events in memory. The localhost DSN is never contacted.

"""Reproduce cross-RPC breadcrumbs with no Sentry server or generated protobuf files."""

import asyncio
import logging

import grpc
import sentry_sdk
from sentry_sdk.integrations.grpc import GRPCIntegration
from sentry_sdk.integrations.logging import LoggingIntegration
from sentry_sdk.transport import Transport


class MemoryTransport(Transport):
    def __init__(self, options):
        super().__init__(options)
        self.events = []

    def capture_envelope(self, envelope):
        event = envelope.get_event()
        if event and any(
            value.get('mechanism', {}).get('type') == 'grpc'
            for value in event.get('exception', {}).get('values', [])
        ):
            self.events.append(event)


class GrpcLogSignal(logging.Handler):
    def __init__(self):
        super().__init__()
        self.arrived = asyncio.Event()

    def emit(self, record):
        if record.name == 'grpc._cython.cygrpc' and record.getMessage().startswith('Unexpected ['):
            self.arrived.set()


async def fail(request, context):
    sentry_sdk.add_breadcrumb(message=f'rpc:{request.decode()}')
    raise RuntimeError(f'rpc:{request.decode()}')


async def main():
    sink = MemoryTransport({'dsn': 'http://public@localhost/1'})
    sentry_sdk.init(
        dsn='http://public@localhost/1',
        transport=sink,
        default_integrations=False,
        integrations=[
            GRPCIntegration(),
            LoggingIntegration(level=logging.INFO, event_level=logging.ERROR),
        ],
    )
    signal = GrpcLogSignal()
    logger = logging.getLogger('grpc._cython.cygrpc')
    logger.addHandler(signal)
    server = grpc.aio.server()
    server.add_generic_rpc_handlers((
        grpc.method_handlers_generic_handler('repro.Service', {
            'Fail': grpc.unary_unary_rpc_method_handler(
                fail, request_deserializer=lambda data: data,
                response_serializer=lambda data: data,
            ),
        }),
    ))
    port = server.add_insecure_port('127.0.0.1:0')
    await server.start()
    try:
        async with grpc.aio.insecure_channel(f'127.0.0.1:{port}') as channel:
            call = channel.unary_unary('/repro.Service/Fail')
            for number in (1, 2):
                signal.arrived.clear()
                try:
                    await call(str(number).encode())
                except grpc.aio.AioRpcError:
                    pass
                await asyncio.wait_for(signal.arrived.wait(), 5)
                event = sink.events[number - 1]
                messages = [b['message'] for b in event.get('breadcrumbs', {}).get('values', [])]
                print(f'RPC {number} event breadcrumbs: {messages}')
    finally:
        await server.stop(0)
        logger.removeHandler(signal)
        sentry_sdk.get_client().close()

    second = [b['message'] for b in sink.events[1]['breadcrumbs']['values']]
    assert any('Unexpected [RuntimeError]' in message for message in second)
    print('Reproduced: second RPC inherited first RPC error log')


if __name__ == '__main__':
    asyncio.run(main())

Expected Result

The Sentry error event for RPC 2 should contain only breadcrumbs generated during RPC 2. It must not contain the gRPC error log from RPC 1. gRPC can still log the exception normally.

Actual Result

Output from the script above with the pinned versions:

RPC 1 event breadcrumbs: ['rpc:1']
RPC 2 event breadcrumbs: ['Unexpected [RuntimeError] raised by servicer method [/repro.Service/Fail]', 'rpc:2']
Reproduced: second RPC inherited first RPC error log

This differs from #5894, which fixed the absence of an async gRPC isolation_scope() in 2.59.0. In 2.68.1, Sentry captures the exception inside its per-handler scope and re-raises. After that scope exits, gRPC's outer _handle_exceptions logs Unexpected [...]. LoggingIntegration adds the record to the restored outer scope. A subsequent request forks that scope and inherits the breadcrumb.

A local application workaround clears the active Sentry request scope at the start of the RPC handler. It prevents the stale log from reaching the next RPC event, but also removes legitimate breadcrumbs added by middleware earlier in the current RPC. Is there a way for the integration to isolate the entire gRPC RPC task, including the framework's post-handler exception log, or to start the request scope without inherited breadcrumbs?

Lingua principale
Python
Stelle
2.2k
Fork
698
Merge medio
1g 3h
PR unite (30g)
158

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di getsentry/sentry-python

Tutte le issue di getsentry/sentry-python

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.