[gRPC aio] Prior RPC error logs leak into later RPC breadcrumbs despite isolation_scope

Offen
  • Stimmt etwas nicht?
#8,152 3 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen

Maintainer antworten meist innerhalb von 1 Tag

Dieses Issue hat noch niemand übernommen.

Bewertung

Schwierigkeit
4/5
Geschätzter Aufwand
3-5 Tage
Anfängerfreundlichkeit
48/100
Issue-Typ
Bug
Klarheit
Größtenteils klar
Aktivitätsstatus
Aktiv
Tech-Stack
grpc, python
Bereich
backend, observability

Rechercherichtung

Start with the gRPC aio server integration in sentry_sdk/integrations/grpc/aio/server.py, then trace how its isolation scope interacts with gRPC's post-handler error logging and LoggingIntegration breadcrumbs. Reproduce the two-call case described in the issue and inspect the second event's breadcrumbs. Done means RPC 2 contains its own breadcrumbs without inheriting RPC 1's error log, while gRPC still logs the exception normally.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Beschreibung

Bug Python

How do you use Sentry?

Self-hosted/on-premise. The reproducer captures events in memory and requires no Sentry server.

Version

sentry-sdk 2.68.1; grpcio 1.83.1; protobuf 7.36.1; Python 3.12.

Steps to Reproduce

Install grpcio==1.83.1, protobuf==7.36.1, and sentry-sdk==2.68.1. Save this as issue_repro.py and run python issue_repro.py. It starts a local gRPC server, makes two failing unary-unary calls, and captures Sentry events in memory. The localhost DSN is never contacted.

"""Reproduce cross-RPC breadcrumbs with no Sentry server or generated protobuf files."""

import asyncio
import logging

import grpc
import sentry_sdk
from sentry_sdk.integrations.grpc import GRPCIntegration
from sentry_sdk.integrations.logging import LoggingIntegration
from sentry_sdk.transport import Transport


class MemoryTransport(Transport):
    def __init__(self, options):
        super().__init__(options)
        self.events = []

    def capture_envelope(self, envelope):
        event = envelope.get_event()
        if event and any(
            value.get('mechanism', {}).get('type') == 'grpc'
            for value in event.get('exception', {}).get('values', [])
        ):
            self.events.append(event)


class GrpcLogSignal(logging.Handler):
    def __init__(self):
        super().__init__()
        self.arrived = asyncio.Event()

    def emit(self, record):
        if record.name == 'grpc._cython.cygrpc' and record.getMessage().startswith('Unexpected ['):
            self.arrived.set()


async def fail(request, context):
    sentry_sdk.add_breadcrumb(message=f'rpc:{request.decode()}')
    raise RuntimeError(f'rpc:{request.decode()}')


async def main():
    sink = MemoryTransport({'dsn': 'http://public@localhost/1'})
    sentry_sdk.init(
        dsn='http://public@localhost/1',
        transport=sink,
        default_integrations=False,
        integrations=[
            GRPCIntegration(),
            LoggingIntegration(level=logging.INFO, event_level=logging.ERROR),
        ],
    )
    signal = GrpcLogSignal()
    logger = logging.getLogger('grpc._cython.cygrpc')
    logger.addHandler(signal)
    server = grpc.aio.server()
    server.add_generic_rpc_handlers((
        grpc.method_handlers_generic_handler('repro.Service', {
            'Fail': grpc.unary_unary_rpc_method_handler(
                fail, request_deserializer=lambda data: data,
                response_serializer=lambda data: data,
            ),
        }),
    ))
    port = server.add_insecure_port('127.0.0.1:0')
    await server.start()
    try:
        async with grpc.aio.insecure_channel(f'127.0.0.1:{port}') as channel:
            call = channel.unary_unary('/repro.Service/Fail')
            for number in (1, 2):
                signal.arrived.clear()
                try:
                    await call(str(number).encode())
                except grpc.aio.AioRpcError:
                    pass
                await asyncio.wait_for(signal.arrived.wait(), 5)
                event = sink.events[number - 1]
                messages = [b['message'] for b in event.get('breadcrumbs', {}).get('values', [])]
                print(f'RPC {number} event breadcrumbs: {messages}')
    finally:
        await server.stop(0)
        logger.removeHandler(signal)
        sentry_sdk.get_client().close()

    second = [b['message'] for b in sink.events[1]['breadcrumbs']['values']]
    assert any('Unexpected [RuntimeError]' in message for message in second)
    print('Reproduced: second RPC inherited first RPC error log')


if __name__ == '__main__':
    asyncio.run(main())

Expected Result

The Sentry error event for RPC 2 should contain only breadcrumbs generated during RPC 2. It must not contain the gRPC error log from RPC 1. gRPC can still log the exception normally.

Actual Result

Output from the script above with the pinned versions:

RPC 1 event breadcrumbs: ['rpc:1']
RPC 2 event breadcrumbs: ['Unexpected [RuntimeError] raised by servicer method [/repro.Service/Fail]', 'rpc:2']
Reproduced: second RPC inherited first RPC error log

This differs from #5894, which fixed the absence of an async gRPC isolation_scope() in 2.59.0. In 2.68.1, Sentry captures the exception inside its per-handler scope and re-raises. After that scope exits, gRPC's outer _handle_exceptions logs Unexpected [...]. LoggingIntegration adds the record to the restored outer scope. A subsequent request forks that scope and inherits the breadcrumb.

A local application workaround clears the active Sentry request scope at the start of the RPC handler. It prevents the stale log from reaching the next RPC event, but also removes legitimate breadcrumbs added by middleware earlier in the current RPC. Is there a way for the integration to isolate the entire gRPC RPC task, including the framework's post-handler exception log, or to start the request scope without inherited breadcrumbs?

Vorherrschende Sprache
Python
Sterne
2.2k
Forks
698
Ø Merge
1 T. 3 Std.
Gemergte PRs (30 T.)
158

Entwicklungsumgebung

Erste Schritte

  1. Lesen Sie das ganze Issue und danach den Beitragsleitfaden des Projekts.
  2. Schreiben Sie ins Issue, dass Sie es übernehmen — das erspart doppelte Arbeit.
  3. Forken Sie das Repository und arbeiten Sie in einem Branch.
  4. Öffnen Sie einen Pull Request, der die Issue-Nummer nennt.

Mehr aus getsentry/sentry-python

Alle Issues in getsentry/sentry-python

Ähnliche Issues

Weitere Issues zu Python

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.