[gRPC aio] Prior RPC error logs leak into later RPC breadcrumbs despite isolation_scope

Start with the gRPC aio server integration in sentry_sdk/integrations/grpc/aio/server.py, then trace how its isolation scope interacts with gRPC's post-handler error logging and LoggingIntegration breadcrumbs. Reproduce the two-call case described in the issue and inspect the second event's breadcrumbs. Done means RPC 2 contains its own breadcrumbs without inheriting RPC 1's error log, while gRPC still logs the exception normally.

How do you use Sentry?

Self-hosted/on-premise. The reproducer captures events in memory and requires no Sentry server.

Version

sentry-sdk 2.68.1; grpcio 1.83.1; protobuf 7.36.1; Python 3.12.

Steps to Reproduce

Install grpcio==1.83.1, protobuf==7.36.1, and sentry-sdk==2.68.1. Save this as issue_repro.py and run python issue_repro.py. It starts a local gRPC server, makes two failing unary-unary calls, and captures Sentry events in memory. The localhost DSN is never contacted.

"""Reproduce cross-RPC breadcrumbs with no Sentry server or generated protobuf files."""

import asyncio
import logging

import grpc
import sentry_sdk
from sentry_sdk.integrations.grpc import GRPCIntegration
from sentry_sdk.integrations.logging import LoggingIntegration
from sentry_sdk.transport import Transport


class MemoryTransport(Transport):
    def __init__(self, options):
        super().__init__(options)
        self.events = []

    def capture_envelope(self, envelope):
        event = envelope.get_event()
        if event and any(
            value.get('mechanism', {}).get('type') == 'grpc'
            for value in event.get('exception', {}).get('values', [])
        ):
            self.events.append(event)


class GrpcLogSignal(logging.Handler):
    def __init__(self):
        super().__init__()
        self.arrived = asyncio.Event()

    def emit(self, record):
        if record.name == 'grpc._cython.cygrpc' and record.getMessage().startswith('Unexpected ['):
            self.arrived.set()


async def fail(request, context):
    sentry_sdk.add_breadcrumb(message=f'rpc:{request.decode()}')
    raise RuntimeError(f'rpc:{request.decode()}')


async def main():
    sink = MemoryTransport({'dsn': 'http://public@localhost/1'})
    sentry_sdk.init(
        dsn='http://public@localhost/1',
        transport=sink,
        default_integrations=False,
        integrations=[
            GRPCIntegration(),
            LoggingIntegration(level=logging.INFO, event_level=logging.ERROR),
        ],
    )
    signal = GrpcLogSignal()
    logger = logging.getLogger('grpc._cython.cygrpc')
    logger.addHandler(signal)
    server = grpc.aio.server()
    server.add_generic_rpc_handlers((
        grpc.method_handlers_generic_handler('repro.Service', {
            'Fail': grpc.unary_unary_rpc_method_handler(
                fail, request_deserializer=lambda data: data,
                response_serializer=lambda data: data,
            ),
        }),
    ))
    port = server.add_insecure_port('127.0.0.1:0')
    await server.start()
    try:
        async with grpc.aio.insecure_channel(f'127.0.0.1:{port}') as channel:
            call = channel.unary_unary('/repro.Service/Fail')
            for number in (1, 2):
                signal.arrived.clear()
                try:
                    await call(str(number).encode())
                except grpc.aio.AioRpcError:
                    pass
                await asyncio.wait_for(signal.arrived.wait(), 5)
                event = sink.events[number - 1]
                messages = [b['message'] for b in event.get('breadcrumbs', {}).get('values', [])]
                print(f'RPC {number} event breadcrumbs: {messages}')
    finally:
        await server.stop(0)
        logger.removeHandler(signal)
        sentry_sdk.get_client().close()

    second = [b['message'] for b in sink.events[1]['breadcrumbs']['values']]
    assert any('Unexpected [RuntimeError]' in message for message in second)
    print('Reproduced: second RPC inherited first RPC error log')


if __name__ == '__main__':
    asyncio.run(main())

Expected Result

The Sentry error event for RPC 2 should contain only breadcrumbs generated during RPC 2. It must not contain the gRPC error log from RPC 1. gRPC can still log the exception normally.

Actual Result

Output from the script above with the pinned versions:

RPC 1 event breadcrumbs: ['rpc:1']
RPC 2 event breadcrumbs: ['Unexpected [RuntimeError] raised by servicer method [/repro.Service/Fail]', 'rpc:2']
Reproduced: second RPC inherited first RPC error log

This differs from #5894, which fixed the absence of an async gRPC isolation_scope() in 2.59.0. In 2.68.1, Sentry captures the exception inside its per-handler scope and re-raises. After that scope exits, gRPC's outer _handle_exceptions logs Unexpected [...]. LoggingIntegration adds the record to the restored outer scope. A subsequent request forks that scope and inherits the breadcrumb.

A local application workaround clears the active Sentry request scope at the start of the RPC handler. It prevents the stale log from reaching the next RPC event, but also removes legitimate breadcrumbs added by middleware earlier in the current RPC. Is there a way for the integration to isolate the entire gRPC RPC task, including the framework's post-handler exception log, or to start the request scope without inherited breadcrumbs?