Disposing the host deadlocks when an EndpointDataSource changes during disposal

Start with src/Http/Routing/src/DataSourceDependentCache.cs, especially Dispose() and Initialize(), and compare the registration cleanup in CompositeEndpointDataSource.HandleChange. Use the linked repro project to reproduce the hang while a changing EndpointDataSource overlaps host disposal. Done means disposal completes under that race and a change arriving during disposal is safely applied or ignored.

Is there an existing issue for this?

• I have searched the existing issues

Describe the bug

Disposing a host can hang forever when an EndpointDataSource signals a change on another thread while the host is being disposed.

DataSourceDependentCache<T>.Dispose() takes _lock and then disposes its change token registration. When the registration's callback, Initialize(), has already been invoked on another thread, disposing the registration waits for that callback to return. The callback is waiting to take _lock, which Dispose() holds, so neither thread continues and IHost.DisposeAsync() never completes.

The thread that signals the change only needs to have entered the callback before Dispose() takes the lock, so the window is small. It is still hit regularly: every cache built on DataSourceDependentCache<T> is exposed, including the router's matcher (DataSourceDependentMatcher) and AuthorizationPolicyCache, and we have dumps of both deadlocking.

We serve endpoints from a custom EndpointDataSource whose endpoints come from definitions that can be updated while the application runs. An update that arrived while an integration test disposed its host hung the test run until the CI step timed out.

Expected Behavior

IHost.DisposeAsync() completes. A change that arrives while the host is being disposed is either applied or ignored.

Steps To Reproduce

Repro project: https://github.com/glen-84/aspnetcore-dispose-deadlock-repro

It is the ASP.NET Core Empty template (dotnet new web) with only Program.cs changed. Program.cs starts an app whose only endpoints come from an EndpointDataSource that a background thread keeps changing, sends one request so the router builds its matcher, then stops and disposes the app while the endpoints keep changing, in a loop:

using System.Diagnostics;
using Microsoft.AspNetCore.Routing.Patterns;
using Microsoft.Extensions.Primitives;

// Starts an app whose endpoints come from a data source that changes on a background thread,
// then disposes the app while the endpoints keep changing, in a loop.
var stopwatch = Stopwatch.StartNew();
using var client = new HttpClient();

for (var iteration = 1; ; iteration++)
{
    var builder = WebApplication.CreateSlimBuilder();
    builder.WebHost.UseUrls("http://127.0.0.1:0");
    builder.Logging.ClearProviders();

    var app = builder.Build();
    var dataSource = new ChangingEndpointDataSource();
    ((IEndpointRouteBuilder)app).DataSources.Add(dataSource);

    await app.StartAsync();

    // The first request builds the router's endpoint cache.
    await client.GetAsync(new Uri(new Uri(app.Urls.First()), "/a"));

    var changing = true;
    var changer = new Thread(() =>
    {
        while (Volatile.Read(ref changing))
        {
            try
            {
                dataSource.ChangeEndpoints();
            }
            catch (AggregateException)
            {
                // A change that reaches the router after the service provider is disposed throws.
            }
        }
    });
    changer.Start();

    await app.StopAsync();

    var dispose = Task.Run(() => app.DisposeAsync().AsTask());

    if (await Task.WhenAny(dispose, Task.Delay(TimeSpan.FromSeconds(30))) != dispose)
    {
        Console.WriteLine(
            $"Dispose {iteration} hung after {stopwatch.Elapsed:g}. "
            + $"Stacks: dotnet-stack report --process-id {Environment.ProcessId}");
        await Task.Delay(Timeout.Infinite);
    }

    Volatile.Write(ref changing, false);
    changer.Join();

    if (iteration % 100 == 0)
    {
        Console.WriteLine($"{iteration} disposes completed after {stopwatch.Elapsed:g}");
    }
}

internal sealed class ChangingEndpointDataSource : EndpointDataSource
{
    private readonly Endpoint[] _first = [CreateEndpoint("/a")];
    private readonly Endpoint[] _second = [CreateEndpoint("/a"), CreateEndpoint("/b")];
    private IReadOnlyList<Endpoint> _endpoints;
    private CancellationTokenSource _changeTokenSource = new();

    public ChangingEndpointDataSource()
    {
        _endpoints = _first;
    }

    public override IReadOnlyList<Endpoint> Endpoints => Volatile.Read(ref _endpoints);

    public override IChangeToken GetChangeToken()
        => new CancellationChangeToken(Volatile.Read(ref _changeTokenSource).Token);

    public void ChangeEndpoints()
    {
        Volatile.Write(ref _endpoints, ReferenceEquals(_endpoints, _first) ? _second : _first);
        Interlocked.Exchange(ref _changeTokenSource, new CancellationTokenSource()).Cancel();
    }

    private static Endpoint CreateEndpoint(string pattern)
        => new RouteEndpointBuilder(
            context => context.Response.WriteAsync("ok"),
            RoutePatternFactory.Parse(pattern),
            order: 0).Build();
}

1. dotnet run --configuration Release
2. When it prints Dispose <n> hung, run the dotnet-stack command it prints in another terminal (dotnet tool install --global dotnet-stack).

On ASP.NET Core 10.0.8, Linux x64, 20 cores, four runs hung at disposes 5, 8, 12, and 13.

Exceptions (if any)

No exception. The two threads of the hung process, trimmed:

Thread disposing the host:
  System.Threading.SpinWait.SpinOnceCore(Int32)
  System.Threading.CancellationTokenSource+Registrations.WaitForCallbackToComplete(Int64)
  Microsoft.AspNetCore.Routing.DataSourceDependentCache`1.Dispose()
  Microsoft.AspNetCore.Routing.Matching.DataSourceDependentMatcher+Lifetime.Dispose()
  Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceProviderEngineScope.DisposeAsync()
  Microsoft.Extensions.Hosting.Internal.Host+<DisposeAsync>d__20.MoveNext()
  Microsoft.Extensions.Hosting.Internal.Host.DisposeAsync()
  Program+<>c__DisplayClass0_0.<<Main>$>b__1()

Thread changing the endpoints:
  System.Threading.Monitor.Enter_Slowpath(System.Object)
  System.Threading.Monitor.Enter(System.Object, Boolean ByRef)
  Microsoft.AspNetCore.Routing.DataSourceDependentCache`1.Initialize()
  Microsoft.AspNetCore.Routing.DataSourceDependentCache`1.<.ctor>b__9_0(System.Object)
  System.Threading.CancellationTokenSource.ExecuteCallbackHandlers(Boolean)
  Microsoft.AspNetCore.Routing.CompositeEndpointDataSource.HandleChange(Boolean)
  Microsoft.Extensions.Primitives.ChangeToken+ChangeTokenRegistration`1.OnChangeTokenFired()
  System.Threading.CancellationTokenSource.ExecuteCallbackHandlers(Boolean)
  ChangingEndpointDataSource.ChangeEndpoints()
  Program+<>c__DisplayClass0_0.<<Main>$>b__0()

In our own tests the same pair of threads deadlocks on the DataSourceDependentCache<T> inside AuthorizationPolicyCache, which WebApplication adds through its automatic UseAuthorization().

.NET Version

10.0.8 (the repro). We also hit it on 11.0.0-rc.3.

Anything else?

Disposing the registration after releasing the lock would avoid the deadlock: take _disposable and set _disposed under the lock, then dispose the registration outside it. Initialize() already checks _disposed before it registers a new callback. CompositeEndpointDataSource.HandleChange disposes its registrations outside its lock for the same reason: "Disposing registrations can block on user defined code on running on other threads that could try to acquire the _lock."

A change that reaches the router after the service provider has been disposed also throws ObjectDisposedException from DataSourceDependentMatcher.CreateMatcher back into the thread that signalled the change (the AggregateException the repro catches).

Our workaround is to stop changing the endpoints from an IHostedService.StopAsync, which waits for a change in progress, so no change overlaps the disposal of the host.

A deadlock of the same shape, between a change token callback and a lock, is reported for Kestrel's certificate watch in #70027.

Drafted by Claude (Anthropic AI assistant).