Occasional crash during gcdump

Open
#6,048 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
45/100
Issue type
Bug
Clarity
Needs clarification
Activity status
Active
Tech stack
csharp
Domain
devtools

Research direction

Start in src/Tools/dotnet-gcdump/DotNetHeapDump/EventPipeDotNetHeapDumper.cs at lines 269 and 274, then compare the full log with the caller in HeapDumper.cs at line 103. Investigate the StreamWriter exception, Task.WaitAll path, and subsequent “RootIndex not set” error; done means the failure cause is identified and gcdump behavior is verified against the reported scenario.

Written by the indexing model from the issue text.

Description

bug

Our tools run the same logic as the dotnet-gcdump tool. It occasionally crashes on macOS (default GitHub Actions Runner image) with the stack trace below. Related code:

var heapInfo = new DotNetHeapInfo();
var memoryGraph = new MemoryGraph(50_000);

if (EventPipeDotNetHeapDumper.DumpFromEventPipe(cancellationToken, processId, null,
    memoryGraph, logWriter, 30, heapInfo))
{
    memoryGraph.AllowReading();
    GCHeapDump.WriteMemoryGraph(memoryGraph, outputPath, "dotnet-gcdump");
    return true;
}

Stack trace of the crash:

  0.5s: EventPipe Listener dying
  0.5s: [Error] Exception during gcdump: System.AggregateException: One or more errors occurred. (Specified argument was out of the range of valid values.)
 ---> System.ArgumentOutOfRangeException: Specified argument was out of the range of valid values.
   at System.IO.StreamWriter.Flush(Boolean flushStream, Boolean flushEncoder)
   at System.IO.StreamWriter.WriteFormatHelper(String format, ReadOnlySpan`1 args, Boolean appendNewLine)
   at System.IO.StreamWriter.WriteLine(String format, Object arg0)
   at Microsoft.Diagnostics.Tools.GCDump.EventPipeDotNetHeapDumper.<>c__DisplayClass3_1.<DumpFromEventPipe>b__5() in /_/src/Tools/dotnet-gcdump/DotNetHeapDump/EventPipeDotNetHeapDumper.cs:line 269
   at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location ---
   at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
   --- End of inner exception stack trace ---
   at System.Threading.Tasks.Task.WaitAllCore(ReadOnlySpan`1 tasks, Int32 millisecondsTimeout, CancellationToken cancellationToken)
   at System.Threading.Tasks.Task.WaitAll(Task[] tasks, Int32 millisecondsTimeout)
   at Microsoft.Diagnostics.Tools.GCDump.EventPipeDotNetHeapDumper.DumpFromEventPipe(CancellationToken ct, Int32 processId, String diagnosticPort, MemoryGraph memoryGraph, TextWriter log, Int32 timeout, DotNetHeapInfo dotNetInfo) in /_/src/Tools/dotnet-gcdump/DotNetHeapDump/EventPipeDotNetHeapDumper.cs:line 274
[  0.5s: Done Dumping .NET heap success=True]

---- System.ApplicationException : RootIndex not set.

This is nearly impossible to reproduce for us.

Full log at: https://github.com/SteeltoeOSS/Steeltoe/actions/runs/34455238672/job/102799980981.
Full source code at: https://github.com/SteeltoeOSS/Steeltoe/blob/main/src/Management/src/Endpoint/Actuators/HeapDump/HeapDumper.cs#L103

Any thoughts on what could be causing this, or where to look further?

Dominant language
C++
Stars
1.3k
Forks
403
Avg merge
2d 11m
Merged PRs (30d)
42

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from dotnet/diagnostics

All issues in dotnet/diagnostics

Similar issues

More C++ issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.