Performance issue when stepping through a deep callstack

Start at ext/debug/debug.c lines 92-98 and inspect how rb_make_backtrace is used while stepping through a deep callstack. Compare the proposed rb_threadptr_backtrace_object and debase approaches, including their *_core.h requirements. Done means stepping through roughly 150 calls no longer incurs the reported slowdown and memory pressure.

Your environment

• ruby -v: 2.7.2p137
• rdbg -v: 1.6.2

Describe the bug
When stepping through with a deep callstack, the debugger becomes slower with each step.

To Reproduce
In a codebase which has a method that's ~150 calls deep in the stack, add a breakpoint, debug, and step through.

Expected behavior
It wouldn't be slow

Additional context
We are facing this issue at Stripe, and we tracked down the problem to: https://github.com/ruby/debug/blob/296daa6742e114395a4ce336b8721d48a6e5e52f/ext/debug/debug.c#L92-L98

This seems to be generating the full backtrace through rb_make_backtrace, just to then take the size. rb_make_backtrace (source) generates an array of strings each time it's called. Those strings end up not being used, and after a couple of iterations, you end up with a big memory footprint, which causes the GC to collect them, significantly slowing down the debugger.

Luckily, it looks like you are already aware that this is inefficient (given the // TODO: more efficient API) :)

Looking into alternatives, I came up with a couple of options (but open to any other ideas):

1. Use rb_threadptr_backtrace_object to skip the formatting of the backtrace, saving some extra memory (but still generating more than necessary)
2. debase solves this with a different approach: https://github.com/ruby-debug/debase/blob/5780803819f16190ce50ddfdad5775acf8f95fd1/ext/hacks.h#L26

Both of them, though, require the inclusion of *_core.h, which is not currently possible (debase solves it by using its own version of ruby_core_source)