Assertion failure: Stashed signal pending on syscall entry SIGSTKFLT, syscall 108 syscallno -38

Start at src/RecordSession.cc:2028 in RecordSession::process_syscall_entry() and inspect the supplied assertion and tail of the trace dump. Try to reproduce recording rr 5.8.0 in the Fedora toolbox environment on the AMD Ryzen 9 7900X setup, focusing on the SIGSTKFLT/syscall-entry state. Done means the rare assertion is explained and the recording no longer crashes under the reproducing case.

Running rr version 5.8.0 in a fedora-toolbox-41 container running on a machine with a AMD Ryzen 9 7900X 12-Core Processor (zen workaround in place), recording a program that executes a bunch of relatively short-lived child processes.

The rr crash happens only rarely, I was trying to reproduce a segfault that happens in CI sometimes. On the way I found this.

I'm not sure what other details are important.

[FATAL src/RecordSession.cc:2028:process_syscall_entry()] 
 (task 3598048 (rec:3598048) at time 91370)
 -> Assertion `t->desched_rec() || is_rrcall_notify_syscall_hook_exit_syscall( t->regs().original_syscallno(), t->arch()) || t->ip() == t->vm() ->privileged_traced_syscall_ip() .increment_by_syscall_insn_length(t->arch())' failed to hold. Stashed signal pending on syscall entry when it shouldn't be: {signo:SIGSTKFLT,errno:SUCCESS,code:sicode(1)}; regs={ ip:0x44c37cc94f0b args:(0x1,0x44c37cc94f30,0xffffffffffffffff,0x44c37cdba920,0,140729759170496) orig_syscall: 108 syscallno: -38 }; last_execution_resume=0x6ffeda04; sig ip=0x6ffeda04
Tail of trace dump:
{
  real_time:539090.662931 global_time:91350, event:`SYSCALL: mmap' (state:EXITING_SYSCALL) tid:3598052, ticks:9390
rax:0x7f27e98d1000 rbx:0x0 rcx:0xffffffffffffffff rdx:0x1 rsi:0x4008 rdi:0x0 rbp:0x7ffeb2800080 rsp:0x7ffeb2800070 r8:0x3 r9:0x0 r10:0x802 r11:0x246 r12:0x802 r13:0x6fffb580 r14:0x7ffeb2800500 r15:0x1000 rip:0x7f27e98bc2fc eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x9 fs_base:0x0 gs_base:0x0
  { map_file:"/usr/lib64/libdl.so.2", addr:0x7f27e98d1000, length:0x5000, prot_flags:"r--p", file_offset:0x0, device:1048670, inode:90633318, data_file:"/usr/lib64/libdl.so.2", data_offset:0x0, file_size:0x3d10 }
}
{
  real_time:539090.662951 global_time:91351, event:`SYSCALL: mmap' (state:ENTERING_SYSCALL) tid:3598052, ticks:9401
rax:0xffffffffffffffda rbx:0x7f27e98d2000 rcx:0xffffffffffffffff rdx:0x5 rsi:0x1000 rdi:0x7f27e98d2000 rbp:0x7ffeb2800080 rsp:0x7ffeb2800070 r8:0x3 r9:0x1000 r10:0x812 r11:0x246 r12:0x812 r13:0x6fffb580 r14:0x7ffeb2800500 r15:0x33c rip:0x7f27e98bc2fc eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x9 fs_base:0x0 gs_base:0x0
}
{
  real_time:539090.662982 global_time:91352, event:`SYSCALL: mmap' (state:EXITING_SYSCALL) tid:3598052, ticks:9401
rax:0x7f27e98d2000 rbx:0x7f27e98d2000 rcx:0xffffffffffffffff rdx:0x5 rsi:0x1000 rdi:0x7f27e98d2000 rbp:0x7ffeb2800080 rsp:0x7ffeb2800070 r8:0x3 r9:0x1000 r10:0x812 r11:0x246 r12:0x812 r13:0x6fffb580 r14:0x7ffeb2800500 r15:0x33c rip:0x7f27e98bc2fc eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x9 fs_base:0x0 gs_base:0x0
  { map_file:"/usr/lib64/libdl.so.2", addr:0x7f27e98d2000, length:0x1000, prot_flags:"r-xp", file_offset:0x1000, device:1048670, inode:90633318, data_file:"/usr/lib64/libdl.so.2", data_offset:0x1000, file_size:0x3d10 }
}
{
  real_time:539090.663003 global_time:91353, event:`SYSCALL: mmap' (state:ENTERING_SYSCALL) tid:3598052, ticks:9409
rax:0xffffffffffffffda rbx:0x7f27e98d3000 rcx:0xffffffffffffffff rdx:0x1 rsi:0x1000 rdi:0x7f27e98d3000 rbp:0x7ffeb2800080 rsp:0x7ffeb2800070 r8:0x3 r9:0x2000 r10:0x812 r11:0x246 r12:0x812 r13:0x6fffb580 r14:0x7ffeb2800500 r15:0x1125 rip:0x7f27e98bc2fc eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x9 fs_base:0x0 gs_base:0x0
}
{
  real_time:539090.663033 global_time:91354, event:`SYSCALL: mmap' (state:EXITING_SYSCALL) tid:3598052, ticks:9409
rax:0x7f27e98d3000 rbx:0x7f27e98d3000 rcx:0xffffffffffffffff rdx:0x1 rsi:0x1000 rdi:0x7f27e98d3000 rbp:0x7ffeb2800080 rsp:0x7ffeb2800070 r8:0x3 r9:0x2000 r10:0x812 r11:0x246 r12:0x812 r13:0x6fffb580 r14:0x7ffeb2800500 r15:0x1125 rip:0x7f27e98bc2fc eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x9 fs_base:0x0 gs_base:0x0
  { map_file:"/usr/lib64/libdl.so.2", addr:0x7f27e98d3000, length:0x1000, prot_flags:"r--p", file_offset:0x2000, device:1048670, inode:90633318, data_file:"/usr/lib64/libdl.so.2", data_offset:0x2000, file_size:0x3d10 }
}
{
  real_time:539090.663052 global_time:91355, event:`SYSCALL: mmap' (state:ENTERING_SYSCALL) tid:3598052, ticks:9417
rax:0xffffffffffffffda rbx:0x7f27e98d4000 rcx:0xffffffffffffffff rdx:0x3 rsi:0x1000 rdi:0x7f27e98d4000 rbp:0x7ffeb2800080 rsp:0x7ffeb2800070 r8:0x3 r9:0x2000 r10:0x812 r11:0x246 r12:0x812 r13:0x6fffb580 r14:0x7ffeb2800500 r15:0x2488 rip:0x7f27e98bc2fc eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x9 fs_base:0x0 gs_base:0x0
}
{
  real_time:539090.663078 global_time:91356, event:`SYSCALL: mmap' (state:EXITING_SYSCALL) tid:3598052, ticks:9417
rax:0x7f27e98d4000 rbx:0x7f27e98d4000 rcx:0xffffffffffffffff rdx:0x3 rsi:0x1000 rdi:0x7f27e98d4000 rbp:0x7ffeb2800080 rsp:0x7ffeb2800070 r8:0x3 r9:0x2000 r10:0x812 r11:0x246 r12:0x812 r13:0x6fffb580 r14:0x7ffeb2800500 r15:0x2488 rip:0x7f27e98bc2fc eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x9 fs_base:0x0 gs_base:0x0
  { map_file:"/usr/lib64/libdl.so.2", addr:0x7f27e98d4000, length:0x1000, prot_flags:"rw-p", file_offset:0x2000, device:1048670, inode:90633318, data_file:"/usr/lib64/libdl.so.2", data_offset:0x2000, file_size:0x3d10 }
}
{
  real_time:539090.663097 global_time:91357, event:`SYSCALL: mmap' (state:ENTERING_SYSCALL) tid:3598052, ticks:9425
rax:0xffffffffffffffda rbx:0x7f27e98d5000 rcx:0xffffffffffffffff rdx:0x3 rsi:0x8 rdi:0x7f27e98d5000 rbp:0x7ffeb2800080 rsp:0x7ffeb2800070 r8:0xffffffff r9:0x0 r10:0x32 r11:0x246 r12:0x32 r13:0x6fffb580 r14:0x7ffeb2800500 r15:0x7f27e98d5008 rip:0x7f27e98bc2fc eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x9 fs_base:0x0 gs_base:0x0
}
{
  real_time:539090.663114 global_time:91358, event:`SYSCALL: mmap' (state:EXITING_SYSCALL) tid:3598052, ticks:9425
rax:0x7f27e98d5000 rbx:0x7f27e98d5000 rcx:0xffffffffffffffff rdx:0x3 rsi:0x8 rdi:0x7f27e98d5000 rbp:0x7ffeb2800080 rsp:0x7ffeb2800070 r8:0xffffffff r9:0x0 r10:0x32 r11:0x246 r12:0x32 r13:0x6fffb580 r14:0x7ffeb2800500 r15:0x7f27e98d5008 rip:0x7f27e98bc2fc eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x9 fs_base:0x0 gs_base:0x0
  { map_file:"<ZERO>", addr:0x7f27e98d5000, length:0x1000, prot_flags:"rw-p", file_offset:0x0, device:0, inode:0, data_file:"", data_offset:0x0, file_size:0x1000 }
}
{
  real_time:539090.663138 global_time:91359, event:`SYSCALL: close' (state:ENTERING_SYSCALL) tid:3598052, ticks:9606
rax:0xffffffffffffffda rbx:0x7f27e98d1040 rcx:0xffffffffffffffff rdx:0x1 rsi:0x7f27e98d12f8 rdi:0x3 rbp:0x7ffeb2800450 rsp:0x7ffeb28002f8 r8:0x7f27e98d12d0 r9:0x4 r10:0xc0000002 r11:0x246 r12:0x3 r13:0x6fffb580 r14:0x7ffeb2800500 r15:0x7f27e98d5008 rip:0x7f27e98bbffb eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x3 fs_base:0x0 gs_base:0x0
}
{
  real_time:539090.663153 global_time:91360, event:`SYSCALL: close' (state:EXITING_SYSCALL) tid:3598052, ticks:9606
rax:0x0 rbx:0x7f27e98d1040 rcx:0xffffffffffffffff rdx:0x1 rsi:0x7f27e98d12f8 rdi:0x3 rbp:0x7ffeb2800450 rsp:0x7ffeb28002f8 r8:0x7f27e98d12d0 r9:0x4 r10:0xc0000002 r11:0x246 r12:0x3 r13:0x6fffb580 r14:0x7ffeb2800500 r15:0x7f27e98d5008 rip:0x7f27e98bbffb eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x3 fs_base:0x0 gs_base:0x0
}
{
  real_time:539090.663174 global_time:91361, event:`INSTRUCTION_TRAP' tid:3598052, ticks:9641
rax:0x8882c288 rbx:0x9007087f80ba5 rcx:0x0 rdx:0x90070 rsi:0x0 rdi:0x6fffb580 rbp:0x7ffeb2801da0 rsp:0x7ffeb2801b00 r8:0x7f27e98d3088 r9:0x4 r10:0xc0000002 r11:0x246 r12:0x2 r13:0x7f27e98d02f0 r14:0x7f27e98d0128 r15:0x1 rip:0x7f27e98b93fe eflags:0x10202 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0xffffffffffffffff fs_base:0x0 gs_base:0x0
}
{
  real_time:539090.663193 global_time:91362, event:`SYSCALL: access' (state:ENTERING_SYSCALL) tid:3598052, ticks:9642
rax:0xffffffffffffffda rbx:0x9007087f80ba5 rcx:0xffffffffffffffff rdx:0x9007000000000 rsi:0x4 rdi:0x7f27e98c5e90 rbp:0x7ffeb2801da0 rsp:0x7ffeb2801af8 r8:0x7f27e98d3088 r9:0x4 r10:0xc0000002 r11:0x246 r12:0x2 r13:0x7f27e98d02f0 r14:0x7f27e98c5e90 r15:0x1 rip:0x7f27e98bbe9b eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x15 fs_base:0x0 gs_base:0x0
}
{
  real_time:539090.663208 global_time:91363, event:`SYSCALL: access' (state:EXITING_SYSCALL) tid:3598052, ticks:9642
rax:0xfffffffffffffffe rbx:0x9007087f80ba5 rcx:0xffffffffffffffff rdx:0x9007000000000 rsi:0x4 rdi:0x7f27e98c5e90 rbp:0x7ffeb2801da0 rsp:0x7ffeb2801af8 r8:0x7f27e98d3088 r9:0x4 r10:0xc0000002 r11:0x246 r12:0x2 r13:0x7f27e98d02f0 r14:0x7f27e98c5e90 r15:0x1 rip:0x7f27e98bbe9b eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x15 fs_base:0x0 gs_base:0x0
}
{
  real_time:539090.663227 global_time:91364, event:`INSTRUCTION_TRAP' tid:3598052, ticks:9652
rax:0x888699ad rbx:0x7ffeb2801ae0 rcx:0x7ffeb2801ae0 rdx:0x90070 rsi:0x0 rdi:0x7f27e98d08d0 rbp:0x7ffeb2801da0 rsp:0x7ffeb2801ae0 r8:0x7f27e98d3088 r9:0x4 r10:0xc0000002 r11:0x246 r12:0x2 r13:0x7f27e98d02f0 r14:0x7f27e98c5e90 r15:0x1 rip:0x7f27e98b7dbb eflags:0x10246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0xffffffffffffffff fs_base:0x0 gs_base:0x0
}
{
  real_time:539090.663255 global_time:91365, event:`SYSCALL: openat' (state:ENTERING_SYSCALL) tid:3598052, ticks:10181
rax:0xffffffffffffffda rbx:0x6fffba90 rcx:0xffffffffffffffff rdx:0x80000 rsi:0x6fffba90 rdi:0xffffff9c rbp:0x7ffeb2800fa0 rsp:0x7ffeb2800fa0 r8:0x7ffeb2801067 r9:0x0 r10:0x0 r11:0x246 r12:0x7f27e98d02f0 r13:0x0 r14:0x0 r15:0x7ffeb2801080 rip:0x7f27e98bc141 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x101 fs_base:0x0 gs_base:0x0
}
{
  real_time:539090.663316 global_time:91366, event:`PATCH_SYSCALL' tid:3598048, ticks:103411
rax:0x68 rbx:0x3e8 rcx:0xffffffffffffffff rdx:0x0 rsi:0x1 rdi:0x0 rbp:0x7ffe334dffc0 rsp:0x7ffe334dfe08 r8:0x44c37cdba920 r9:0x0 r10:0x44c37cd818f0 r11:0x246 r12:0x3 r13:0x0 r14:0x7feae29dd000 r15:0x55b18d48ea38 rip:0x44c37cc94f29 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0xffffffffffffffff fs_base:0x6ffe4b80 gs_base:0x0
  { tid:3598048, addr:0x44c37cdc4292, length:0x5e }
  { tid:3598048, addr:0x44c37cc94f29, length:0x7 }
}
{
  real_time:539090.663338 global_time:91367, event:`SYSCALLBUF_FLUSH' tid:3598048, ticks:103433
  { syscall:'getgid', ret:0x3e8, size:0x10 }
}
{
  real_time:539090.663357 global_time:91368, event:`PATCH_SYSCALL' tid:3598048, ticks:103433
rax:0x6b rbx:0x3e8 rcx:0xffffffffffffffff rdx:0xffffffffffffffff rsi:0x44c37cc94f30 rdi:0x1 rbp:0x7ffe334dfe00 rsp:0x7ffe334dfdf8 r8:0x0 r9:0x7ffe334dffc0 r10:0x44c37cdba920 r11:0x246 r12:0x3 r13:0x0 r14:0x7feae29dd000 r15:0x55b18d48ea38 rip:0x44c37ccc2f1b eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0xffffffffffffffff fs_base:0x6ffe4b80 gs_base:0x0
  { tid:3598048, addr:0x44c37cdc42f0, length:0x5e }
  { tid:3598048, addr:0x44c37ccc2f1b, length:0x8 }
}
{
  real_time:539090.663361 global_time:91369, event:`SYSCALLBUF_RESET' tid:3598048, ticks:103433
}
=== Start rr backtrace:
rr(_ZN2rr13dump_rr_stackEv+0x5c) [0x5562a225e68c]
rr(_ZN2rr15emergency_debugEPNS_4TaskE+0x1a8) [0x5562a213df28]
rr(+0xd8951) [0x5562a2143951]
rr(+0xda495) [0x5562a2145495]
rr(_ZN2rr13RecordSession21process_syscall_entryEPNS_10RecordTaskEPNS0_9StepStateEPNS0_12RecordResultENS_13SupportedArchE+0x391) [0x5562a2173b01]
rr(_ZN2rr13RecordSession29handle_seccomp_traced_syscallEPNS_10RecordTaskEPNS0_9StepStateEPNS0_12RecordResultEPb+0x202) [0x5562a216ef52]
rr(_ZN2rr13RecordSession19handle_ptrace_eventEPPNS_10RecordTaskEPNS0_9StepStateEPNS0_12RecordResultEPb+0x635) [0x5562a216fea5]
rr(_ZN2rr13RecordSession11record_stepEv+0x296) [0x5562a2178ba6]
rr(_ZN2rr13RecordCommand3runERSt6vectorINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESaIS7_EE+0xdd2) [0x5562a2168822]
rr(main+0x166) [0x5562a20b8006]
/usr/lib64/libc.so.6(+0x3248) [0x7f8c91eef248]
/usr/lib64/libc.so.6(__libc_start_main+0x8b) [0x7f8c91eef30b]
rr(_start+0x25) [0x5562a20b96e5]
=== End rr backtrace