Crash during local unwinding with aarch64 pointer authentication
Maintainers usually reply within 4 days
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 22/100
- Issue type
- Bug
- Clarity
- Mostly clear
- Activity status
- Stale
- Tech stack
- c
- Domain
- operating-systems
Research direction
The bug is in src/aarch64/Gtrace.c: around line 486, the saved link register is read via ACCESS_MEM_FAST into pc without stripping the pointer-authentication bits, while the DWARF path already calls aarch64_strip_pac_local. Start by reading _ULaarch64_tdep_trace and that helper, then reproduce on a PAC-enabled aarch64 machine. Done means the traced PC matches the DWARF-computed IP with no PAC bits set, and the trace no longer crashes.
Written by the indexing model from the issue text.
Description
Hello,
I'm trying to add libunwind to do local crash-reporting on an application using pointer authentication on aarch64. The support for that looks like it should have been added in https://github.com/libunwind/libunwind/pull/360, but it doesn't seem complete.
When running with UNW_DEBUG_LEVEL=99, this is the last log output before the segfault:
UNW_DEBUG_LEVEL output
``` >_ULaarch64_tdep_stash_frame: ip=0xaaaae08174ac cfa=0xfffff30295d0 type 0 cfa [where=3 val=31] cfaoff=384 ra=0xfffff3029458 fp [where=2 val=-384 @0xfffff3029450] lr [where=2 val=-376 @0xfffff3029458] sp [where=6 val=0 @0x0] >_ULaarch64_tdep_stash_frame: standard frame >_ULaarch64_step: dwarf_step()=1 >trace_init_addr: frame va aaaae0a32637 type -2 last 0 cfa sp+384 fp @ cfa-384 lr @ cfa-376 sp @ cfa+0 >_ULaarch64_tdep_trace: frame va aaaae0a32637 type -2 last 0 cfa sp+384 fp @ cfa-384 lr @ cfa-376 sp @ cfa+0 >_ULaarch64_tdep_trace: new cfa 0xfffff30295d0 pc 0x32aaaae08174ac sp 0xfffff30295d0 fp 0xfffff30295d0 >_ULaarch64_tdep_trace: depth 2 cfa 0xfffff30295d0 pc 0x32aaaae08174ab sp 0xfffff30295d0 fp 0xfffff30295d0 >trace_lookup: updating slot 1846 after 0 steps, replacing 0x0 >_ULaarch64_step: (cursor=0xfffff30284a8, ip=0x0032aaaae08174ac, cfa=0x0000fffff30295d0)) >access_mem: mem[0032aaaae08174ac] -> invalid >_ULaarch64_step: Invalid address found in the call stack: 0x32aaaae08174ac >get_rs_cache: acquiring lock >_ULaarch64_dwarf_find_proc_info: looking for IP=0x32aaaae08174ab >_ULaarch64_dwarf_callback: checking , base=0xaaaae0740000) >_ULaarch64_dwarf_callback: checking linux-vdso.so.1, base=0xffff80537000) >_ULaarch64_dwarf_callback: checking /opt/libunwind-1.7/lib/libunwind.so.8, base=0xffff804a0000) >_ULaarch64_dwarf_callback: checking /lib/aarch64-linux-gnu/libz.so.1, base=0xffff80460000) >_ULaarch64_dwarf_callback: checking /lib/aarch64-linux-gnu/libcrypt.so.1, base=0xffff80410000) >_ULaarch64_dwarf_callback: checking /lib/aarch64-linux-gnu/libm.so.6, base=0xffff80360000) >_ULaarch64_dwarf_callback: checking /lib/aarch64-linux-gnu/libgcc_s.so.1, base=0xffff80320000) >_ULaarch64_dwarf_callback: checking /lib/aarch64-linux-gnu/libc.so.6, base=0xffff80170000) >_ULaarch64_dwarf_callback: checking /lib/ld-linux-aarch64.so.1, base=0xffff804fa000) >_ULaarch64_dwarf_callback: checking /lib/aarch64-linux-gnu/liblzma.so.5, base=0xffff80130000) >put_rs_cache: unmasking signals/interrupts and releasing lock >_ULaarch64_step: dwarf_step()=-10 ```I attached a debugger and found some interesting events leading up to the crash:
at Gtrace.c:465 (permalink):
(gdb) print (void*)pc
$2 = (void *) 0xaaaaaad92637 <rb_vm_bugreport+2055>
(gdb) print (void*)((struct cursor *)c)->dwarf.ip
$4 = (void *) 0xaaaaaab774ac <rb_bug_without_die+136>
So looks like pc is the program counter in this frame, and c->dwarf.ip is the program counter in the previous frame as computed from the DWARF info. The pointer authentication seems to be correctly stripped from the computed previous IP by the call to aarch64_strip_pac_local from #360. So far, so good.
(we can verify for sure that this call has pointer authentication, because gdb says so - it says [PAC] for frame 4)
Backtrace at this point
``` (gdb) bt #0 _ULaarch64_tdep_trace (cursor=cursor@entry=0xffffffffe078, buffer=buffer@entry=0xaaaaaafdfa50 , size=size@entry=0xffffffffdc84) at aarch64/Gtrace.c:465 #1 0x0000fffff7f637d8 in unw_backtrace (buffer=buffer@entry=0xaaaaaafdfa50 , size=size@entry=1024) at mi/backtrace.c:70 #2 0x0000aaaaaad92638 in rb_print_backtrace () at vm_dump.c:785 #3 rb_vm_bugreport (ctx=ctx@entry=0x0) at vm_dump.c:1093 #4 0x0000aaaaaab774ac [PAC] in rb_bug_without_die (fmt=fmt@entry=0xaaaaaae97390 "%s", args=...) at error.c:790 #5 0x0000aaaaaaadd8b8 [PAC] in rb_bug (fmt=fmt@entry=0xaaaaaae97390 "%s") at error.c:798 #6 0x0000aaaaaab77544 [PAC] in kernel_rb_bug (self=, str=) at error.c:3029 #7 0x0000aaaaaad6936c [PAC] in vm_call_cfunc_with_frame_ (stack_bottom=, argv=, argc=, calling=, reg_cfp=, ec=) at /home/ktsanaktsidis.linux/ruby/vm_insnhelper.c:3285 #8 vm_call_cfunc_with_frame (calling=, reg_cfp=, ec=) at /home/ktsanaktsidis.linux/ruby/vm_insnhelper.c:3306 #9 vm_call_cfunc (ec=0xaaaaaafe9610, reg_cfp=0xfffff77feed0, calling=) at /home/ktsanaktsidis.linux/ruby/vm_insnhelper.c:3408 #10 0x0000aaaaaad83070 [PAC] in vm_sendish (method_explorer=, block_handler=, cd=, reg_cfp=, ec=) at /home/ktsanaktsidis.linux/ruby/vm_callinfo.h:367 #11 vm_exec_core (ec=0xaaaaaafe9610, initial=281474842483576, initial@entry=0) at /home/ktsanaktsidis.linux/ruby/insns.def:820 #12 0x0000aaaaaad76288 [PAC] in rb_vm_exec (ec=0xaaaaaafe9610, jit_enable_p=jit_enable_p@entry=true) at vm.c:2410 #13 0x0000aaaaaad8bb9c [PAC] in rb_iseq_eval_main (iseq=iseq@entry=0xfffff7b721c0) at vm.c:2669 #14 0x0000aaaaaab7cc94 in rb_ec_exec_node (ec=ec@entry=0xaaaaaafe9610, n=n@entry=0xfffff7b721c0) at eval.c:289 #15 0x0000aaaaaab8215c [PAC] in ruby_run_node (n=0xfffff7b721c0) at eval.c:330 #16 0x0000aaaaaaae46bc [PAC] in rb_main (argv=0xfffffffff9c8, argc=2) at ./main.c:39 #17 main (argc=, argv=) at ./main.c:58 ```BUT, then the subsequent code seems to re-compute the PC-in-previous-frame by looking at the saved value of LR on the stack here - https://github.com/libunwind/libunwind/blob/87a98de7fa5669ef1ea103984d8d26d0cf859df5/src/aarch64/Gtrace.c#L486. The value retrieved from the stack seems to be the version of the return address with pointer authentication:
(gdb) info line
Line 486 of "aarch64/Gtrace.c" starts at address 0xfffff7f676ac <_ULaarch64_tdep_trace+812>
and ends at 0xfffff7f676bc <_ULaarch64_tdep_trace+828>.
(gdb) list
481
482 case UNW_AARCH64_FRAME_STANDARD:
483 /* Advance standard traceable frame. */
484 cfa = (f->cfa_reg_sp ? sp : fp) + f->cfa_reg_offset;
485 if (likely(f->lr_cfa_offset != -1))
486 ACCESS_MEM_FAST(ret, c->validate, d, cfa + f->lr_cfa_offset, pc);
487 else if (lr != 0)
488 {
489 /* Use the saved link register as the new pc. */
490 pc = lr;
(gdb) print (void*)pc
$5 = (void *) 0xaaaaaad92637 <rb_vm_bugreport+2055>
(gdb) next
(gdb) print (void*)pc
$7 = (void *) 0x72aaaaaab774ac
(gdb) print (void*)((struct cursor *)c)->dwarf.ip
$8 = (void *) 0xaaaaaab774ac <rb_bug_without_die+136>
The value saved into the pc variable here is 0x72aaaaaab774ac, which looks a lot like the actual return address (0xaaaaaab774ac) except with bonus pointer authentication bits.
Attempting to continue execution from here then leads to a segfault, because:
- This incorrect-pc-with-pac gets passed into the next iteration of
unw_stepandtrace_init_addr, - And then it gets set onto
loc[UNW_AARCH64_PC]here: https://github.com/libunwind/libunwind/blob/87a98de7fa5669ef1ea103984d8d26d0cf859df5/src/aarch64/Gtrace.c#L250 - and that subsequently leads to a segfault with this stack, which attempts to dereference this address
(gdb) bt
#0 0x0000fffff7f649ac in access_mem (as=<optimized out>, addr=32275797330326700, val=0xffffffffdb48,
write=<optimized out>, arg=<optimized out>) at aarch64/Ginit.c:362
#1 0x0000fffff7f66af0 in is_plt_entry (c=0xffffffffe078) at aarch64/Gstep.c:44
#2 _ULaarch64_step (cursor=cursor@entry=0xffffffffe078) at aarch64/Gstep.c:228
#3 0x0000fffff7f6799c in trace_init_addr (sp=281474976706976, fp=281474976706976, pc=32275797330326699,
cfa=281474976706976, cursor=0xffffffffe078, f=<optimized out>) at ../include/tdep-aarch64/libunwind_i.h:161
#4 trace_lookup (sp=281474976706976, fp=281474976706976, pc=32275797330326699, cfa=281474976706976,
cache=0xfffff7fe9fe0, cursor=0xffffffffe078) at aarch64/Gtrace.c:333
#5 _ULaarch64_tdep_trace (cursor=cursor@entry=0xffffffffe078, buffer=buffer@entry=0xaaaaaafdfa50 <trace>,
size=size@entry=0xffffffffdc84) at aarch64/Gtrace.c:451
#6 0x0000fffff7f637d8 in unw_backtrace (buffer=buffer@entry=0xaaaaaafdfa50 <trace>, size=size@entry=1024)
at mi/backtrace.c:70
#7 0x0000aaaaaad92638 in rb_print_backtrace () at vm_dump.c:785
#8 rb_vm_bugreport (ctx=ctx@entry=0x0) at vm_dump.c:1093
#9 0x0000aaaaaab774ac [PAC] in rb_bug_without_die (fmt=fmt@entry=0xaaaaaae97390 "%s", args=...) at error.c:790
#10 0x0000aaaaaaadd8b8 [PAC] in rb_bug (fmt=fmt@entry=0xaaaaaae97390 "%s") at error.c:798
#11 0x0000aaaaaab77544 [PAC] in kernel_rb_bug (self=<optimized out>, str=<optimized out>) at error.c:3029
#12 0x0000aaaaaad6936c [PAC] in vm_call_cfunc_with_frame_ (stack_bottom=<optimized out>, argv=<optimized out>,
argc=<optimized out>, calling=<optimized out>, reg_cfp=<optimized out>, ec=<optimized out>)
at /home/ktsanaktsidis.linux/ruby/vm_insnhelper.c:3285
#13 vm_call_cfunc_with_frame (calling=<optimized out>, reg_cfp=<optimized out>, ec=<optimized out>)
at /home/ktsanaktsidis.linux/ruby/vm_insnhelper.c:3306
#14 vm_call_cfunc (ec=0xaaaaaafe9610, reg_cfp=0xfffff77feed0, calling=<optimized out>)
at /home/ktsanaktsidis.linux/ruby/vm_insnhelper.c:3408
#15 0x0000aaaaaad83070 [PAC] in vm_sendish (method_explorer=<optimized out>, block_handler=<optimized out>,
cd=<optimized out>, reg_cfp=<optimized out>, ec=<optimized out>)
at /home/ktsanaktsidis.linux/ruby/vm_callinfo.h:367
#16 vm_exec_core (ec=0xaaaaaafe9610, initial=32275797330326700, initial@entry=0)
at /home/ktsanaktsidis.linux/ruby/insns.def:820
#17 0x0000aaaaaad76288 [PAC] in rb_vm_exec (ec=0xaaaaaafe9610, jit_enable_p=jit_enable_p@entry=true) at vm.c:2410
#18 0x0000aaaaaad8bb9c [PAC] in rb_iseq_eval_main (iseq=iseq@entry=0xfffff7b721c0) at vm.c:2669
#19 0x0000aaaaaab7cc94 in rb_ec_exec_node (ec=ec@entry=0xaaaaaafe9610, n=n@entry=0xfffff7b721c0) at eval.c:289
#20 0x0000aaaaaab8215c [PAC] in ruby_run_node (n=0xfffff7b721c0) at eval.c:330
#21 0x0000aaaaaaae46bc [PAC] in rb_main (argv=0xfffffffff9c8, argc=2) at ./main.c:39
#22 main (argc=<optimized out>, argv=<optimized out>) at ./main.c:58
(gdb) print (void*)32275797330326700
$9 = (void *) 0x72aaaaaab774ac
I don't really understand libunwind well enough to know what the fix is though. It seems to me that tdep_trace should just read the previous frame PC from the dwarf result directly, rather than looking at the LR? I suppose this unifies the frame-pointer-chasing and dwarf unwinding code paths though doesn't it. At the very least this calculation of PC-from-saved-LR needs to also strip the pointer auth bits.
Any thoughts? Thanks for reading this long rant!
- Dominant language
- C
- Stars
- 1.3k
- Forks
- 353
- Avg merge
- 3d 19h
- Merged PRs (30d)
- 11
Getting set up
This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from libunwind/libunwind
-
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
Maintainers usually reply within 4 days
-
build fails on muslOpenbug
Difficulty 3/5 1-2 days Newbie friendliness 55/100
Maintainers usually reply within 4 days
-
v1.8.3 not taggedOpenbug
Difficulty 1/5 Under an hour Newbie friendliness 55/100
libunwind/libunwind#1076 · 1 comment ·
Maintainers usually reply within 4 days
-
Difficulty 5/5 Over a week Newbie friendliness 35/100
Maintainers usually reply within 4 days
-
New release?Openenhancement
Difficulty 5/5 Over a week Newbie friendliness 12/100
libunwind/libunwind#1030 · 7 comments ·
Maintainers usually reply within 4 days
All issues in libunwind/libunwind
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
zephyrproject-rtos/zephyr#123881 ·
Maintainers usually reply within 2 days
-
Difficulty 1/5 Under an hour Newbie friendliness 65/100
ly05010419/MacNativeGames#1252 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 70/100
Maintainers usually reply within 2 days
-
Component - Build
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
Maintainers usually reply within 1 day
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
HarbourMasters/Lighthouse#655 ·
Maintainers usually reply within 1 day