Skip to main content

How to Debug the Kernel

One of the major benefits of using a virtual environment is the ability to inspect and modify the state of the whole system under user control. The primary interface to these features is through the TCP-based GDB remote protocol compatible stub.

Important Notes on the Kernel Debug Stub​

  • The kernel debug stub presents the multiple CPUs in the system as 'threads' of the process being debugged. For instance, on an iPhone 6, there are two threads: 1 and 2, corresponding to CPU 0 and CPU 1. (GDB protocol does not allow for a thread called 0.)
  • Only all-stop mode is supported: if one CPU stops, the others stop as well.
  • Single-stepping is supported by using AArch64 single-stepping features. In case of vCont packets, if a CPU single-steps and other CPUs do not have actions specified, they do not perform a step.
  • Hardware breakpoints and watchpoints are supported. (Up to 4 of each at a time.) Software breakpoint packets issued to the stub are converted into hardware breakpoints.
  • Memory addresses from debugger are passed through virtual-to-physical address translation - this is necessary to make it work. Also, only actual RAM is visible to the debugger; access to MMIO regions is ignored / returns 0.
  • Only one concurrent debug stub connection is supported per VM.
  • High latency can cause the VM to visually “freeze” or “stutter” as the underlying kernel is breakpointing and communicating with your GDB. If you use features like watch / awatch / rwatch with conditions, every breakpoint is sent to your local machine, the condition is calculated, and then resumed if it was meant to ignore. This is how GDB functions, but it is typically not noticeable for local kernel debugging. On the local GDB end, nothing will visually indicate that this is happening unless the condition is met, in which case you’ll receive a prompt.

Initial Setup​

  1. Set up your virtual device.
  2. Connect to the VPN if you are using the cloud product.
  3. Grab the Corellium vmlinux file with symbols appropriate for your device:

Connecting to the Kernel Debug Stub​

Using GDB​

You will need an AArch64 compatible GDB. The versions that ship with Ubuntu 16.04 on AArch64, as well as the Linaro public releases since 7.1.1 (x86_64 Linux) have been tested to work. The AArch64 debug protocol was somewhat in flux until 2015, so very old GDB versions may not interoperate correctly.

To Connect to the Stub​

Note: The address and port provided here are for example purposes only. You will need to use the address and port for your particular virtual device. You can find the address and port for your device at the end of the "kernel gdb" link, located at the bottom of the virtual device page.

stub command

Example

(gdb) target remote 10.11.1.1:44219
Remote debugging using 10.11.1.1:44219
warning: No executable has been specified and target does not support
determining executable automatically. Try using the "file" command.
0x00000008030e40c8 in ?? ()

To Switch CPUs (in this case, to CPU 1)

(gdb)thread 2
[Switching to thread 2 (Thread 2)]
#0 0x00000008030e40c8 in ?? ()

To Access Monitor Commands

(gdb) monitor sr ttbr1_el1=0x0000000034d4593d
CPU 1, ttbr1_el1 := 0x0000000034d4593d (before: 0x0000000000000000)

Otherwise, use regular GDB commands to control the debug stub.

Using IDA​

The following instructions are for IDA 7.0 versions.

  1. Select Debugger | Switch debugger... from main menu, then pick Remote GDB debugger in the dialog box.

  2. Then, again from main menu, select Debugger | Debugger options.... Click the Set specific options button and make sure the Use stepping support checkbox is checked.

  3. Finally, select Debugger | Process options... from main menu, enter the stub's address in the Hostname and Port fields. After this setup, which is saved in the IDA database, select Debugger | Attach to process... to attach to the running device.

To access monitor commands from IDA, locate the GDB command line bar at the bottom of the window (just above the status bar, next to a GDB button). Enter the monitor commands there, without the word "monitor" itself. For instance, instead of monitor sr, simply write sr and press Enter. The output will appear in IDA's text output window above.

Using LLDB​

The platform supports the version of LLDB that ships with Xcode on macOS. Debugging with LLDB is beneficial for virtual devices running iOS because LLDB already has built-in knowledge of the Mach-O format and the fact that it is dealing with an XNU kernel. It is possible to use the debugger with or without the Mach-O kernel binary. Using the binary slows down load time, but gives you access to the symbols that are in the binary.

To Connect to the Stub Without the Binary​

Note: The address and port provided here are for example purposes only. You will need to use the address and port for your particular virtual device. You can find the address and port for your device at the end of the "kernel gdb" link, located at the bottom of the virtual device page. Without binary

Example

$ lldb
(lldb) gdb-remote 10.11.1.1:44219
Kernel UUID: B99BA98C-3AAA-30E9-B733-07845A9EC56B
Load Address: 0xfffffff007004000
WARNING: Unable to locate kernel binary on the debugger system.
Process 1 stopped
* thread #1, stop reason = signal SIGINT
    frame #0: 0xfffffff0071a513c
->  0xfffffff0071a513c: bl 0xfffffff00709fc00
    0xfffffff0071a5140: mrs x8, TPIDR_EL1
    0xfffffff0071a5144: ldr x8, [x8, #0x428]
    0xfffffff0071a5148: str xzr, [x8, #0xf0]
Target 0: (No executable module.) stopped.
(lldb)

To Connect to the Stub with the Binary​

Example

$ lldb ~/Documents/Work/kerneli61125
(lldb) gdb-remote 10.11.1.31:39535
Kernel UUID: B99BA98C-3AAA-30E9-B733-07845A9EC56B
Load Address: 0xfffffff007004000
Kernel slid 0x0 in memory.
Loaded kernel file /Users/planetbeing/Documents/Work/kerneli61125
Loading 140 kext modules warning: Can't find binary/dSYM for com.apple.kec.corecrypto (A6668145-C49A-3D84-96E6-80AE4AA9E4B4)
.warning: Can't find binary/dSYM for com.apple.kec.Libm (51AFA03E-8041-3D11-BD40-A6D1AED1C667)
.warning: Can't find binary/dSYM for com.apple.kec.pthread (75DF2E44-845A-3C15-987F-D53AC36CFD72)
.warning: Can't find binary/dSYM for com.apple.driver.usb.cdc (0B64EFC9-CF03-37AF-8C86-4A0C419BC87B)
.warning: Can't find binary/dSYM for com.apple.driver.AppleUSBDeviceMux (84B68AED-C037-3166-BA42-906FE26FA76F)
. done.
Process 1 stopped
* thread #1, stop reason = signal SIGINT
    frame #0: 0xfffffff0071a513c kerneli61125`___lldb_unnamed_symbol1665$$kerneli61125 + 296
kerneli61125`___lldb_unnamed_symbol1665$$kerneli61125:
-> 0xfffffff0071a513c <+296>: bl 0xfffffff00709fc00 ; ___lldb_unnamed_symbol102$$kerneli61125
    0xfffffff0071a5140 <+300>: mrs x8, TPIDR_EL1
    0xfffffff0071a5144 <+304>: ldr x8, [x8, #0x428]
    0xfffffff0071a5148 <+308>: str xzr, [x8, #0xf0]
Target 0: (kerneli61125) stopped.
(lldb)

The GDB stub represents CPU cores as threads.

(lldb) thread list
Process 1 stopped
* thread #1: tid = 0x0001, 0xfffffff0071a513c, stop reason = signal SIGINT
  thread #2: tid = 0x0002, 0xfffffff0071a513c
(lldb) thread select 2
* thread #2
    frame #0: 0xfffffff0071a513c
->  0xfffffff0071a513c: bl 0xfffffff00709fc00
    0xfffffff0071a5140: mrs x8, TPIDR_EL1
    0xfffffff0071a5144: ldr x8, [x8, #0x428]
    0xfffffff0071a5148: str xzr, [x8, #0xf0]
(lldb)

While LLDB is specified in the UI, users are free to use GDB as well.

$ gdb
(gdb) target remote 10.11.1.1:4000
Remote debugging using 10.11.1.1:4000
0xffff80001117094c in ?? ()
(gdb) info reg
x0 0xe0 224
x1 0x11c430 1164336
x2 0x0 0
x3 0x4000000000000000 4611686018427387904
x4 0x0 0
x5 0xffff8000dde56000 -140733765558272
x6 0x80 128
...
(gdb) x/10i $pc
=> 0xffff80001117094c: hint #0x1d
   0xffff800011170950: ret
   0xffff800011170954: hint #0x19
   0xffff800011170958: stp x29, x30, [sp,#-16]!
   0xffff80001117095c: mov x29, sp
   0xffff800011170960: bl 0xffff800011170940
   0xffff800011170964: ldp x29, x30, [sp],#16
   0xffff800011170968: hint #0x1d
   0xffff80001117096c: ret
   0xffff800011170970: hint #0x19

You can use the regular LLDB commands to control the debug stub.

Your debugger will work as if it was attached to a hardware debugger (think OpenOCD).