Skip to main content

How to Debug the Kernel

One of the major benefits of using a virtual environment is the ability to inspect and modify the state of the whole system under user control. The primary interface to these features is through the TCP-based GDB remote protocol compatible stub.

Important Notes on the Kernel Debug Stub

  • The kernel debug stub presents the multiple CPUs in the system as 'threads' of the process being debugged. For instance, on an iPhone 6, there are two threads: 1 and 2, corresponding to CPU 0 and CPU 1. (GDB protocol does not allow for a thread called 0.)
  • Only all-stop mode is supported: if one CPU stops, the others stop as well.
  • Single-stepping is supported by using AArch64 single-stepping features. In the case of vCont packets, if a CPU single-steps and other CPUs do not have actions specified, they do not perform a step.
  • Hardware breakpoints and watchpoints are supported. (Up to 4 of each at a time.) Software breakpoint packets issued to the stub are converted into hardware breakpoints.
  • Memory addresses from the debugger are passed through virtual-to-physical address translation - this is necessary to make it work. Also, only actual RAM is visible to the debugger; access to MMIO regions is ignored and returns 0.
  • Only one concurrent debug stub connection is supported per VM.
  • High latency can cause the VM to visually “freeze” or “stutter” as the underlying kernel is setting breakpoints and communicating with your GDB. If you use features like watch / awatch / rwatch with conditions, every breakpoint is sent to your local machine, the condition is calculated, and then resumed if it was meant to ignore. This is how GDB functions, but it is typically not noticeable for local kernel debugging. On the local GDB end, nothing will visually indicate that this is happening unless the condition is met, in which case you’ll receive a prompt.

Initial Setup

  1. Set up your virtual device.
  2. Connect to the VPN if you are using the cloud product.
  3. Grab the Corellium vmlinux file with symbols appropriate for your device:

Connecting to the Kernel Debug Stub

Using GDB / LLDB

To connect using GDB / LLDB, check out our article on GDB / LLDB.

To Connect to the Stub

Note: The address and port provided here are for example purposes only. You will need to use the address and port for your particular virtual device. You can find the address and port for your device at the end of the "kernel gdb" link.

GDB Command

Example

(gdb) target remote 10.11.1.1:44219
Remote debugging using 10.11.1.1:44219
warning: No executable has been specified and target does not support
determining executable automatically. Try using the "file" command.
0x00000008030e40c8 in ?? ()

To Switch CPUs (in this case, to CPU 1)

(gdb)thread 2
[Switching to thread 2 (Thread 2)]
#0 0x00000008030e40c8 in ?? ()

To Access Monitor Commands

(gdb) monitor sr ttbr1_el1=0x0000000034d4593d
CPU 1, ttbr1_el1 := 0x0000000034d4593d (before: 0x0000000000000000)

Otherwise, use regular GDB commands to control the debug stub.

To Connect to the Stub Without the Binary

Note: The address and port provided here are for example purposes only. You will need to use the address and port for your particular virtual device. You can find the address and port for your device at the end of the "kernel gdb" link.

GDB Command

Example

$ lldb
(lldb) gdb-remote 10.11.1.1:44219
Kernel UUID: B99BA98C-3AAA-30E9-B733-07845A9EC56B
Load Address: 0xfffffff007004000
WARNING: Unable to locate kernel binary on the debugger system.
Process 1 stopped
* thread #1, stop reason = signal SIGINT
frame #0: 0xfffffff0071a513c
-> 0xfffffff0071a513c: bl 0xfffffff00709fc00
0xfffffff0071a5140: mrs x8, TPIDR_EL1
0xfffffff0071a5144: ldr x8, [x8, #0x428]
0xfffffff0071a5148: str xzr, [x8, #0xf0]
Target 0: (No executable module.) stopped.
(lldb)

To Connect to the Stub with the Binary

Example

$ lldb ~/Documents/Work/kerneli61125
(lldb) gdb-remote 10.11.1.31:39535
Kernel UUID: B99BA98C-3AAA-30E9-B733-07845A9EC56B
Load Address: 0xfffffff007004000
Kernel slid 0x0 in memory.
Loaded kernel file /Users/planetbeing/Documents/Work/kerneli61125
Loading 140 kext modules warning: Can't find binary/dSYM for com.apple.kec.corecrypto (A6668145-C49A-3D84-96E6-80AE4AA9E4B4)
.warning: Can't find binary/dSYM for com.apple.kec.Libm (51AFA03E-8041-3D11-BD40-A6D1AED1C667)
.warning: Can't find binary/dSYM for com.apple.kec.pthread (75DF2E44-845A-3C15-987F-D53AC36CFD72)
.warning: Can't find binary/dSYM for com.apple.driver.usb.cdc (0B64EFC9-CF03-37AF-8C86-4A0C419BC87B)
.warning: Can't find binary/dSYM for com.apple.driver.AppleUSBDeviceMux (84B68AED-C037-3166-BA42-906FE26FA76F)
. done.
Process 1 stopped
* thread #1, stop reason = signal SIGINT
frame #0: 0xfffffff0071a513c kerneli61125`___lldb_unnamed_symbol1665$$kerneli61125 + 296
kerneli61125`___lldb_unnamed_symbol1665$$kerneli61125:
-> 0xfffffff0071a513c <+296>: bl 0xfffffff00709fc00 ; ___lldb_unnamed_symbol102$$kerneli61125
0xfffffff0071a5140 <+300>: mrs x8, TPIDR_EL1
0xfffffff0071a5144 <+304>: ldr x8, [x8, #0x428]
0xfffffff0071a5148 <+308>: str xzr, [x8, #0xf0]
Target 0: (kerneli61125) stopped.
(lldb)

The GDB stub represents CPU cores as threads.

(lldb) thread list
Process 1 stopped
* thread #1: tid = 0x0001, 0xfffffff0071a513c, stop reason = signal SIGINT
thread #2: tid = 0x0002, 0xfffffff0071a513c
(lldb) thread select 2
* thread #2
frame #0: 0xfffffff0071a513c
-> 0xfffffff0071a513c: bl 0xfffffff00709fc00
0xfffffff0071a5140: mrs x8, TPIDR_EL1
0xfffffff0071a5144: ldr x8, [x8, #0x428]
0xfffffff0071a5148: str xzr, [x8, #0xf0]
(lldb)

While LLDB is specified in the UI, users are free to use GDB as well.

$ gdb
(gdb) target remote 10.11.1.1:4000
Remote debugging using 10.11.1.1:4000
0xffff80001117094c in ?? ()
(gdb) info reg
x0 0xe0 224
x1 0x11c430 1164336
x2 0x0 0
x3 0x4000000000000000 4611686018427387904
x4 0x0 0
x5 0xffff8000dde56000 -140733765558272
x6 0x80 128
...
(gdb) x/10i $pc
=> 0xffff80001117094c: hint #0x1d
0xffff800011170950: ret
0xffff800011170954: hint #0x19
0xffff800011170958: stp x29, x30, [sp,#-16]!
0xffff80001117095c: mov x29, sp
0xffff800011170960: bl 0xffff800011170940
0xffff800011170964: ldp x29, x30, [sp],#16
0xffff800011170968: hint #0x1d
0xffff80001117096c: ret
0xffff800011170970: hint #0x19

You can use the regular LLDB commands to control the debug stub.

Your debugger will work as if it was attached to a hardware debugger (think OpenOCD).

Using IDA

The following instructions are for IDA 7.0 versions.

  1. Select Debugger | Switch debugger... from the main menu, then pick Remote GDB debugger in the dialog box.

  2. Then, again from the main menu, select Debugger then Debugger options.... Click the Set specific options button and make sure the Use stepping support checkbox is checked.

  3. Finally, select Debugger | Process options... from the main menu, enter the stub's address in the Hostname and Port fields. After this setup, which is saved in the IDA database, select Debugger | Attach to process... to attach to the running device.

To access monitor commands from IDA, locate the GDB command line bar at the bottom of the window (just above the status bar, next to a GDB button). Enter the monitor commands there, without the word "monitor" itself. For instance, instead of monitor sr, simply write sr and press Enter. The output will appear in IDA's text output window above.