I am trying to learn to write Mac OS X drivers using the IO Kit, and the information I have found looks like getting a system set up for kernel debugging is a challenge. 1) it appears that there is no graphical debugger like WinDBG on windows - only gdb. PR symtab/26003 (infinite loop loading symbols from separate debug objfile) PR build/26029 (GDB build failure on SPARC) February 8th, 2020: GDB 9.1 Released! The latest version of GDB, version 9.1, is available for download. This version of GDB includes the following changes and enhancements: Building GDB and GDBserver now requires GNU make. Gdb free download - WinGDB, VisualGDB, Affinic Debugger, and many more programs. Enter to Search. Debug code in a graphical user interface front-end for any GDB debugger. Download Links.: Lite version is completely FREE to download, use and redistribute. It also works as Professional version for the first 30 days. Download Affinic Debugger for Linux x86/x86-64 – Lite Version. Download Affinic Debugger for Windows – Lite Version. Download Affinic Debugger for Mac OS X.
As security researchers, we often find ourselves needing to look deep into various kernels to fully understand our target and accomplish our goals. Doing so on the Windows platform is no mystery, as there have been countless well-written posts about kernel debugging setups. For macOS, however, the situation is slightly different.
There are many great posts describing how to set up kernel debugging between two machines, but all of them suggest that SIP (System Integrity Protection) should be disabled for kernel debugging. This creates a problem if we want to investigate the inner workings of macOS’s security mechanisms, since turning off SIP will also turn off most of the foundational security features of the operating system.
This blog post will describe a couple of setups that allow you to have SIP enabled while debugging.
Our setup is as follows:
- HOST: macOS Catalina 10.15.4 with supplemental update
- GUEST: macOS Catalina 10.15.4 with supplemental update
- VMware Fusion 11.5.3
- Debugger: LLDB
The Good
We will start by debugging the original release version of the kernel, which is included by default on macOS. This is by far the easiest method we will see.
Our first step is to download the Kernel Debug Kit (KDK) from Apple’s Developer Downloads. Before we do that however, we need to determine the build version we are interested in. This can be accomplished with the following command on the guest VM:
% sw_vers
ProductName: Mac OS X
ProductVersion: 10.15.4
BuildVersion: 19E287
ProductName: Mac OS X
ProductVersion: 10.15.4
BuildVersion: 19E287
Listing 1 – Retrieving the kernel build version info
Once we know the BuildVersion number, we can download the corresponding KDK and install it on our host. It will be installed under /Library/Developer/KDKs/.
% ls -l /Library/Developer/KDKs/KDK_10.15.4_19E287.kdk/System/Library/Kernels/
total 212112
-rwxr-xr-x 1 root wheel 16030560 Mar 5 07:38 kernel
drwxr-xr-x 3 root wheel 96 Mar 5 07:38 kernel.dSYM
-rwxr-xr-x 1 root wheel 23795528 Mar 5 07:27 kernel.debug
drwxr-xr-x 3 root wheel 96 Mar 5 07:27 kernel.debug.dSYM
-rwxr-xr-x 1 root wheel 19329072 Mar 5 07:39 kernel.development
drwxr-xr-x 3 root wheel 96 Mar 5 07:39 kernel.development.dSYM
-rwxr-xr-x 1 root wheel 49436536 Mar 5 07:30 kernel.kasan
total 212112
-rwxr-xr-x 1 root wheel 16030560 Mar 5 07:38 kernel
drwxr-xr-x 3 root wheel 96 Mar 5 07:38 kernel.dSYM
-rwxr-xr-x 1 root wheel 23795528 Mar 5 07:27 kernel.debug
drwxr-xr-x 3 root wheel 96 Mar 5 07:27 kernel.debug.dSYM
-rwxr-xr-x 1 root wheel 19329072 Mar 5 07:39 kernel.development
drwxr-xr-x 3 root wheel 96 Mar 5 07:39 kernel.development.dSYM
-rwxr-xr-x 1 root wheel 49436536 Mar 5 07:30 kernel.kasan
Listing 2 – KDK location
The next step is to enable remote debugging on our guest VM. Fortunately, VMware Fusion has a feature called gdb stub, which sets up a GDB server and allows a debugger to debug any VM (including Windows) using the GDB remote protocol. Since LLDB supports the GDB protocol, we can use this approach.
To enable the GDB stub for our guest VM, we need to add the following line to our virtual machine vmx configuration file:
Listing 3 – Enabling the VMware gdb stub
Next, we boot up the VM, launch lldb on our host, and issue the following commands to specify our kernel and also enable loading of symbol files:
(lldb) target create /Library/Developer/KDKs/KDK_10.15.4_19E287.kdk/System/Library/Kernels/kernel
(lldb) settings set target.load-script-from-symbol-file true
(lldb) gdb-remote 8864
(lldb) settings set target.load-script-from-symbol-file true
(lldb) gdb-remote 8864
Listing 4 – Starting LLBD with appropriate symbols
The first command will tell lldb where to find the kernel symbols. This command is not strictly necessary, as lldb will search the /Library/Developer/KDKs path and any other which is indexed by Spotlight, but it can be still a good practice in case the search fails.
The second command will tell lldb to load any scripts found inside the symbol (dSYM) directories. This is extremely useful, as these scripts typically extend the functionality of lldb. In the case of the kernel, we get about 400 new commands available to us.
The last command tells lldb which port the remote server is listening on. If we don’t specify the IP address or hostname, like here, it will connect to the localhost. By default, VMware listens on port 8864 on the localhost, so this is where we connect to.
If everything has gone well, we should be breaking into our VM and can start debugging. We can also break again at any point using the “CTRL+C” shortcut from the debugger.
(lldb) gdb-remote 8864
Kernel UUID: AB0AA7EE-3D03-3C21-91AD-5719D79D7AF6
Load Address: 0xffffff8002600000
Kernel slid 0x2400000 in memory.
(...)
Process 1 stopped
* thread #3, name = '0xffffff8009854a40', queue = 'cpu-0', stop reason = signal SIGTRAP
frame #0: 0xffffff800284dede kernel`machine_idle at pmCPU.c:181:3 [opt]
Target 0: (kernel) stopped.
(lldb) image list
[ 0] AB0AA7EE-3D03-3C21-91AD-5719D79D7AF6 0xffffff8002600000 /Library/Developer/KDKs/KDK_10.15.4_19E287.kdk/System/Library/Kernels/kernel
/Library/Developer/KDKs/KDK_10.15.4_19E287.kdk/System/Library/Kernels/kernel.dSYM/Contents/Resources/DWARF/kernel
(...)
Kernel UUID: AB0AA7EE-3D03-3C21-91AD-5719D79D7AF6
Load Address: 0xffffff8002600000
Kernel slid 0x2400000 in memory.
(...)
Process 1 stopped
* thread #3, name = '0xffffff8009854a40', queue = 'cpu-0', stop reason = signal SIGTRAP
frame #0: 0xffffff800284dede kernel`machine_idle at pmCPU.c:181:3 [opt]
Target 0: (kernel) stopped.
(lldb) image list
[ 0] AB0AA7EE-3D03-3C21-91AD-5719D79D7AF6 0xffffff8002600000 /Library/Developer/KDKs/KDK_10.15.4_19E287.kdk/System/Library/Kernels/kernel
/Library/Developer/KDKs/KDK_10.15.4_19E287.kdk/System/Library/Kernels/kernel.dSYM/Contents/Resources/DWARF/kernel
(...)
Listing 5 – LLDB macOS kernel debugging with symbols
One thing to note is that if our VM and Host are running the same kernel, we could also use the basic kernel binary (/System/Library/Kernels/kernel) as a target. However, we would not have any access to the symbols.
(lldb) target create /System/Library/Kernels/kernel
Current executable set to '/System/Library/Kernels/kernel' (x86_64).
(lldb) gdb-remote 8864
Process 1 stopped
* thread #1, stop reason = signal SIGTRAP
frame #0: 0xffffff8003b980f6
-> 0xffffff8003b980f6: rep stosb byte ptr es:[rdi], al
0xffffff8003b980f8: ret
0xffffff8003b980f9: add byte ptr [rax], al
0xffffff8003b980fb: add byte ptr [rax], al
Target 0: (kernel) stopped.
(lldb) image list
[ 0] AB0AA7EE-3D03-3C21-91AD-5719D79D7AF6 0xffffff8000200000 /System/Library/Kernels/kernel
(lldb) memory read 0xffffff8000200000
0xffffff8000200000: cf fa ed fe 07 00 00 01 03 00 00 00 02 00 00 00 ????............
0xffffff8000200010: 12 00 00 00 d0 0f 00 00 01 00 20 00 00 00 00 00 ....?..... .....
(lldb) continue
Current executable set to '/System/Library/Kernels/kernel' (x86_64).
(lldb) gdb-remote 8864
Process 1 stopped
* thread #1, stop reason = signal SIGTRAP
frame #0: 0xffffff8003b980f6
-> 0xffffff8003b980f6: rep stosb byte ptr es:[rdi], al
0xffffff8003b980f8: ret
0xffffff8003b980f9: add byte ptr [rax], al
0xffffff8003b980fb: add byte ptr [rax], al
Target 0: (kernel) stopped.
(lldb) image list
[ 0] AB0AA7EE-3D03-3C21-91AD-5719D79D7AF6 0xffffff8000200000 /System/Library/Kernels/kernel
(lldb) memory read 0xffffff8000200000
0xffffff8000200000: cf fa ed fe 07 00 00 01 03 00 00 00 02 00 00 00 ????............
0xffffff8000200010: 12 00 00 00 d0 0f 00 00 01 00 20 00 00 00 00 00 ....?..... .....
(lldb) continue
Listing 6 – LLDB macOS kernel debuging without symbols
Gdb Online Debugger
Finally, it is important to note that VMware uses hardware breakpoints by default, which limits us to four total. However, this limitation can be overcome by setting the hideBreakpoints setting in the vmx configuration file to FALSE, as shown below:
Listing 6 – Disabling VMWare use of hardware breakpoints
The Bad
In the previous section, we discussed how to set up macOS kernel debugging with the release version of the kernel. Nevertheless, it is important to mention that Apple also releases debug or development kernels as well. According to Apple, they are compiled with “additional assertions and error checking” and these are the ones that can stop and wait for a debugger to connect after initial startup.
For this setup, these kernels are not strictly needed as we will still use the previous feature of VMware. More specifically, the macOS kernel will not be responsible for the actual debugging. However, in case we do need these kernels for a special use case, this is how we can set this up.
Before we do anything else on our guest VM, we first need to temporarily disable SIP. The reason for this because the /System/ path is write-protected by SIP. We can do that by booting into recovery mode (CMD+R after turning on the VM), running the csrutil disable command, and rebooting.
Since the release of macOS Catalina, the / path is mounted as read-only as an additional protection beyond SIP. Therefore, we will need to make it writable so that we copy the debug kernel to the correct location.
Once rebooted, we mount the root directory as writable:
Listing 7 – Mounting the root directory as writable
And copy the kernel of our choice (debug in this case):
Gdb Debugger Tutorial
sudo cp /Library/Developer/KDKs/KDK_10.15.4_19E287.kdk/System/Library/Kernels/kernel.debug /System/Library/Kernels/
Listing 8 – Copying the debug kernel to the appropriate location
Then we need to invalidate the kernel cache. This is required because macOS doesn’t run the kernel binary directly, but rather as a prelinked kernel, which is built from the kernel and the kernel extensions. Normally, the prelinking happens when we install a new kernel extension or kernel, but not in this case. Here, we simply copy in the development or debug version of the kernel.
sudo kextcache -invalidate /
sudo kextcache -invalidate /Volumes/Macintosh HD
sudo kextcache -invalidate /Volumes/Macintosh HD
Listing 9 – Invalidating the kernel cache
Finally, we will need to set the NVRAM boot arguments to boot into the debug kernel instead of the regular one, and boot into recovery mode to turn SIP back on using the csrutil enable command.
Listing 10 – Setting the boot kernel
We can now run the same commands on our host as before, this time specifying the debug kernel.
(lldb) target create /Library/Developer/KDKs/KDK_10.15.4_19E287.kdk/System/Library/Kernels/kernel.debug
(lldb) settings set target.load-script-from-symbol-file true
(lldb) gdb-remote 8864
(lldb) settings set target.load-script-from-symbol-file true
(lldb) gdb-remote 8864
Listing 11 – Commands to start debug kernel debugging
At this point, we are able to perform macOS kernel debugging on a debug rather than a release version of the kernel.
Process 1 stopped
* thread #2, name = '0xffffff80158f0e28', queue = 'cpu-0', stop reason = signal SIGTRAP
frame #0: 0xffffff80052ee796 kernel.debug`machine_idle at pmCPU.c:181:3
Target 0: (kernel.debug) stopped.
(lldb) image list
[ 0] 16545FA7-C11F-3D9E-88FA-8DDB13E1A439 0xffffff8005000000 /Library/Developer/KDKs/KDK_10.15.4_19E287.kdk/System/Library/Kernels/kernel.debug
/Library/Developer/KDKs/KDK_10.15.4_19E287.kdk/System/Library/Kernels/kernel.debug.dSYM/Contents/Resources/DWARF/kernel.debug
[ 1] A6D59354-C9A1-3C61-87A7-C04DD74421B1 0xffffff7f8609f000 //System/Library/Extensions/corecrypto.kext/Contents/MacOS/corecrypto
(...)
* thread #2, name = '0xffffff80158f0e28', queue = 'cpu-0', stop reason = signal SIGTRAP
frame #0: 0xffffff80052ee796 kernel.debug`machine_idle at pmCPU.c:181:3
Target 0: (kernel.debug) stopped.
(lldb) image list
[ 0] 16545FA7-C11F-3D9E-88FA-8DDB13E1A439 0xffffff8005000000 /Library/Developer/KDKs/KDK_10.15.4_19E287.kdk/System/Library/Kernels/kernel.debug
/Library/Developer/KDKs/KDK_10.15.4_19E287.kdk/System/Library/Kernels/kernel.debug.dSYM/Contents/Resources/DWARF/kernel.debug
[ 1] A6D59354-C9A1-3C61-87A7-C04DD74421B1 0xffffff7f8609f000 //System/Library/Extensions/corecrypto.kext/Contents/MacOS/corecrypto
(...)
Listing 12 – Debugging the macOS debug kernel
The Ugly
For completeness, we’ll briefly discuss how to set up kernel debugging using the macOS kernel instead of VMware gdb stub. This is the most commonly covered method elsewhere. We will also show that we can enable SIP despite the common misbelief that it has to be turned off.
The steps are essentially the same as before with a small difference. In this case, the NVRAM variables in the debugger box should be set as shown below:
Listing 13 – Setting the NVRAM variables for kernel debugging
In essence, these settings indicate that the debuggee can perform network-level debugging and break on interrupts. The interrupt is tricky, however, as we need to press
CMD + OPTION + CONTROL + SHIFT + ESCAPE
to break into the debugger. Furthermore, this has to be done on the target VM we are debugging. While others have had success in causing interrupts using this method, we were not as lucky. Instead, we made a keyboard shortcut in VMware Key Mappings using CMD + B
for this combination.“Windows” and “Alt” buttons as the target key mappings, however they translate into “Command” and “Option” respectively in macOS. VMware doesn’t provide an option to specify the macOS version of the keys.
Figure 1: VMware Key Mappings configuration
Finally, on the host machine we will use the kdp-remote command with our debugee IP address, instead of gdb-remote. Please note that before we issue this command, we will need to break into debugger on the debuggee.
Listing 14 – Using kdp-remote
Conclusion
In this post, we demonstrated three different ways of debugging the macOS kernel while determining that permanent disabling of SIP is not necessary. Due to the ease of setup, our preferred method is the first one we described, but others are available if the circumstances require them.
Additional resources:
- Presentation: SyScan360 – Stefan Esser – OS X El Capitan sinking the SH/IP
About the Author
Csaba Fitzl has worked for 6 years as a network engineer and 8 years as a blue/red teamer in a large enterprise focusing on malware analysis, threat hunting, exploitation, and defense evasion. Currently, he is focusing on macOS research and working at OffSec as a content developer. He gives talks and workshops at various international IT security conferences, including Hacktivity, hack.lu, Troopers, SecurityFest, DEFCON, and Objective By The Sea. @theevilbit
GDB: The GNU Project Debugger
What is GDB?
GDB, the GNU Project debugger, allows you to see what is going on`inside' another program while it executes -- or what another programwas doing at the moment it crashed.
GDB can do four main kinds of things (plus other things in supportof these) to help you catch bugs in the act:
- Start your program, specifying anything that might affect its behavior.
- Make your program stop on specified conditions.
- Examine what has happened, when your program has stopped.
- Change things in your program, so you can experiment withcorrecting the effects of one bug and go on to learn about another.
What Languages does GDB Support?
GDB supports the following languages (in alphabetical order):- Ada
- Assembly
- C
- C++
- D
- Fortran
- Go
- Objective-C
- OpenCL
- Modula-2
- Pascal
- Rust
GDB version 9.2
Version 9.2 of GDB, the GNUDebugger, is now available for download. See the ANNOUNCEMENT for detailsincluding changes in this release.An errata list (PROBLEMS) and documentationare also available.
News
The GDB 10 branch (gdb-10-branch) has been created.To check out a copy of the branch use:
The latest version of GDB, version 9.2, is available for download.
This is a minor corrective release over GDB 9.1, fixing the followingissues:
- PR tui/25586 (Resizing the source/disassembly or command window produces corrupted display)
- PR gdb/25650 (GDB can't 'printf' a convenience variable holding an inferior address)
- PR build/25981 (Use of short i386 register names breaks compilation on recent Solaris 11.4)
- PR symtab/26003 (infinite loop loading symbols from separate debug objfile)
- PR build/26029 (GDB build failure on SPARC)
The latest version of GDB, version 9.1, is available for download.
This version of GDB includes the following changes and enhancements:
- Building GDB and GDBserver now requires GNU make >= 3.82.
- If you choose to build GDB without using the GNU readline version bundled with the GDB sources, building GDB new requires GNU readline >= 7.0.
- Removed targets and native configurations:
- GDB no longer supports debugging the Cell Broadband Engine;
- GDB no longer supports Solaris 10.
- New TI PRU Simulator (pru-*-elf).
- Python Enhancements:
- GDB can now be compiled with Python 3 on Windows;
- Various Python API enhancements;
- Usability enhancements:
- [experimental] Multithreaded symbol loading for higher performance (turned off by default, use 'maint set worker-threads unlimited' to turn this feature on);
- Command names can now use the '.' character;
- GDB can now place breakpoints on nested functions and subroutines in Fortran;
- GDB now shows the Ada task names at more places, e.g. in task switching messages.
- Styling enhancements to various commands to improve readability.
- GDB now has a standard infrastructure to support dash-style command options ('-OPT'). One benefit is that commands that use it can easily support completion of command line arguments. Try 'CMD -[TAB]' or 'help CMD' to find options supported by a command. Over time, we intend to migrate most commands to this infrastructure.
- Enhancements to existing commands:
- 'printf' and 'eval' can now print C-style and Ada-style strings without calling functions in the program;
- 'info sources' has been enhance to allow only printing files whose name match a REGEXP;
- New value 'presence' for the 'set print frame-arguments' setting, to only indicate the presence of arguments with '...' instead of printing the argument names and values;
- The 'focus', 'winheight', '+', '-', '>', '<' TUI commands are now case sensitive;
- New options support for the following commands that allow overriding a number of relevant global settings (as set by e.g. 'set print [...]' commands): 'print', 'compile print', 'backtrace', 'frame apply', 'tfaas', 'faas';
- 'info types' support for '-q' to disable printing of some header information;
- In settings, 'unlimited' can now be abbreviated with 'u'.
- New commands:
- 'define-prefix' to define user-defined prefix commands;
- '|' or 'pipe' to execute a command and send its output to a shell command.
- 'with' to run a given command with a setting temporarily changed to a given value;
- 'set may-call-functions' to control whether subprogram can be called from GDB;
- 'set print finish [on|off]' to control whether the returned value should be printed when using the 'finish' command;
- 'set print max-depth' to simplify the printing of deeply nested structures;
- 'set print raw-values [on|off]' to turn on and off pretty printers;
- 'set logging debugredirect [on|off]' to control whether to redirect debug output to the log file;
- Various new 'set style' commands;
- 'set print frame-info [...]' to control what information to print when printing a frame.
- 'set tui compact-source' to enable the 'compact' mode for the TUI source window;
- 'info modules [...]' to query information about Fortran modules;
- The 'set/show print raw-frame-arguments' commands replace the 'set/show print raw frame-arguments' (now with a dash instead of a space). The latter is now deprecated and may be removed in a future release.
- New GDB/MI commands
- '-complete' to list possible completions;
- '-catch-throw', '-catch-rethrow', and '-catch-catch', the GDB/MI equivalent of the 'catch throw', 'catch rethrow', and 'catch catch' commands (respectively);
- '-symbol-info-functions', '-symbol-info-types', and '-symbol-info-variables', the GDB/MI equivalent of the 'info functions', 'info types', and 'info variables' commands (respectively);
- '-symbol-info-modules', '-symbol-info-module-functions', and '-symbol-info-module-variables', the GDB/MI equivalent of 'info modules', 'info module functions' and 'info module variables'.
- Other MI changes
- The default version of the MI interpreter is now 3 (-i=mi3);
- The output of information about multi-location breakpoints (which is syntactically incorrect in MI 2) has changed in MI 3;
- Backtraces and frames include a new optional field 'addr_flags'.
- Several new builtin convenience variables
- $_gdb_major and $_gdb_minor;
- $_gdb_setting, $_gdb_setting_str, $_gdb_maint_setting and
- $_gdb_maint_setting_str
- $_cimag and $_creal
- $_shell_exitcode and $_shell_exitsignal
- Miscellaneous enhancements:
- Support for a new configure option '--with-system-gdbinit-dir', where system gdbinit files are to be loaded from at startup;
- 'thread-exited' event is now available in the annotations interface;
- The TUI SingleKey keymap is now named 'SingleKey' (requires GNU readline >= 8.0).
The GDB maintainers are looking for contributors interestedin reversible debugging.
Late breaking information, such as recently added features, can befound in the NEWS file in the gdb source tree. Old announcements are in thenews archive.Please send FSF & GNU inquiries & questions to [email protected]. There are also other ways tocontact the FSF.
This page is maintained by the GDBdevelopers.
Copyright Free Software Foundation, Inc., 51 Franklin St - FifthFloor, Boston, MA 02110-1301 USA.
Verbatim copying and distribution of this entire article ispermitted in any medium, provided this notice is preserved.
Last modified 2020-09-12.