Device Drivers, Part 10: Kernel-Space Debuggers in Linux

Shweta, back from hospital, was relaxing in the library, reading various books. Ever since she learned of the ioctl way of debugging, she was impatient to find out more about debugging in kernel-space. She was curious about how and where to run the kernel-space debugger, if there was any. This was in contrast with application/user-space debugging, where we have the OS running underneath, and a shell or a GUI over it to run the debugger (like gdb, and the data display debugger, ddd). Then she came across this interesting kernel-space debugging mechanism using kgdb, provided as part of the kernel itself, since kernel 2.6.26.

The debugger challenge in kernel-space

As we need some interface to be up to run a debugger to debug anything, a kernel debugger could be visualised in two possible ways:

• Put the debugger into the kernel itself, accessible via the usual console. For example, in the case of kdb, which was not official until kernel 2.6.35, one had to download source code (two sets of patches — one architecture-dependent, one architecture-independent) from this FTP address and then patch these into the kernel source. However, since kernel 2.6.35, the majority of it is in the officially released kernel source. In either case, kdb support needs to be enabled in kernel source, with the kernel compiled, installed and booted with. The boot screen itself would give the kdb debugging interface.
• Put a minimal debugging server into the kernel; a client would connect to it from a remote host or local user-space over some interface (say serial or Ethernet). This is kgdb, the kernel’s gdb server, to be used with gdb as its client. Since kernel 2.6.26, its serial interface is part of the official kernel release. However, if you’re interested in a network interface, you still need to patch with one of the releases from the kgdb project page. In either case, you need to enable kgdb support in the kernel, recompile, install and boot the new kernel.

Please note that in both the above cases, the complete kernel source for the kernel to be debugged is needed, unlike for building modules, where just headers are sufficient. Here is how to play around with kgdb over the serial interface.

Setting up the Linux kernel with kgdb

Here are the prerequisites: Either the kernel source package for the running kernel should be installed on your system, or a corresponding kernel source release should have been downloaded from kernel.org.

First of all, the kernel to be debugged needs to have kgdb enabled and built into it. To achieve that, the kernel source has to be configured with CONFIG_KGDB=y. Additionally, for kgdb over serial, CONFIG_KGDB_SERIAL_CONSOLE=y needs to be configured. And CONFIG_DEBUG_INFO is preferred for symbolic data to be built into the kernel, to make debugging with gdb more meaningful. CONFIG_FRAME_POINTER=y enables frame pointers in the kernel, allowing gdb to construct more accurate stack back-traces. All these options are available under “Kernel hacking” in the menu obtained in the kernel source directory (preferably as root, or using sudo), by issuing the following command:

$ make mrproper      # To clean up properly
$ make oldconfig     # Configure the kernel same as the current running one
$ make menuconfig    # Start the ncurses based menu for further configuration

See the highlighted selections in Figure 1, for how and where these options would be:

• “KGDB: kernel debugging with remote gdb” –> CONFIG_KGDB
• “KGDB: use kgdb over the serial console” –> CONFIG_KGDB_SERIAL_CONSOLE
• “Compile the kernel with debug info” –> CONFIG_DEBUG_INFO
• “Compile the kernel with frame pointers” –> CONFIG_FRAME_POINTER

Once configuration is saved, build the kernel (run make), and then a make install to install it, along with adding an entry for the installed kernel in the GRUB configuration file. Depending on the distribution, the GRUB configuration file may be /boot/grub/menu.lst, /etc/grub.cfg, or something similar. Once installed, the kgdb-related kernel boot parameters need to be added to this new entry, as shown in the highlighted text in Figure 2.

kgdboc is for gdb connecting over the console, and the basic format is kgdboc= <serial_device>, <baud-rate> where:

• <serial_device> is the serial device file (port) on the system running the kernel to be debugged
• <baud-rate> is the baud rate of this serial port

kgdbwait tells the kernel to delay booting till a gdb client connects to it; this parameter should be given only after kgdboc.

With this, we’re ready to begin. Make a copy of the vmlinux kernel image for use on the gdb client system. Reboot, and at the GRUB menu, choose the new kernel, and then it will wait for gdb to connect over the serial port.

All the above snapshots are with kernel version 2.6.33.14. The same should work for any 2.6.3x release of the kernel source. Also, the snapshots for kgdb are captured over the serial device file /dev/ttyS0, i.e., the first serial port.

Setting up gdb on another system

Following are the prerequisites:

• Serial ports of the system to be debugged, and the other system to run gdb, should be connected using a null modem (i.e., a cross-over serial) cable.
• The vmlinux kernel image built, with kgdb enabled, needs to be copied from the system to be debugged, into the working directory on the system where gdb is going to be run.

To get gdb to connect to the waiting kernel, launch gdb from the shell and run these commands:

(gdb) file vmlinux
(gdb) set remote interrupt-sequence Ctrl-C
(gdb) set remotebaud 115200
(gdb) target remote /dev/ttyS0
(gdb) continue

In the above commands, vmlinux is the kernel image copied from the system to be debugged.

Debugging using gdb with kgdb

After this, it is all like debugging an application from gdb. One may stop execution using Ctrl+C, add break points using b[reak], stop execution using s[tep] or n[ext] … — the usual gdb way. There are enough GDB tutorials available online, if you need them. In fact, if you are not comfortable with text-based GDB, use any of the standard GUI tools over gdb, like ddd, Eclipse, etc.

Summing up

By now, Shweta was excited about wanting to try out kgdb. Since she needed two systems to try it out, she went to the Linux device drivers’ lab. There, she set up the systems and ran gdb as described above.