The last time I was involved with diagnosing kernel panics was in the early 2000, using tools like the Linux Kernel Crash Dump (LKCD) project [1] with some success. Further attempts with Netdump around 2002 had only moderate success.
I decided to get back to the basics and see how much the field of diagnosing kernel crashes has improved, by looking at the Kdump utilities [2].
The problem with the older tools was the fact that you cant quite trust a kernel that just crashed. The questions you need to ask are if the kernel structures and data are intact and, if all the important drivers are still correctly initialized, so that the dumps can be written to the file system (local, NFS or scp). It is better to use an entity outside of the kernel to save the kernel dumps.
Kdump addresses these concerns by providing two components - a fast-booting mechanism that allows booting a Linux kernel from the context of an already running kernel without going through BIOS, called Kexec and capturing the crash dump from the context of a freshly booted kernel and not from the context of the crashed kernel, which guarantees that the necessary kernel drivers will be initialized, named Kdump (confusingly the same name as the main package).
In the following example I'll demonstrate a simple scenario on how to install, configure, crash and diagnose a kernel dump file with kdump and the crash [3] utility on CentOS 6.3.
First let's install all the necessary packages:
The debuginfo packages provide the uncompressed vmlinux kernel that the crash utility depends on. It should be present at /usr/lib/debug/lib/modules/2.6.32-279.19.1.el6.x86_64/vmlinux.
Next make sure that the kernel is compiled with support for saving crash dumps etc:
This is already the case with CentOS 6.3, but if the modules are not enabled you'll have to get the kernel source and enable the options as shown above.
Another prerequisite for the crash kernel to be loaded at the time of the problem is to reserve some memory area for it. This needs to be configured in the original kernel's bootloader and can be done with the bootloader option crashkernel=XM[ @YM], where X is the size of the reserved area in megabytes and Y specifies the start address. If you omit the second parameter or set it to 0M, the system automatically selects an appropriate location. This is already done on CentOS and to verify it run:
If this is not the case make sure you append crashkernel=auto (or specify memory as suggested above) in the kernel parameters string in grub.conf and reboot:
Run this to verify that the kernel has reserved memory for the crash kernel (note the crashkernel=129M@0M parameter):
Start kdump and make it persist reboots:
Kdump has many usefull configuration options allowing the dump to be copied using NFS or ssh to a remote host etc, but for this example the defaults are sufficient. To see the configs cat the following files:
Time to install the crash system utility for diagnosing the core dumps:
Now that all the prerequisites have been installed and the kdump daemon is running with the crash kernel resident in memory, let's crash the production kernel artificially by using the Magic SysRequest Keys that I wrote about here.
Line 1 enables the Magic SysRequest Keys and line 2 triggers kernel panic.
Now observe the console and you should see the crash kernel dumping the memory of the production kernel and creating the vmcore file on the specified file system (by default the file will go in a sub-directory in /var/crash/).
To analyse what caused the crash we can run the crash utility like so:
The three interesting lines are line 15, 16 and 17.
The PANIC line shows the reason for the panic, the PID of the process that caused the panic and the COMMAND that was running.
We can see a backtrace and the processes by running bt and ps respectively:
As you can see this tells us that the kernel panic was caused by the bash process with id of 1295, executing the sysrq call.
For more information refer to the free Linux Kernel Crash Book by Igor Ljubuncic [4].
