Debug Kernel Panic in Centos

When you face a new machine and have to compile a kernel yourself for some reason (not to be idle), you will encounter a kernel panic.

Kernel panic is very annoying, panic at startup is even more annoying. Panic without any log is particularly annoying without the hard disk.

Provide several fooling methods to solve the problem: (The following content is for the redhat system, but most of the methods are general)

Determine the link that caused panic####

Briefly describe the startup process:

1 Power On                                 Maybe Err:Worlds Collides
2 BIOS                                     Maybe Err:Worlds Collides
3 Load Grub From MBR                       Maybe Err:See nothing
4 load Grub and show it                    Maybe Err:Grub loads failed
5 Grub reads menu.list                     Maybe Err:Grub loads failed
6 Grub loads kernel image                  Maybe Err:Grub loads failed
7 kernel mounts root filesystem            Maybe Err:PANIC
8 kernel runs init                         Maybe Err:PANIC
9 init runs scripts to start               Maybe Err:PANIC

First of all, you have to confirm which link caused the panic. If your log is relatively complete, you can generally judge the problem caused by the link above based on clues. Generally panic occurs in step7, step8, and step9 links.

step9: It's no problem to get here, it's nothing more than /etc/rc.sysinit and other devices that shouldn't have been mounted, and services that don't know what horses are started, you can directly type various log debugging in the startup script
step8: This is more tricky. Generally, initramfs is decompressed and executed some scripts to report errors. Fortunately, most of them are at the init level. Generally, you can add parameters after the grub kernel, mount the shell to adjust, or directly use tools to modify initramfs Files in the mirror image, repackage the dichotomy positioning
step7: It's also tricky. Generally, panic always suspects the disk drive. I think this is a major cause of panic. In addition, if the screen does not output anything, it is estimated that the video driver will have to play around.

The following provides some tricks for step7 and step8 panic

Add debugging switch to the kernel parameter, turn off ACPI, selinux

1234	title CentOS (2.6.32-358.el6.x86_64) root (hd0,0) kernel /vmlinuz-2.6.32-358.el6.x86_64 ro root=/dev/mapper/vg_localhost-lv_root rd_NO_LUKS rd_NO_MD rd_LVM_LV=vg_localhost/lv_swap crashkernel=128M LANG=zh_CN.UTF-8 rd_LVM_LV=vg_localhost/lv_root KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM debug selinux=0 acpi=0 initrd /initramfs-2.6.32-358.el6.x86_64.img

This is the time to test the character, no matter what the situation is, the most likely problem is eliminated first. In addition, the redhat system will add parameters such as rhgb quiet to the end of the kernel, and remove them all. When booting up, stare at the screen to see if there is any suspicious output. Because the system may output a large number of logs in a short period of time without recording, you need to set the log output rate and pause timing. More detailed parameters can be found here:

https://wiki.archlinux.org/index.php/Boot_debugging

Add the dracut debugging switch to the kernel parameter####

In the later stage of the redhat system, dracut was used to build the initrd image. The manual about dracut is here:

http://people.redhat.com/harald/dracut-rhel6.html#lsinitrd

The debug switch is here:

https://fedoraproject.org/wiki/How_to_debug_Dracut_problems

You can set rdshell, jump into the shell provided by dracut after panic, and type dmesg to see the log information.

In the end, I really don’t know why, and you are very idle, you can consider adding serial device debugging when starting

Some tooltip

When compiling the kernel, make menuconfig, the kernel after 3.10 supports searching for a certain switch and then pressing the number key to jump directly to the setting item of that switch. This is very useful
Redhat uses dracut to build initrd, and the configuration file is in /usr/share/dracut/ by default. If you are too lazy to change the initramfs, you can directly modify the configuration file inside, and then make install again.
Sometimes you can't confirm where the root file system is mounted, you can refer to here:

http://free-electrons.com/blog/find-root-device/

Other references:

http://www.tuxradar.com/content/how-fix-linux-boot-problems