Linux Kernel Debugging Essentials workshop

Linux Kernel Debugging Essentials workshop

• 1.
  Linux Kernel DebuggingEssentials Lubomir Rintel base48 Hackerspace, Brno http://base48.cz/ February 19th , 2014 lkundrak@v3.sk Kernel Debugging February 19th , 2014 1 / 12
• 2.
  Plan Understanding the waykernel can fail Introduction to infrastructure Obtaining & unserstanding the traces Heplful in-kernel mechanisms Building and trying the replacement code lkundrak@v3.sk Kernel Debugging February 19th , 2014 2 / 12
• 3.
  Kernel failures Warning, possiblynon-fatal: OOPS A fatal error: Panic Usually accompained by a state dump Result of a bug or a hardware failure # echo c >/proc/sysrq-trigger lkundrak@v3.sk Kernel Debugging February 19th , 2014 3 / 12
• 4.
  Kernel configuration Vanilla kernels Distributionkernels Configuration options (build-time) Runtime configuration (sysctls) Boot time configuration (kernel parameters) Documentation/kernel-parameters.txt lkundrak@v3.sk Kernel Debugging February 19th , 2014 4 / 12
• 5.
  Lab setup For details& non-Fedora setup, see the hackerspace wiki # yum -y install kernel-debug{,-devel} # debuginfo -y install kernel-debug # yum -y install kexec-tools crash lkundrak@v3.sk Kernel Debugging February 19th , 2014 5 / 12
• 6.
  Console output Message buffer Screen,UART, USB debugging dongle Early console for boot problems Slowed down output possible Syslog via Netpoll API # dmesg rhgb, quiet console={ttyS*,uart} earlyprintk={vga,serial,dbgp} kernel.printk_delay kernel.printk_ratelimit netconsole.ko g_dbgp.ko lkundrak@v3.sk Kernel Debugging February 19th , 2014 6 / 12
• 7.
  Reading OOPS messages Processorstatus Traceback # objdump -lSd {vmlinux,module.ko} # addr2line lkundrak@v3.sk Kernel Debugging February 19th , 2014 7 / 12
• 8.
  Memory dumps Crash dumps OHCI1394DMA, can be early too Hypervisor specific dumps We can extract console output from them! crashkernel=auto # systemctl start kdump.service # firescope lkundrak@v3.sk Kernel Debugging February 19th , 2014 8 / 12
• 9.
  Analyzing dumps crash lkundrak@v3.sk Kernel Debugging February19th , 2014 9 / 12
• 10.
  Helpful kernel tools Kernelhacking menu Lockdep detector Hung task detector List corruption detector Stack protector Memory poison CONFIG_DEBUG_* kernel.sysrq kernel.*panic* kernel.prove_locking kernel.hung_task_timeout_secs lkundrak@v3.sk Kernel Debugging February 19th , 2014 10 / 12
• 11.
  Loading our codeinto kernel Understand kernel coding Pick a good book Linux Kernel Development, Robert Love, Addison-Wesley: Developer’s Library Understanding the Linux Kernel, Corbet et al., O’Reilly (available online) Try out what do the debugging facilities do! # make -C /lib/modules/$(uname -r)/build SUBDIRS=$PWD modules # insmod module.ko lkundrak@v3.sk Kernel Debugging February 19th , 2014 11 / 12
• 12.
  Thank you forattending! The source to the slides are embedded in the PDF file. Use pdftk or Acrobat Reader’s ”attachments” to extract them. lkundrak@v3.sk Kernel Debugging February 19th , 2014 12 / 12