Debugging Kernel Crash
The kdump-tools package facilitates the configuration and management of kernel crash dumps in Linux systems. It automates the setup of kdump, a mechanism that captures the system memory (vmcore) when the kernel encounters a critical failure. This captured vmcore is invaluable for post-mortem analysis and debugging.
System Requirement: (Pre Requisites for Kdump-tools)
For kdump-tools, dedicated memory should be reserved to load secondary kernel during kernel panic, hence 256M of system Memory will be reserved if total memory is more than 8G. If total memory is less than 8GB, Kdump will not work.
Simulating Kernel Crash Manually (Demo)
1. Enable sysrq triggers:
echo 1 > /proc/sys/kernel/sysrq
2. Trigger a panic:
echo c > /proc/sysrq-trigger (system will start rebooting to handle crash)
3. After Reboot, kernel dump will be stored in path /var/crash/cores/
4. Extract kdump Tar files
For example:
cd /var/log/crash/cores/
tar -xf core_kdump_202506160433.tar
5. Extract vmlinux Tar file
For example:
cd /lib/debug/
tar -xf vmlinux-6.1.76-g8720581cb.tar.gz
6. To analyze the dump, run the crash commandr
crash /lib/debug/vmlinux /path/to/vmcore
For example:
# crash /lib/debug/vmlinux /var/log/crash/cores/202506160433/dump.202506160433
crash 8.0.6
Copyright (C) 2002-2025 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation
Copyright (C) 1999-2006 Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited
Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011, 2020-2024 NEC Corporation
Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
Copyright (C) 2015, 2021 VMware, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter “help copying” to see the conditions.
This program has absolutely no warranty. Enter “help warranty” for details.
GNU gdb (GDB) 10.2
Copyright (C) 2021 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type “show copying” and “show warranty” for details.
This GDB was configured as “x86_64-pc-linux-gnu”.
Type “show configuration” for configuration details.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type “help”.
Type “apropos word” to search for commands related to “word”.
KERNEL: /lib/debug/vmlinux [TAINTED]
DUMPFILE: /var/log/crash/cores/202506160433/dump.202506160433 [PARTIAL DUMP]
CPUS: 4
DATE: Mon Jun 16 04:33:22 UTC 2025
UPTIME: 09:59:00
LOAD AVERAGE: 0.18, 0.29, 0.40
TASKS: 229
NODENAME: DUT1
RELEASE: 6.1.76-g8720581cb
VERSION: #1 SMP PREEMPT_DYNAMIC Wed Mar 5 11:17:59 UTC 2025
MACHINE: x86_64 (2400 Mhz)
MEMORY: 16 GB
PANIC: "Kernel panic - not syncing: sysrq triggered crash"
PID: 7826
COMMAND: "bash"
TASK: ffff888106216e00 [THREAD_INFO: ffff888106216e00]
CPU: 0
STATE: TASK_RUNNING (PANIC)
crash>
crash>
crash> bt
PID: 4534 TASK: ffff888105de8000 CPU: 0 COMMAND: "bash"
#0 [ffffc900005afc78] machine_kexec at ffffffff81056fe1
#1 [ffffc900005afcc8] __crash_kexec at ffffffff8113bf92
#2 [ffffc900005afd88] panic at ffffffff81e5878a
#3 [ffffc900005afe08] sysrq_handle_crash at ffffffff81733781
#4 [ffffc900005afe10] __handle_sysrq.cold at ffffffff81e81771
#5 [ffffc900005afe40] write_sysrq_trigger at ffffffff8173422f
#6 [ffffc900005afe50] proc_reg_write at ffffffff8135a050
#7 [ffffc900005afe68] vfs_write at ffffffff812d5c52
#8 [ffffc900005aff00] ksys_write at ffffffff812d6246
#9 [ffffc900005aff38] do_syscall_64 at ffffffff81ebf182
#10 [ffffc900005aff50] entry_SYSCALL_64_after_hwframe at ffffffff820000dc
RIP: 00007f59a8dee2c0 RSP: 00007ffca5d70058 RFLAGS: 00000202
RAX: ffffffffffffffda RBX: 0000000000000002 RCX: 00007f59a8dee2c0
RDX: 0000000000000002 RSI: 0000555df62882c0 RDI: 0000000000000001
RBP: 0000555df62882c0 R8: 0000000000000007 R9: 0000000000000073
R10: 0000000000000000 R11: 0000000000000202 R12: 0000000000000002
R13: 00007f59a8ec9760 R14: 0000000000000002 R15: 00007f59a8ec49e0
ORIG_RAX: 0000000000000001 CS: 0033 SS: 002b
crash>
Basic Commands in crash
Once inside the crash shell (as shown above crash> prompt), several commands are available to inspect various aspects of the system's state at the time of the crash:
bt: Displays the stack trace of all tasks, helping identify where each process was executing when the crash occurred.
ps: Lists all processes and their statuses, providing insight into the system's process table at the crash moment.
vm: Shows virtual memory information, useful for diagnosing memory-related issues.
files: Displays open files for a specific process, aiding in understanding resource utilization.
help: Lists all available commands within the crash utility for further exploration.
Interpreting the Output
The output from these commands can help identify the root cause of the crash. For instance, the bt command's stack trace can pinpoint the exact function where the kernel panicked, while the ps command can reveal any processes in uninterrupted sleep states, indicating potential deadlocks or resource waits.
For a comprehensive guide on using the crash utility, refer to Red Hat's documentation.