Debugging Kernel Crash
The kernel dump tools (kdump-tools) package facilitates the configuration and management of kernel crash dumps in Linux systems. It automates the setup of kdump, a mechanism that captures the system memory (vmcore) when the kernel encounters a critical failure. This captured vmcore is invaluable for post-mortem analysis and debugging.
Kernel Dump Extraction and Analysis
To analyze a kernel memory dump, extract the relevant files and run the crash analysis utility.
Note: This utility does not function if the system has less than 7 GB of memory.
With Technical Support Archive
1. Create a technical support archive using CLI.
#cmlsh
>en
#show cores
#show techsupport all
#exit
The system saves the archive at:
/var/log/OcNOS_tech_support_all_<DATE>_<TIME>.tar.gz
2. Extract the technical support archive and core dump files.
#tar -xf /var/log/OcNOS_tech_support_all_<date>_<time>.tar.gz
#tar -xf core_kdump_<timestamp>.tar
This extracts the vmcore at:
./<timestamp>/dump.<timestamp>
3. Extract debug-enabled kernel image (vmlinux tar file).
cd /lib/debug/
tar -xf vmlinux-<kernel_version>.tar.gz
4. Analyze the dump using crash.
crash /lib/debug/vmlinux /path/to/vmcore
Without Technical Support Archive
1. Extract the kdump archive (tar files).
cd /var/log/crash/cores/
tar -xf core_kdump_<timestamp>.tar
2. Extract debug-enabled kernel image (vmlinux tar file).
cd /lib/debug/
tar -xf vmlinux-<kernel_version>.tar.gz
3. To analyze the dump, run the crash command:
crash /lib/debug/vmlinux /path/to/vmcore
#crash /lib/debug/vmlinux /var/log/crash/cores/<timestamp>/dump.<timestamp>
crash 8.0.6
Copyright (C) 2002-2025 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation
Copyright (C) 1999-2006 Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited
Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011, 2020-2024 NEC Corporation
Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
Copyright (C) 2015, 2021 VMware, Inc.
This program is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Enter “help copying” to see the conditions. This program has absolutely no warranty. Enter “help warranty” for details.
GNU gdb (GDB) 10.2
Copyright (C) 2021 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type “show copying” and “show warranty” for details.
This GDB was configured as “x86_64-pc-linux-gnu”.
Type “show configuration” for configuration details.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type “help”.
Type “apropos word” to search for commands related to “word”.
KERNEL: /lib/debug/vmlinux [TAINTED]
DUMPFILE: /var/log/crash/cores/202506160433/dump.202506160433 [PARTIAL DUMP]
CPUS: 4
DATE: Mon Jun 16 04:33:22 UTC 2025
UPTIME: 09:59:00
LOAD AVERAGE: 0.18, 0.29, 0.40
TASKS: 229
NODENAME: DUT1
RELEASE: 6.1.76-g8720581cb
VERSION: #1 SMP PREEMPT_DYNAMIC Wed Mar 5 11:17:59 UTC 2025
MACHINE: x86_64 (2400 Mhz)
MEMORY: 16 GB
PANIC: "Kernel panic - not syncing: sysrq triggered crash"
PID: 7826
COMMAND: "bash"
TASK: ffff888106216e00 [THREAD_INFO: ffff888106216e00]
CPU: 0
STATE: TASK_RUNNING (PANIC)
crash>
crash>
crash> bt
PID: 4534 TASK: ffff888105de8000 CPU: 0 COMMAND: "bash"
#0 [ffffc900005afc78] machine_kexec at ffffffff81056fe1
#1 [ffffc900005afcc8] __crash_kexec at ffffffff8113bf92
#2 [ffffc900005afd88] panic at ffffffff81e5878a
#3 [ffffc900005afe08] sysrq_handle_crash at ffffffff81733781
#4 [ffffc900005afe10] __handle_sysrq.cold at ffffffff81e81771
#5 [ffffc900005afe40] write_sysrq_trigger at ffffffff8173422f
#6 [ffffc900005afe50] proc_reg_write at ffffffff8135a050
#7 [ffffc900005afe68] vfs_write at ffffffff812d5c52
#8 [ffffc900005aff00] ksys_write at ffffffff812d6246
#9 [ffffc900005aff38] do_syscall_64 at ffffffff81ebf182
#10 [ffffc900005aff50] entry_SYSCALL_64_after_hwframe at ffffffff820000dc
RIP: 00007f59a8dee2c0 RSP: 00007ffca5d70058 RFLAGS: 00000202
RAX: ffffffffffffffda RBX: 0000000000000002 RCX: 00007f59a8dee2c0
RDX: 0000000000000002 RSI: 0000555df62882c0 RDI: 0000000000000001
RBP: 0000555df62882c0 R8: 0000000000000007 R9: 0000000000000073
R10: 0000000000000000 R11: 0000000000000202 R12: 0000000000000002
R13: 00007f59a8ec9760 R14: 0000000000000002 R15: 00007f59a8ec49e0
ORIG_RAX: 0000000000000001 CS: 0033 SS: 002b
crash>
Basic Commands in Crash
After launching the crash shell (indicated by the crash> prompt), use the following commands to inspect the system state at the time of the crash:
bt: Displays the stack trace for all tasks. This helps identify where each process was executing when the crash occurred.
ps: Lists all processes and their statuses, providing insight into the system's process table at the moment of the crash.
vm: Shows virtual memory information, useful for diagnosing memory-related issues.
files: Displays open files for a specific process, aiding in understanding resource utilization.
help: Lists all available commands within the crash utility for further exploration.