The Ultimate Guide to Viewing and Analyzing Core Dump Files on Ubuntu

When developing software, particularly in languages like C and C++, crashes are inevitable. The dreaded “Segmentation fault (core dumped)” message can be frustrating, but the core dump it generates is a valuable debugging tool.

In this comprehensive guide, I’ll walk you through locating, configuring, and analyzing core dumps on Linux systems, with a special focus on Ubuntu distributions.

What is a Core Dump?

A core dump is essentially a snapshot of a program’s memory at the moment it crashed. This snapshot contains the program’s entire state, including the values of variables, the call stack, and register contents. Core dumps are invaluable for diagnosing issues that are difficult to reproduce or occur intermittently.

Read: A Deep Dive into Linux Operating System Architecture: Components and Functions

Enabling Core Dumps in Linux

By default, many Linux distributions, including Ubuntu, have core dumps disabled. Let’s first check if core dumps are enabled:

ulimit -c

If this returns 0, core dumps are disabled. To enable them for your current terminal session:

# Set core dump size to unlimited
ulimit -c unlimited

# Verify the setting
ulimit -c

This change is only active for your current terminal session. To make it permanent, add the command to your shell’s configuration file (like .bashrc).

Where Are Core Dumps Located?

The location of core dumps depends on your Linux distribution and configuration. Here are the most common places to look:

1. Current Working Directory

In traditional Linux systems, core dumps appear in the directory where the program was executed, named simply core.

2. For Ubuntu and systemd-based Systems

On modern Ubuntu systems (20.04 and newer), core dumps are often handled by systemd’s coredump service or by apport (Ubuntu’s crash reporting system). You can find them in:

• /var/lib/apport/coredump/ (for Ubuntu with apport)
• /var/crash/ (for Ubuntu .crash files)
• /var/lib/systemd/coredump/ (for systemd-based systems)

You can check where your system is configured to place core dumps with:

cat /proc/sys/kernel/core_pattern

If it starts with a pipe symbol (|), your system is forwarding core dumps to a program like apport or systemd-coredump.

3. Checking Apport Logs

If you can’t find your core dump file, check the apport logs:

cat /var/log/apport.log

This often reveals where core dumps are being saved or why they’re not being generated.

Read: A Guide to Viewing and Monitoring Error Logs in Ubuntu

Configuring Core Dumps for Ubuntu

For Ubuntu users, you might need to configure apport to handle core dumps for non-packaged applications:

# Create the apport configuration directory if it doesn't exist
mkdir -p ~/.config/apport

# Configure apport to handle core dumps for non-packaged applications
echo '[main]
unpackaged=true' > ~/.config/apport/settings

Analyzing Core Dumps with GDB

The GNU Debugger (GDB) is the tool of choice for analyzing core dumps. Here’s how to use it effectively:

1. Building Your Program with Debug Symbols

For meaningful debugging, your program needs to be compiled with debug symbols:

# For C programs
gcc -Wall -Wextra -Werror -ggdb -O0 -std=gnu17 -o myprogram myprogram.c

# For C++ programs
g++ -Wall -Wextra -Werror -ggdb -O0 -std=gnu++17 -o myprogram myprogram.cpp

The -ggdb -O0 flags are crucial for debugging:

• -ggdb: Generates debug information specifically for GDB
• -O0: Disables compiler optimizations which can make debugging confusing

2. Opening the Core Dump in GDB

Once you have a core dump, open it with GDB:

gdb path/to/your/executable path/to/core/dump

For example:

gdb ./myprogram core

Or for Ubuntu’s apport-managed core dumps:

# First, extract the core dump from the .crash file
apport-unpack /var/crash/myprogram.1000.crash extracted_crash
gdb ./myprogram extracted_crash/CoreDump

3. Basic GDB Commands for Core Dump Analysis

After opening the core dump in GDB, use these commands to analyze it:

bt                # Display the backtrace (call stack)
bt full           # Display the backtrace with all variables
frame N           # Select frame N from the backtrace
info locals       # Display local variables in current frame
print variable    # Print value of a variable
x/Nx address      # Examine N words of memory at address

The bt (backtrace) command is particularly useful as it shows the function call stack at the time of the crash, helping you pinpoint where things went wrong.

Practical Example: Forcing and Analyzing a Core Dump

Let’s walk through a practical example of forcing a core dump and analyzing it:

1. First, enable core dumps:

ulimit -c unlimited

2. Create a simple C program that will crash:

#include 

int main(void) {
    // Deliberately cause a segmentation fault
    int *ptr = NULL;
    *ptr = 42;  // This will crash
    return 0;
}

3. Compile it with debug symbols:

gcc -Wall -ggdb -O0 -o crash crash.c

4. Run the program (it will crash):

./crash

5. Locate the core dump (location varies by system)
6. Analyze it with GDB:

gdb ./crash core

7. In GDB, view the backtrace:

(gdb) bt

You should see where the program crashed, pointing to the line that attempted to dereference the null pointer.

Direct Debugging with GDB

Instead of generating a core dump and then analyzing it, you can run your program directly in GDB:

gdb ./myprogram

Then, within GDB:

(gdb) run

When the program crashes, GDB will catch it and allow you to examine the state immediately. This approach has several advantages:

• You can set breakpoints before the crash
• You can examine variables and memory in real-time
• You can single-step through code to observe behavior

Troubleshooting Core Dumps

Core Dumps Not Generated

If core dumps aren’t being generated despite enabling them:

1. Check system limits:

ulimit -a

2. Verify the core pattern:

cat /proc/sys/kernel/core_pattern

3. For Ubuntu, check if apport is handling crashes:

systemctl status apport

Core Dumps Too Large

If your core dumps are consuming too much disk space:

# Limit core dump size to 100MB
ulimit -c 102400

Handling Multiple Core Dumps

Core dumps don’t automatically overwrite existing files. Always remove old core files before generating new ones:

rm core

Advanced Techniques

Logging GDB Output

To save GDB analysis for later review:

(gdb) set logging file debug_log.txt
(gdb) set logging on
(gdb) bt full
(gdb) info locals
(gdb) set logging off

Examining Multiple Threads

For multithreaded programs:

(gdb) info threads
(gdb) thread apply all bt

Using Mozilla’s RR for Time-Travel Debugging

For complex debugging scenarios, consider Mozilla’s RR tool, which allows you to record program execution and “time travel” back and forth through the execution:

# Install RR
sudo apt-get install rr

# Record program execution
rr record ./myprogram

# Replay and debug
rr replay

Conclusion

Core dumps are powerful debugging tools that provide a snapshot of your program at the moment of failure. By understanding how to locate, configure, and analyze core dumps, you’ve added a valuable skill to your debugging arsenal. While the process might seem complex at first, the insight core dumps provide into otherwise elusive bugs makes them worth mastering.

Remember that different Linux distributions handle core dumps differently, so always check your system’s specific configuration. With practice, analyzing core dumps will become second nature, helping you solve even the most challenging bugs.

FAQ

Q: Why can’t I find my core dump files?

A: Core dumps might be redirected to different locations depending on your system configuration. Check /var/lib/apport/coredump/, /var/crash/, or run cat /proc/sys/kernel/core_pattern to see where they’re being sent.

Q: How do I make core dumps persist across reboots?

A: Add ulimit -c unlimited to your .bashrc or /etc/security/limits.conf file.

Q: Can I analyze core dumps from programs without source code?

A: Yes, but with limited information. You’ll see function addresses but not source code lines or variable names.

Q: How do I analyze core dumps from programs with shared libraries?

A: Ensure GDB can find the same shared libraries that were loaded when the program crashed. You might need to set the LD_LIBRARY_PATH environment variable.

Q: How do I prevent core dumps from filling up my disk?

A: Set a reasonable limit with ulimit -c SIZE where SIZE is in blocks (1024 bytes).

Q: Can I get core dumps from programs running as services?

A: Yes, but you might need to configure systemd or other service managers to allow core dumps and specify where to save them.

Q: How do I analyze core dumps from programs that use threads?

A: Use thread apply all bt in GDB to see backtraces for all threads.

Q: Why does my core dump file have a strange name?

A: Modern systems often use naming patterns that include the PID and other information to avoid overwriting existing core dumps.

The post The Ultimate Guide to Viewing and Analyzing Core Dump Files on Ubuntu appeared first on net2.