Mastering Systemd Logging: A Sysadmin’s Guide to journalctl for Linux Troubleshooting

On Linux systems, crucial messages from the kernel, system services, and running applications are carefully recorded in log files. Think of these logs as the system’s diary.

Different types of information often go into separate files – security events might have their own log, while scheduled `cron` tasks report to another.

Because errors and important events are logged, system administrators rely heavily on these files to diagnose and troubleshoot a wide range of Linux issues. By analyzing logs collected by `systemd`, sysadmins can investigate problems like failed login attempts, application crashes, memory issues, or boot failures.

In this guide, we’ll explore how to use `systemd`’s logging tools, focusing primarily on the powerful `journalctl` command. You’ll learn how to find and view log entries effectively and harness the valuable information they contain to solve system problems more quickly and easily. Understanding how

to use journalctl for systemd log analysis is an essential skill for any Linux administrator.

1. An Introduction to systemd

When your Linux system starts up, `systemd` acts as the initial process and orchestrator. It manages which programs run at boot and throughout the system’s operation. Beyond just starting services, `systemd` also handles tasks like maintaining a system activity journal (our main focus here), scheduling jobs similar to `cron`, managing the network configuration, handling user logins, and much more.

One of the significant advantages `systemd` brought is its approach to process and system logging. Older methods often involved various daemons (like `syslog`, `syslog-ng`, `rsyslog`) managing separate log files scattered across the system. This decentralization could make comprehensive analysis difficult. `systemd`, through its `journald` daemon, provides a centralized logging mechanism.

To tackle the complexity of dispersed logs, `systemd` implements a unified management strategy for logs originating from users, the kernel, and system processes, similar to how it manages services and the boot sequence. This centralized collection and management is handled by the systemd Journal, implemented by the `journald` daemon.

Courtesy: ZDNet

Having a single tool, `journalctl`, to interact with this centralized log data empowers users and system administrators to view and analyze system activity efficiently for rapid diagnostics.

Furthermore, the journal stores log data in a binary format. This allows the logs to be structured and indexed efficiently, and enables `journalctl` to display them in various output formats as needed. While sysadmins might prefer the familiar standard syslog format for daily checks, the binary storage makes other formats possible. For instance, if you needed to visualize service outages or kernel events, you could output log entries as JSON objects to feed into graphing software, demonstrating the flexibility of systemd binary log formats.

Read: How to set up a firewall on Ubuntu 22.04

2. Understanding the journald Daemon

The `journald` daemon is the core component of `systemd` responsible for managing log files. It actively collects messages from numerous sources.

`journald` captures standard Syslog messages, kernel log messages (like those from `dmesg`), messages generated during the early boot process and by the initial RAM disk (initrd), and even messages written to the standard output and standard error streams of all managed services. It then indexes this diverse data and makes it readily available through `journalctl` for easy manipulation and analysis.

The native journal file format is binary, structured, and indexed. This design leads to faster log retrieval and significantly improved search capabilities compared to plain text logs. It also allows `journald` to store rich metadata alongside each log message, such as timestamps, user IDs (UIDs), process IDs (PIDs), and service unit names.

The `systemd` journal can function as a complete replacement for traditional syslog implementations, or it can coexist and integrate with them, depending on your system’s configuration and requirements. For many administrators, the capabilities of the `systemd` journal, accessed via `journalctl`, meet most logging needs while still allowing integration with other familiar logging tools.

Read: How to fix high memory usage in Linux

3. Timestamps in Log Records

A key advantage of the binary journal format is its reliable handling of timestamps. Logs can be viewed accurately in the system’s local time or converted to any other timezone on the fly. Before diving deep into log analysis, it’s helpful to ensure your system’s time is set correctly. The `timedatectl` utility, also part of the `systemd` suite, helps manage time and timezone settings.

To see a list of timezones available on your system, run:

timedatectl list-timezones

Listing available timezones with timedatectl

Scroll through the list (press ‘q’ to quit). Once you find the appropriate timezone identifier (e.g., `America/New_York`, `Europe/London`), you can set it using the `set-timezone` option. Remember to use `sudo` as this changes a system setting:

sudo timedatectl set-timezone 

(Replace “ with the identifier you chose, like `Europe/Paris`).

Setting the system timezone

To verify that the system recognized the change, use the `status` option:

timedatectl status

Checking the current time and timezone status

The output should now reflect the timezone you just set. From this point on, `journalctl` will display timestamps in this local time by default. If you specifically need to view timestamps in Coordinated Universal Time (UTC), you can use the `–utc` flag with `journalctl`:

journalctl --utc

Understanding how to configure timezone for journalctl logs ensures accurate time correlation during analysis.

4. Viewing Basic Logs

The primary command for interacting with the logs collected by `journald` is `journalctl`. Simply running the command without any options will display the journal entries gathered so far.

journalctl

By default, `journalctl` shows the logs within a pager (like `less`), with the oldest entries appearing at the top. You can navigate using standard pager keys (like PageUp, PageDown, arrow keys, `/` for searching, `q` to quit).

Basic log view using journalctl

If `systemd` has been active on your system for a while, this command might return hundreds or even thousands of pages of logs, highlighting the comprehensive nature of the journal database.

The default output format might look familiar if you’ve worked with traditional syslog messages. However, system administrators need to remember that the data displayed by `journalctl` often comes from a wider array of sources than standard syslog implementations typically collect. This includes crucial Linux error logs from the kernel, messages from the initial RAM disk (initrd), output from early boot processes, and standard output/error streams from applications managed by `systemd`.

Read: How to keep Ubuntu clean 

5. Journal Filtering Options

Given the potentially huge volume of logs, filtering is essential for effective analysis. For detailed information on the numerous ways to filter logs by time, service, priority, and other criteria, please refer to our dedicated article on how to analyze Linux systemd logs using journalctl advanced filtering options.

6. Customizing Journal Output Format

Beyond filtering, `journalctl` allows you to change how the log data is presented. You can adjust the verbosity, truncate long lines, or bypass the pager entirely to send the output elsewhere.

If log lines are very long and wrapping makes them hard to read, you can truncate them using the `–no-full` option:

journalctl --no-full

This often results in a more compact display, showing the beginning of each message, sometimes ending with ellipses (…) if truncated.

Truncated output using journalctl –no-full

7. Outputting Logs to Files or Standard Output

By default, `journalctl` uses a pager. If you want to send the log output directly to standard output (stdout) – for example, to pipe it to text processing tools like `grep`, `sed`, `awk`, or redirect it to a file – use the `–no-pager` option:

journalctl --no-pager

You would typically immediately follow this command with a pipe (`|`) or redirection (`>`). For example: journalctl --no-pager > my_system_logs.txt

JSON Output Formats

For programmatic processing or integration with other tools, outputting logs in JSON format is often useful. Use the -o (output) flag followed by `json`:

journalctl -b -u ssh.service -o json

(This example gets logs for the `ssh.service` from the current boot in standard JSON format).

Log output in JSON format

For a more human-readable JSON output with indentation and line breaks, use `json-pretty`:

journalctl -b -u ssh.service -o json-pretty

Log output in pretty-printed JSON format

`journalctl` supports several other output formats (like `short`, `verbose`, `export`). For a complete list and details, consult the official documentation or the relevant man page (journalctl man page).

Read: Ubuntu/Debian monitoring tools guide for system administrators

8. Cleaning Up Journal Files

Storing comprehensive logs, especially with persistence enabled, can consume significant disk space over time. It’s important to know how to check and manage this usage.

To see how much disk space the journal is currently occupying, use the `–disk-usage` flag:

journalctl --disk-usage

Checking disk space used by the journal

If the journal is consuming too much space, you can manually trigger a cleanup. The `–vacuum-size` option removes older archived journal files until the total disk usage drops below the specified size:

sudo journalctl --vacuum-size=100M

(This command attempts to reduce the journal size to approximately 100 Megabytes by deleting the oldest files. You need `sudo` because it modifies system files).

Reducing journal size using –vacuum-size

After running the vacuum command, you can check the disk usage again to confirm:

journalctl --disk-usage

Disk usage after cleanup

Alternatively, you can clean up based on time using `–vacuum-time`. This option removes archived journal files containing entries older than the specified time duration. For example, to keep only entries from the last year:

sudo journalctl --vacuum-time=1years

Understanding these journald cleanup and size management commands is crucial for long-term system maintenance.

9. Limiting Journal Growth Automatically

Instead of manual cleanups, you can configure `journald` to automatically limit the amount of disk space it uses. This is done by editing the main configuration file: `/etc/systemd/journald.conf`.

Within the `[Journal]` section of this file, several parameters control size limits. Uncomment (remove the leading `#`) and set values for the ones you need:

• SystemMaxUse=: Sets the absolute maximum disk space the persistent journal can use (e.g., `SystemMaxUse=4G` for 4 Gigabytes).
• SystemKeepFree=: Specifies the amount of disk space `journald` should leave free on the partition where the persistent journal resides (e.g., `SystemKeepFree=1G`).
• SystemMaxFileSize=: Limits the maximum size an individual archived journal file can reach before being rotated.
• RuntimeMaxUse=: Sets the maximum space the journal can use in volatile storage (e.g., `/run/log/journal`).
• RuntimeKeepFree=: Specifies the space to leave free when writing to volatile storage.
• RuntimeMaxFileSize=: Limits the size of individual journal files in volatile storage before rotation.

Setting these values helps prevent the journal from consuming excessive disk space. Remember to restart the `journald` service after modifying the configuration file for changes to take effect: sudo systemctl restart systemd-journald. Configuring these limits helps in preventing journald excessive disk usage.

Conclusion

The `systemd` journal offers a robust and centralized solution for managing system and application logs on modern Linux systems. Its structured binary format, automatic metadata collection, and the powerful querying capabilities of `journalctl` make it an invaluable tool for system administrators. By understanding how to view, format, clean up, and configure the journal, you can effectively leverage this system for efficient troubleshooting and analysis, ensuring better system reliability and quicker problem resolution.

---

Frequently Asked Questions (FAQ)

Q1: What is the basic command to view systemd journal logs?

Simply run journalctl in your terminal. This will open the logs in a pager, showing the oldest entries first.

Q2: How can I set the correct timezone for my journalctl timestamps?

Use the timedatectl command. First, list available timezones with timedatectl list-timezones. Then, set your desired timezone using sudo timedatectl set-timezone . Verify with timedatectl status.

Q3: Why might my journalctl logs disappear after a reboot and how do I fix it?

By default, `journald` might use “volatile” storage (in memory). To make logs persistent, either create the directory sudo mkdir -p /var/log/journal/ or edit /etc/systemd/journald.conf and set Storage=persistent. Restart the service with sudo systemctl restart systemd-journald.

Q4: How can I check how much disk space my systemd journal logs are using?

Use the command journalctl --disk-usage. This will report the total size of the journal files on disk.

Q5: What are the commands to reduce the disk space used by journalctl logs (cleanup)?

You can use sudo journalctl --vacuum-size= (e.g., --vacuum-size=500M) to remove older files until the total size is below the target, or sudo journalctl --vacuum-time= (e.g., --vacuum-time=3months) to remove files older than the specified duration.

Q6: How can I automatically limit the maximum size of the systemd journal?

Edit the /etc/systemd/journald.conf file. Uncomment and set values for parameters like SystemMaxUse= (e.g., SystemMaxUse=2G) to define the maximum total size. Restart `journald` after saving the changes.