Low latency Linux for industrial embedded systems – Part III

Welcome to the concluding chapter of this three-part blog series on the low latency Ubuntu kernel for industrial embedded systems.  

Each blog is standalone and can be read independently from the others, although you may want to start at the beginning for some continuity. If you need a quick refresher on userland and kernel space, we recommend you check Part I out first. In Part II, we looked at preemption in Linux, the timer interrupt frequency, and the config options adopted by the low latency Ubuntu kernel.

If you are a bit rusty with these concepts, going back to Part I and Part II may help to refresh your memory. Otherwise, keep moving as you are almost at the end of your low latency journey!

Latency in industrial embedded systems

The low latency Ubuntu kernel is suitable for low-jitter workloads and industrial embedded applications with latency requirements in the milliseconds’ range. Latency in an embedded system refers to the time elapsed when servicing a request, whereas jitter is the variability in the timing of an expected signal. Low jitter corresponds to reduced variability and undetectable variations in the kernel responsiveness when servicing sensitive applications [1].

[2]

How does the low latency Ubuntu kernel meet such stringent requirements? For that, let’s quickly revisit the concepts delved at greater length in Part I and Part II of this series.

A program usually executes in User Mode in userland and activates a kernel routine when requesting one of its services. Once the Linux kernel has serviced the request, it resumes execution in the user address space. Linux introduced kernel preemption in v2.5.4, enabling high-priority tasks to preempt running processes and issue system calls asking to switch to the kernel address space. Hence, in a preemptible kernel, a process can preempt an instance of a running program, forcibly interrupting the CPU and performing a context switch. 

The challenge in making a fully preemptive Linux kernel is detecting where preemption must not be allowed to occur, lest the interrupted execution thread results in catastrophic consequences.

Of the available preemption models available upstream (PREEMPT_NONE, PREEMPT_VOLUNTARY, and PREEMPT), the low latency Ubuntu kernel has the maximum preemption in mainline. By enabling CONFIG_PREEMPT, the low latency Ubuntu disables preemption only at critical locations, where the kernel must protect the data from concurrent access. Besides a few other minor config changes, the low latency kernel services most workloads when coupled with a 1000 Hz (CONFIG_HZ_1000) timer interrupt frequency.

Low latency or real time?

A kernel with the maximum preemption available in mainline (PREEMPT), coupled with four times the timer granularity of a generic kernel (HZ_1000 vs HZ_250), invariably calls for comparisons with real-time alternatives. 

PREEMPT_RT, slowly being mainlined, is arguably the preferred choice of Linux developers working on embedded systems requiring real-time capabilities. The latency requirements and the consequences of missing a deadline dictate whether a system needs PREEMPT_RT or not. To make an informed decision, one must look at the data. If the latency demands are particularly tight and the consequences of a missed deadline are catastrophic, PREEMPT_RT is likely a good choice. The low latency Ubuntu kernel is instead a great candidate if the system has latency requirements in the order of milliseconds. Low latency can also suit your application if missing a deadline results in system degradation but is not fatal, or even if the hardware handles most of the real-time performance.

While low latency is not as firm as a full-blown PREEMPT_RT integration, it is close and does not have the same compatibility issues. CONFIG_PREEMPT uses entirely upstream code and is a configuration of mainline, with the kernel working as intended. On the other hand, PREEMPT_RT is an intrusive patchset that may not be compatible with all required drivers and may require debugging/reworking.

Another aspect to consider is that the kernel must not consume the entirety of a system’s resources to bring an acceptable return on investment. While preempt-full Linux kernels provide real-time performance, this may come with an operational cost, in that a real-time OS minimises response latency to events, not optimized throughput. On the other hand, the low latency Ubuntu kernel enables a well-balanced solution reducing overhead while maintaining responsiveness. Furthermore, the low-latency Ubuntu kernel is more favourable for throughput and user space CPU access than real-time alternatives [1].

Overall, unless it does not meet an exact preemption specification, the low-latency kernel fits well low-jitter, low-latency, soft “real-time” workloads. Dedicated devices (e.g.,life-supporting medical equipment) with catastrophic consequences in case of failure are instead better suited to a fully-preemptive solution than a balanced one.

If you are ready for a more in-depth technical assessment, check this specialised report comparing the Ubuntu low-latency kernel and an Ubuntu Linux kernel patched to be fully preemptive.

The low latency Ubuntu release cadence

The low latency Ubuntu kernel is currently available on AMD64, with the kernel experts from Canonical ready to target more architectures based on demand. 

Canonical releases the low latency alongside the standard kernel, following the same cadence as the generic Ubuntu release. As a developer, you can access the same set of features, resiliency, and security with the low latency kernel that you have come to expect from Ubuntu. 

Canonical publishes new versions of Ubuntu with a regular cadence, allowing developers regular secure access to the latest open source upstream capabilities. Each Ubuntu release is associated with its delivery date and codename (e.g. Ubuntu ‘Focal Fossa’ 20.04 in April 2020). 

LTS or Long Term Support releases are enterprise-grade versions of Ubuntu, published in April every two years. Ubuntu LTS releases receive an initial maintenance period and optional five years of Extended Security Maintenance (ESM). Enterprises and the community can access the entire lifecycle via an Ubuntu Advantage subscription or a free personal subscription. You can contact us for more information.

Canonical also ships production-ready Ubuntu every six months between LTS versions. These interim releases come with nine months of support, allowing users to run the latest kernels, libraries, and software versions.

As a final note, the low latency Ubuntu kernel also ships with the same main kernel: v5.15 for Jammy Jellyfish, v5.13 for Impish Indri, v5.4 for Focal Fossa and so on.

Conclusion

Real-time from a kernel perspective is a measure of latency from an interrupt request to a task entry. Real-time from a kernel perspective is a measure of latency from an interrupt request to a task entry. Intuitively, such delay affects the overall performance of a computer system, usually measured in terms of latency or throughput. Traditionally, Linux was designed for maximum throughput, whereas a real-time operating system (RTOS) attempts to minimise latency. 

The low latency Ubuntu kernel also minimises latency as much as possible, without relying on the more intrusive PREEMPT_RT patch. The low latency Ubuntu kernel has the maximum preemption currently available in mainline (PREEMPT), coupled with four times the timer granularity of the generic Ubuntu kernel (HZ_1000 for low latency vs HZ_250 for generic). It is a “soft” real-time kernel with fully upstream code. Compared to real-time alternatives, it results in a lower system overhead, is less costly to maintain, more favourable for higher throughput, and available CPU for user space processes.

Whether a real-time or low latency kernel better serves your application, depends on its latency requirements. Let’s work together on your use case today.