Low latency Linux kernel for industrial embedded systems – Part II

Welcome to Part II of this three-part blog series on adopting the low latency Ubuntu kernel for your embedded systems. In case you missed it, check out Part I for a brief intro on preemptable processes in multiuser systems and memory split into kernel and user space.

The low-latency Ubuntu kernel ships with a 1000 Hz tick timer granularity (CONFIG_HZ_1000) and the maximum preemption (CONFIG_PREEMPT) available in the mainline Linux kernel. Consequently, it services most low-jitter and low-latency workloads and is a good fit for industrial embedded applications with latency requirements in the milliseconds’ range.

If the above makes perfect sense to you, stay tuned for Part III of this three-part blog series, where we will explore the considerations behind adopting low-latency Linux for your industrial embedded application. If preemption in the Linux kernel caught you off guard and you need a quick refresher, keep reading.

Preemption in industrial embedded systems

In a preemptible kernel, a process can preempt an instance of a running program, forcibly interrupting the CPU and performing a context switch. Rendering the Linux kernel preemptible essentially meant enabling the assignment of the CPU to a higher-priority task, even though the processor was already executing some kernel routine in Kernel Mode that had neither been blocked nor completed.

The challenge in making a fully preemptive kernel is knowing where preempting the current execution thread will have catastrophic consequences.  For instance, a higher-priority process may attempt to modify a shared in-kernel data structure previously accessed by a lower-priority, preempted process. Enabling context switches while executing processes in the Kernel Mode that the kernel must protect from preemption could thus result in corrupted data.

The Linux kernel implements various preemption models under its configuration options at build time. Please note that we are not concerned with external patchsets not available in the mainline source tree:

• PREEMPT_BKL: The big kernel lock (BKL), a global spinlock and one of the first preemption attempts, protected the kernel from concurrency during the move towards multiprocessor architectures [1]. It was removed with Linux v2.6.39 [2].
• PREEMPT_NONE: Preemption option optimising throughput, targeting kernels for Linux servers.
• PREEMPT_VOLUNTARY: This config option adds preemption points to the kernel, enabling voluntary interrupts of low-priority processes. By providing faster application responses with only slightly reduced throughput, CONFIG_PREEMPT_VOLUNTARY suits desktop environments.
• PREEMPT: Enabling CONFIG_PREEMPT reduces the kernel latency by making low-priority processes involuntarily preempt themselves. Preemption is disabled only at critical locations where the kernel must protect data from concurrency. Such config option fits embedded applications with latency requirements in the order of milliseconds [3].

As per the above screen capture, our machine runs the 20.04 release of Ubuntu Desktop with version 5.4 of the Linux kernel. You can see PREEMPT_VOLUNTARY is the default preemption model selected, as the kernel is for a desktop system, whereas CONFIG_PREEMPT is not set.

On the other hand, by enabling PREEMPT the low-latency Ubuntu Linux kernel is a configuration flavour of mainline with the highest preemption level available in the kernel.

Timer interrupt frequency

The timer interrupt’s frequency is another configuration worth mentioning to understand the low-latency Ubuntu Linux kernel.

The timer interrupt handler interrupts the kernel at a rate set by the HZ constant. The frequency affects the timer resolutions as a 100 Hz value for the timer granularity will yield a max resolution of 10ms (1 Hz equating to 1000ms), 250Hz will result in 4ms, and 1000Hz in the best-case resolution of 1ms  [4].

Tweaking CONFIG_HZ allows configuring the timer interrupt frequency. HZ_250 is the default config option, with 100 Hz more suited for servers [5]. The kernel of our Ubuntu Desktop has CONFIG_HZ set at 250, the preferred choice for such an environment:

The frequency of the timer interrupt in the low-latency Ubuntu kernel is 1000 Hz as systems requiring rapid responses to interrupts aim for timer resolutions of 1ms. 

Conclusion

Let’s take a moment to reflect on the learnings so far. 

Linux is a multiuser system with preemptable processes enforcing hardware protection of resources. The main obstacle to shipping a fully-preemptable kernel is specifying all the locations where preemption must not be allowed to occur. Making the Linux kernel preemptable essentially amounted to enabling high-priority programs to execute by interrupting already-running lower-priority ones, despite not having serviced their kernel requests yet.  Different preemption models are available in mainline: a kernel with CONFIG_PREEMPT enabled has the highest preemption available in the mainline Linux kernel.

When coupled with a 1000 Hz tick (CONFIG_HZ_1000), the low latency Ubuntu kernel services most low-jitter workloads and smoothly fits systems with latency requirements in the milliseconds’ range.

But what are the use cases of the low latency kernel? What are its features and what is its release schedule? Stay tuned for the final chapter of this mini-series to find out!