Monitoring Karafka Jobs Progress Using Web UI

Introduction

Karafka is a Ruby and Rails framework designed to simplify processing messages consumed from Apache Kafka.

One of Karafka’s components is the https://karafka.io/docs/Web-UI-Getting-Started/. It provides a convenient way for developers to monitor and manage their Karafka-based applications without using the command line or third-party software.

The interface, amongst others, includes:

  • historical metrics,
  • real-time aggregated metrics,
  • real-time information on resource usage,
  • errors details,
  • performance statistics,
  • stale partitions detection (LSO hangs),
  • routing pattern matched topics subscriptions.


It bridges the technical workings of Karafka and the humans overseeing them, ensuring smoother operations, quicker troubleshooting, and enhanced overall visibility.

Why
Sponsored
Track Job Progress?

Monitoring Karafka’s job progress can be crucial, especially for longer tasks. Here’s why:

  • Extended Jobs: Some jobs naturally take longer due to the data they handle. Monitoring helps differentiate between a naturally long job and one facing issues,

  • Stuck Jobs: Jobs that hang or get stuck can go unnoticed without monitoring. This wastes resources and can slow down the entire system,

  • Batch Processing: Karafka often works on batches of messages, processing each in sequence. Keeping track ensures no single message causes a hold-up, controlling the flow smoothly.

In short, monitoring Karafka’s jobs helps keep things efficient, timely, and problem-free.

Sponsored

Implementing Progress Notifications

Karafka Web UI supports process and consumer tagging. Tags can be used to add additional information about consumers and their execution, and Karafka processes themselves.

Adding progress monitoring with tags is super easy. All you need to do is to tag progress inside the consumer:

class EventsConsumer 

In case you want to abstract that away, you can always create a simple custom iterator:

class EventsConsumer 

Once your code is in place, there is nothing more you need to do. Karafka Web UI Tagging API will do the rest.

This is how it will look in the Web UI:

Conclusion

Keeping track of Karafka's job progress is key. Some jobs take longer, but knowing if they are progressing is essential. The Tagging API helps with this by making it easy to see job details in the Web UI. This allows for quick checks on job status and ensures everything runs smoothly. With the Tagging API and Web UI combined, managing and overseeing Karafka jobs becomes more straightforward and efficient.

The post Monitoring Karafka Jobs Progress Using Web UI appeared first on Closer to Code.

Ubuntu Server Admin

Recent Posts

How to Install nvidia-smi on Ubuntu or Debian Linux

In this article, we will see how to install nvidia-smi on Ubuntu or Debian Linux.…

9 hours ago

How to Install clang tool on Ubuntu or Debian Linux

In this article, we will see how to install clang tool on Ubuntu or Debian…

1 day ago

How to resolve Ubuntu 20.04 Container Signature Errors on Raspberry Pi ARM Devices

When working with Docker containers on Raspberry Pi devices, you might encounter frustrating signature verification…

2 days ago

How to fix DNS Resolution Issues with OpenVPN on Ubuntu 18.04

You’ve recently upgraded to Ubuntu 18.04 and found that your OpenVPN connection no longer resolves…

2 days ago

How to Fix Ubuntu 18.04 System Monitor Launch Issues

Have you ever tried to open System Monitor on your Ubuntu 18.04 system only to…

3 days ago

What is System Hardening? Essential Checklists from OS to Applications

System hardening means locking down a system and reducing its attack surface: removing unnecessary software…

3 days ago