Karafka framework 1.4.0 Release Notes (Ruby + Kafka)

This release mostly solves problems related to message deserialization and normalizes some of the naming conventions to ease during the upgrade to the upcoming 2.0 version.

Note: This release is the last release with ruby-kafka under the hood. We’ve already started the process of moving to rdkafka-ruby.

Note: If you are using Sidekiq-Backend plugin, please make sure that you’ve processed all the jobs from your Sidekiq queue before upgrading Karafka gems.

Changes (features, incompatibilities, etc)

consumer#metadata is now consumer#batch_metadata

This change is trivial: if you use batch consuming mode and you use the Consumer#metadata method, replace it with Consumer#batch_metadata.

# Karafka 1.3
class UsersConsumer < ApplicationConsumer
  def consume
    puts metadata
  end
end

# Karafka 1.4
class UsersConsumer < ApplicationConsumer
  def consume
    puts batch_metadata
  end
end

Message

Sponsored

metadata available under #metadata method

Up to version 1.3, all the message metadata would be directly available under the root scope of the params object using both direct method reference as well as with #[] accessor.

While it felt like “The Rails way”, it had several side-effects, amongst which the biggest were the need of having a hash like API, issues with accessing metadata without payload deserialization, and a lack of clear separation between payload and the metadata.

From now on, you can use the params.metadata object to fetch all the metadata.

Note: we’ve preserved the direct metadata values fetching from the params object to preserve backwards compatibility.

# 1.3
params['partition'] #=> 0
params.partition #=> 0

# 1.4
params['partition'] #=> NoMethodError (undefined method '[]')

# This will work due to backward compatibility
params.partition #=> 0

# This is the recommended way of accessing metadata
params.metadata.partition #=> 0

# This will also work as metadata is a struct now
params.metadata[:partition] #=> 0
params.metadata['partition'] #=> 0

Message metadata access allowed without message deserialization

When accessing metadata, the payload is not being deserialized until #payload method is being used.

null message support in the default JSON deserializer

When the Kafka message payload is null / nil, deserialization won’t fail. Support for it was added as some of the Karafka users use log compaction with a nil payload. In case like that, #payload will return nil.

Karafka::Params::Params no longer inherits from a Hash

Karafka::Params::Params is now just a struct. This change is introduced to normalize the setup, limit the corner cases and simplify the interface only to methods that are really needed.

Documentation

Our Wiki has been updated accordingly to the 1.4 status. Please notify us if you find any incompatibilities.

Getting started with Karafka

If you want to get started with Kafka and Karafka as fast as possible, then the best idea is to just clone our example repository:

git clone https://github.com/karafka/example-app ./example_app

then, just bundle install all the dependencies:

cd ./example_app
bundle install

and follow the instructions from the example app Wiki.

The post Karafka framework 1.4.0 Release Notes (Ruby + Kafka) appeared first on Running with Ruby.