Jul 29, 2022Kafka onboarding primerThis article quickly summarizes what is Kafka and what features it brings. It is intended as an initial onboarding document for teams that might be interested in Kafka as their data streaming / messaging solution. Overview Kafka is a streaming solution for sending and receiving records. The records are stored…Kafka9 min readKafka9 min read
Jul 26, 2022Kafka as a remote array — binary search exampleVery often when developers come to me asking to onboard a new Kafka usecase, they believe that Kafka can immediately solve all the problems. While it is indeed an excellent project, it can also be somewhat simplified to a simple abstraction of “remote record array” with several enrichments (high performance…Kafka2 min readKafka2 min read
Jul 22, 2022Semantic versioning of shared librariesSemantic versioning allows for better communication of what kind of changes have occurred between releases. Following this framework allows developers to reduce risk when upgrading their library dependency. If we want to semantically version a shared library, we first need to document actually what does “API / functionality” mean for…Programming4 min readProgramming4 min read
Sep 13, 2021Kafka mesh filter in EnvoyFollowing up with ideas discussed in original Kafka-Envoy protocol filter issue, I finished initial development for Kafka-mesh filter in Envoy. It is now available in contrib builds. Right now, we allow downstream producers to send records to Envoy instance, which acts as a proxy for collection of Kafka clusters. …Kafka5 min readKafka5 min read
Jun 28, 2021Strategy design pattern in parameterized tests for JavaParameterized tests in Java allow us to avoid duplication when writing multiple tests. I have found them to be very useful while writing system tests for client libraries, that need to provide very similar behaviour depending while slightly different user input is provided. Parameterized tests are very expressive, we could…Java4 min readJava4 min read
Nov 24, 2020(deprecated) Optimizing KafkaMirrorMaker2 translated offset resolution for cluster failover scenariosThis article is deprecated as MM2 provides this feature as of Kafka 2.7 (KIP-545). KafkaMirrorMaker2 provides us with a tool to replicate messages from source to target cluster, as well as consumer group offsets. When it comes to consumer group offsets, KMM2 periodically translated offsets to checkpoints topic (named ${prefix}.checkpoints.internal). By…Kafka3 min readKafka3 min read
Aug 3, 2020RabbitMQ (AMQP mostly) and NATS /NATS Streaming comparisonFrom a (very) high level point-of-view, both products solve the same problem — getting our message bytes from producer to consumer, what allows our architectures to avoid coupling. Let’s take a look at are the similarities and differences provided by these two solutions. RabbitMQ server allows us to send and…Programming7 min readProgramming7 min read
Feb 20, 2020Deploying Envoy and Kafka to collect broker-level metricsEnvoy 1.13 provides Kafka broker-level filter, that allows us to collect the request/response metric for a given broker. The filter decodes the received requests/responses, and updates the correct metrics — this way we can find out how many requests were received by given broker, how many responses were sent, and…Software Development3 min readSoftware Development3 min read
