Deploying Envoy and Kafka to collect broker-level metrics

  • routing all Kafka-related traffic through Envoy (including internal cluster communication),
  • routing only Kafka-client traffic through Envoy.
configuration for Envoy
configuration for Kafka broker 1
configuration for Kafka broker 2
kafka.broker1.request.update_metadata_request: 2
kafka.broker1.response.update_metadata_response: 2
kafka.broker2.request.update_metadata_request: 1
kafka.broker2.response.update_metadata_response: 1
configuration for Kafka broker 1
configuration for Kafka broker 2
# create topic 'mytesttopic' and make sure it is replicated:
bin/kafka-topics.sh \
--bootstrap-server localhost:19092,localhost:19093 \
--create \
--topic mytesttopic \
--replication-factor 2 \
--partitions 20
# start the consumer:
bin/kafka-console-consumer.sh \
--bootstrap-server localhost:19092,localhost:19093 \
--topic mytesttopic
# caused by `kafka-topics` invocation
kafka.broker1.request.create_topics_request: 1
# constantly increasing while consumer is alive
kafka.broker1.request.fetch_request: 107
kafka.broker1.response.fetch_response: 107
kafka.broker2.request.fetch_request: 107
kafka.broker2.response.fetch_response: 107

--

--

--

Software developer, Java mostly, C & C++ sometimes (https://www.linkedin.com/in/adam-kotwasinski/)

Love podcasts or audiobooks? Learn on the go with our new app.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Adam Kotwasinski

Adam Kotwasinski

Software developer, Java mostly, C & C++ sometimes (https://www.linkedin.com/in/adam-kotwasinski/)

More from Medium

Using Search Template — ElasticSearch

Kubernetes: Horizontal Pod Autoscaling (HPA)

Last Mile Security : A way to secure backend APIs

Microservices Integration Test under Cloud Native Architecture