Great article!
One more minor limitation — you can also have “dangling references” in S3 if you use compaction, as S3 does not know when the record gets removed.

The trivial way to solve that would be to somehow track “externally-persisted” records and periodically re-scan the Kafka broker log or fork the server to register the compacted id-s and then run a cleanup action in S3.

--

--

--

Software developer, Java mostly, C & C++ sometimes (https://www.linkedin.com/in/adam-kotwasinski/)

Love podcasts or audiobooks? Learn on the go with our new app.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Adam Kotwasinski

Adam Kotwasinski

Software developer, Java mostly, C & C++ sometimes (https://www.linkedin.com/in/adam-kotwasinski/)

More from Medium

Creativity and Waves of Emotion

Uganda diary

It’s Time to Social Distance from Social Media