Great article!
One more minor limitation — you can also have “dangling references” in S3 if you use compaction, as S3 does not know when the record gets removed.

The trivial way to solve that would be to somehow track “externally-persisted” records and periodically re-scan the Kafka broker log or fork the server to register the compacted id-s and then run a cleanup action in S3.

--

Software developer, Java mostly, C & C++ sometimes (https://www.linkedin.com/in/adam-kotwasinski/)

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store