Modern Stream Processing Engines Compared — Kafka Streams VS Spark Structured Streaming
There is quite a bit to learn about any stream processing engine. But at a reasonably high level they actually are very similar and have lots in common. Not only do all have to offer a high-level stream processing API to describe distributed streaming dataflows, but also a low-level API for more sophisticated streaming topologies. The engines translate the dataflow description into their internal runtime representation. That’s where the differences are and where we’ll be looking at.
This talk compares two modern stream processing engines — Kafka Streams and Spark Structured Streaming. We’ll be talking about their internals and how the engines manage stateless and stateful streams. You will learn about their similarities and differences that should shed more light on the question when to use which engine.