Kafka meets Functional Programming
Having worked for various clients as a software consultant, it has always been my responsibility to write and make write good software – program that is type-driven, purely functional and maintainable. With this talk, we intend to prove that, regardless of the complexities and impurities that we have to deal when implementing a use-case, the software that we expose to clients can still be made reasonable, testable, deterministic and typesafe. This is done by taking a Kafka streaming application as the use case.
As Kafka becomes a powerful stack for various companies to solve their business use cases, I believe it is important to talk about how Scala and FP can make us write good software that works with Kafka. Hence I am titling my talk to be “Kafka meets Functional Programming”.
Starting from listing down some of the runtime complexities and consequent challenges that we faced when playing with Kafka, I will describe a part of the use case we implemented for an insurance company, where we wanted to abstract out a Kafka streams application that can work for multiple mobility datasets.
After listing down the challenges and then describing a streaming use-case, we will start with an initial version of the solution written in Scala with a few type class constraints. Then, we will list down a bunch of issues with the code, motivating us to further improve. Then we go step by step towards a final code base listing down the issues motivating us to improve the code at every stage to finally reach somewhere we see InvariantFunctors and type class constraints playing out nicely.
Next, we will focus on writing test cases for a Kafka application that involves playing around with an unsafe data type such as Avro’s GenericRecord. This part of the talk intends to prove that, “If bad things are done for reasons, make sure we do good things in the test package.” We will see some nice examples that gets us away from mutability horribleness by making use of any F that can form MonadState, reaching out to scalacheck.Gen as an instance.
We will also touch base through an upcoming plugin sbt-kafka-compose that abstracts out the vagueness and fragility of docker integration tests with Kafka. This is also a place where John’s effect rotation code using ZIO being applied !
And hopefully, I will have more things to show, such as a pure tagless final code base streamlining a Kafka application. Finally we are here to prove, how all these make our code extremely robust and safe!