Kafka Role in Microservices Solutions

Apache Kafka is Open Source Streaming platform for streaming and also a distributed log cluster. Kafka makes sense In lots of scenarios like Microservices, Analytics, and Machine Learning, Observability and Logs Injection, Click Stream Analytics and even for the non-traditional messaging platform. 


Kafka: Eventual Consistency, Stream and Beyond

The main story about Kafka could easily be sell out as a messaging solution for eventual consistent microservices. Microservices should have isolation as one of the main properties, which means, they should own they databases and share not DBs which external consumers. Having said that, there is a need for eventual consistency "bus" let's say because you can't just do a single database query and answer any question you need. It's possible that applications do the famous, client-side join, however this could easily become an anti-pattern depending on the data you are reading(Scan multiple DB partitions). 

So there are a couple of solutions that Kafka enables, First of all, you could easily use Kinises todo the same job, however, the main issue would be that you will be coupled to AWS and won't be able to port your data workloads to other clouds if needed.  Having Kafka means that we could use any stream-based solutions like Spark, Flink or even Kafka-Streams in order to process the data in near-real-time or even in batch.

Kafka is great because it allows us to use the same architecture and tools todo Batch and Streaming(Kappa Architecture). So not only you have a solution that works great for your microservices eventual consistency but also you have a solution with is great for analytics.

Kafka could easily be used as a source of truth. Like AWS Aurora does "The Log is the Database", this means applications could store raw events in Kafka and use Kafka features like log compaction in order to remove duplicates and old data(save space). These patterns enable polyglot processing and easily the application could use any solution that makes sense like Redis for Caching, Elastic Search for Full-Text Search, JanusGraph for Graph, Cassandra for TimeSeries and so on and on.

Kafka performance & Scalability

According to LinkedIn benchmarks can easily do 2M writes per second on commodity hardware. A single producer thread could produce +800K records per second all that with median latency around 2ms and p99 with 14ms. Linkedin Numbers from 2019 are supper impressive 7 Trillion messages per day. Kafka's performance does not degrade based on the size of your disk but based on the size of the message, as the bigger the message is, as poor the performance is.

Kafka is not a Database

Via Kafka Streams, you can have "views" of the data. However, Kafka(the log) wants a guarantee data integrity and thats is something the application needs to work on it. Also, there is will a replication lag, especially if we are talking multi-region, however no matter the solution you do, we are talking about physics so you will have replication lag anyway.

Amazon Kinesis Comparison & Multi-region Reliability 

Kafka is super easy to use however is not that easy to operate under scale, especially if there is multi-region involved where you might have issues like replication-lag(Mirror Maker), recently Spotify migrated out Kafka due replication issues using mirror maker and also reliability. Kafka has 3x low latency than AWS Kinesis, However, Kafka could get hard to operate if you using as a source of truth in a multi-region scenario like Spotify describe it. In order to improve fix issues, you could easily use Confluent Cloud or Even AWS MSK which is Kafka managed service.

Kafka: Great for Data Ingestion

Kafka is pretty good for big data because of his ability to ingest big volumes of data, thats a pattern not only for analytics but for other infrastructures solutions like log ingestion(in the observability realm), Click Stream data ingestion(for business analysis, data trends, strategic analysis, and user profile understanding).

Cheers,
Diego Pacheco


Popular posts from this blog

C Unit Testing with Check

Image
When I start on IT, several years ago(+15), C was the first language that I learned. But was not the first language I mastered and deployed in production which was...surprisingly Visual Basic 6.0. However, when I learned C or any language back on that old and arcane time noddy was into testing. However now is impossible to work professionally in any language without tests. C is a low-level language but still kicks ass and has several interesting libs/frameworks. People use to fear C a lot because of lack of syntactical sugar but once you get used to it is not that hard at all. Today I want to show how you can do Unit Testing with C. I will not show how to do Unit Testing with C++ but with C ANSI actually. We will be using Check a unit test lib for C ANSI.
read more »

Having fun with Zig Language

Image
Zig is a general-purpose language created in 2016 by Andrew Kelly .  Zig aims to fix the issues C language has while being pragmatic and simple at the same time. Zig really shines as being simple by not having any hidden control flows, not having hidden memory allocations, no pre-processors, and no macros. Basically, it's one language and one language only. Since pre-processors and macros can get really complex and become their own universe. Sure it's not the Zig Philosophy.  Zig Compiler(Zig cc) also is a C compiler, and the interoperability from C to Zig is very good. Clearly, we can see influence from other languages like Go and Rust; however, Zig is pretty unique and different from Go and Rust.  Zig is fast. Zig is perfect for system programming. I bet we will see a lot of solutions being built in Zig. As a languages aficionado, I like to learn at least one language per year and sometimes more; Zig was my 2021 pick, did not regret it. Even if you dont use all language...
Keep reading

HMAC in Java

Image
In 2018 AWS CTO(Vogels) said : "Security is everyone's job". For the last 2 years, there was so many famous and big data leaks and breaks that made that statement be very true more than ever. Security often could mean worst performance and worst user experience so in order to get it right you really need to think about the designs before jumping into to code and consider performance and user experience has main requirements.  I was thinking about writing about how to do HMAC in Java for a while and recently Redis 6.0.0 come out and to my surprise, there was a refactoring on the password part in order to use HMAC.  Security easily could scare engineers often because it is not something well spread yet but I believe this will change soon. It does not matter if you have to deal with PII Data or not, security is super relevant because everybody is running their workloads at the cloud or with IoT and Edge devices which means more distribution, more code, more points of fail...
read more »