Diego Pacheco Tech blog

ksqlDB is an Event Streaming Database. Basically, you turn Kafka Streams(topics) into Databases. Allowing you to have some level of consistency and fast lookups on real-time data. ksqlDB allows you to perform SQL queries on your Kafka topics. It's possible todo pull-based or even push-based queries. Kafka is getting closer and closer to be a Database and that's exciting. To be 100% clear Kafka is not a full-fledge database and I would cherry-pick use cases that fit into the model. There is no one-size-fits-all. So today I want to share a simple POC in a video I made for you guys where we will see push queries in action. So Let's get started!

Kafka Streams is a Streaming library similar to Spark and Flink which works with Apache Kafka. Kafka stream can be useful to process historical and near-real-time big data workloads but also for non-realtime analytical computations at scale for the online world as well. Kafka-Streams is elastic, highly scalable, fault-tolerant, and fully integrated with Kafka. You can use Kafka Stream with Java, Scala, Kotlin JVM applications and also have exactly-once processing semantics. Kafka-Streams is being used by the New York Times, Pinterest, Line, Trivago, Zalando, and many other companies. Today I want to share a video of Kafka-Streams running with Kafka 2.6 and Java JDK 15. So Let's get started. 

There are lots of os problems or use cases that can only address properly addressed today by using streaming techniques. Streaming is needed due to the high volume of data and is almost 100% sure you will need it as your business grow up. Today I want a lecture I recorded last week about the subject. You will see a brief evolution of big data solutions from Hadoop to Modern Streaming with Spark and Flink. You also will see why streaming is needed and why we cannot use traditional solutions for it. I hope you guys like it. Let's go!

Apache Kafka  is Open Source Streaming platform for streaming and also a distributed log cluster. Kafka makes sense In lots of scenarios like Microservices, Analytics, and Machine Learning, Observability and Logs Injection, Click Stream Analytics and even for the non-traditional messaging platform. 

Kafka is the leading distributed Log OSS solution. It is great to inject large amounts of data and is used by many companies nowadays. Kafka is used in many use cases like Data Injection, Stream/Batch processing, Microservices CQRS, and ES and many others. Some years ago a showed how to get started with Kafka . Now I want to share a pretty useful and simple tool called Kaf . Kaf is written in Go language and was inspired by Docker and Kubectl. Today I will how to download and use Kaf. Have fun.

Apache Kafka is a high throughput messaging system created by LinkedIN in 2011 using Scala . Kafka is a high throughput system pretty much, in other words it means will handle the load you have, so you can use it as a buffer for back pressure or spooling mechanism. Now a days people use Kafka a lot with other Big Data Technologies like Apache Storm, Apache Hadoop or even with Apache Spark, so there is a common pattern like you read data from Kafka, process in spark/storm/hadoop and store it in the end into a NoSQL database like Cassandra for instance.  People end up using Kafka for Analytics (that's what i meant by big data), monitoring, log activity and sometimes for building block as part of bigger architectures / systems. You can see more on LinkedIN view on this post . Kafka is Durable(can be persisted on DISK) and very FAST and can scale great deal of loads like 800 billions messages per day at LinkedIN.  Besides LinkedIN, Twitter, Netflix, Spotify, Mozilla and other...