3 Reasons Why Apache Kafka is Powering the Data-driven Business

Companies of all sizes strive to explore the business value of their own data to keep competitive. The so-called data-driven organizations make better decisions by collecting and making all kinds of data immediately and securely available so it can be processed and analyzed throughout the enterprise. Until recently, collecting large amounts of data (in order of several million messages/sec) in real time at a single platform was not possible. Apache Kafka is an open source, highly-available, cloud-native, horizontally-scalable, real-time streaming platform for the enterprise, that solves this problem. While developers love its elegant design, APIs, toolset, and support, business treasures the possibilities opened by the ability to collect and analyze large amounts of data in real time. In this blog article, we highlight three reasons why enterprises are adopting Kafka as a streaming platform on their journey towards being more data driven.

1. Kafka is (also) a highly scalable and secure messaging system

At its core, Kafka is a pub/sub messaging system that scales very well. It scales constantly to the number of producers and consumers, and new servers added increase the capacity almost linearly. Its security features support encryption of data both in transit and at rest, and enforcement of access control policies. Thus, it can be used as a central messaging platform for the whole enterprise. These two characteristics (scalability and security) allow Kafka to replace legacy MQ systems that are costly to maintain and don’t scale, used to decouple components and buffer messages. Two systems integrated with Kafka. Then three or four. Then really complex.

2. Break silos and integrate (legacy and modern) systems with Kafka

As business grows, technologies evolve and events like acquisitions occur, businesses must adapt quickly and at low cost. Thus, they need way to rescue data off of accumulated silos, including legacy systems, in a cloud native, microservice-based fashion. Kafka Connect allows data from a wide range of systems to be transferred from and into Kafka. The supported systems range from relational databases, messaging systems, to cloud and mobile apps, supporting both batch and real time data integration modes. This can be used to make existing assets available to newer, cloud-native microservices without changing existing core systems. This is important in the process of chopping large, monolithic applications into smaller, agile parts. While streaming data into Kafka can be in many cases achieved merely configuratively with Kafka Connect, new connectors can be developed, since it’s relatively easily due to its modern architecture, documentation, and availability of open-source connectors. Examples: Stream data from legacy relational-db system to cloud-based. Then publish api with some of the data. Website activity tracking: Publish site activity (page views, searches, etc) as event streams to feed monitoring systems and data warehouses. Log aggregator: Ship application logs from many source systems to log analysis tools like Splunk, Elastic/Kibana. IoT: Collect events from a wide range of devices.

3. Analyze data within or outside Kafka

The data-driven business is only possible when data is not only collected and made available through the enterprise, but analyzed too. Data that flows through Kafka can be analyzed by both Kafka tools, and external ones. The Kafka Streams API is a Java library targeted at experienced developers. KSQL is a Kafka tools aimed towards data analysts that can use an expression language to build streams. Write about exactly-once and other advantages of Kafka native tools. Since Kafka is an integration platform, it is also dead simple to stream the data to another system that might be already established in an enterprise or are good at solving specific problems, like Spark, Hadoop, Elastic, Splunk, Mongodb, Cassandra, and others.

Conclusion

Kafka is an important component in the data driven company because it solves the problem of making data securely available in real time to the entire company. It allows business teams to have access to data previously in silos and analyze them to generate data-driven insights. Also, software components can be less coupled to increase the speed of innovation without impacting current proven and operational systems, that are hard to change.