Getting started with Kafka

Kafka is a fast, scalable, durable, and fault-tolerant publish-subscribe messaging system.
Kafka Fundamentals:

  • Topic is a stream of records
  • Records have a key , value and timestamp.
  • Producer API to produce a stream of records.
  •  Consumer API to consume a stream of records.

Topic is like a feedname /shopping-cart. /user-signup which producers write to and consumeres read from.
Kafka uses zookeeper to form kafka brokers into cluster. Zookeeper does coordination for kafka  consumer and kafka cluster.
Each node in kafka cluster is called a kafka broker.

Kafka Producers :
Producer sends record to topics, Producer picks which partition to send record to per topic.
Kafka Consumers :
Consumers are grouped into a consumer group, Consumer group has unique ID , Each consumer group is a subsrcriber , maintainsits offset. A record is delivered to one consumer in a consumer group

Commands to start zookeeper :

bin/ config/

for kafka server :

bin/ config/

Create a topic :

bin/ –create –zookeeper localhost:2181 –replication-factor 1 –partitions 1 –topic test
bin/ –list –zookeeper localhost:2181
Send a message (Producer ) :

bin/ –broker-list localhost:9092 –topic test
start a consumer:

bin/ –bootstrap-server localhost:9092 –topic test –from-beginning
Source :

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s