Getting started with Kafka

Kafka is a fast, scalable, durable, and fault-tolerant publish-subscribe messaging system.
Kafka Fundamentals:

  • Topic is a stream of records
  • Records have a key , value and timestamp.
  • Producer API to produce a stream of records.
  •  Consumer API to consume a stream of records.

Topic is like a feedname /shopping-cart. /user-signup which producers write to and consumeres read from.
Kafka uses zookeeper to form kafka brokers into cluster. Zookeeper does coordination for kafka  consumer and kafka cluster.
Each node in kafka cluster is called a kafka broker.

Kafka Producers :
Producer sends record to topics, Producer picks which partition to send record to per topic.
Kafka Consumers :
Consumers are grouped into a consumer group, Consumer group has unique ID , Each consumer group is a subsrcriber , maintainsits offset. A record is delivered to one consumer in a consumer group

Commands to start zookeeper :

bin/zookeeper-server-start.sh config/zookeeper.properties

for kafka server :

bin/kafka-server-start.sh config/server.properties

Create a topic :

bin/kafka-topics.sh –create –zookeeper localhost:2181 –replication-factor 1 –partitions 1 –topic test
bin/kafka-topics.sh –list –zookeeper localhost:2181
Send a message (Producer ) :

bin/kafka-console-producer.sh –broker-list localhost:9092 –topic test
start a consumer:

bin/kafka-console-consumer.sh –bootstrap-server localhost:9092 –topic test –from-beginning
Source : https://kafka.apache.org/quickstart

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

w

Connecting to %s