Kafka is a fast, scalable, durable, and fault-tolerant publish-subscribe messaging system.
Kafka Fundamentals:
- Topic is a stream of records
- Records have a key , value and timestamp.
- Producer API to produce a stream of records.
- Consumer API to consume a stream of records.
Topic is like a feedname /shopping-cart. /user-signup which producers write to and consumeres read from.
Kafka uses zookeeper to form kafka brokers into cluster. Zookeeper does coordination for kafka consumer and kafka cluster.
Each node in kafka cluster is called a kafka broker.
Kafka Producers :
Producer sends record to topics, Producer picks which partition to send record to per topic.
Kafka Consumers :
Consumers are grouped into a consumer group, Consumer group has unique ID , Each consumer group is a subsrcriber , maintainsits offset. A record is delivered to one consumer in a consumer group
Commands to start zookeeper :
bin/zookeeper-server-start.sh config/zookeeper.properties
for kafka server :
bin/kafka-server-start.sh config/server.properties
Create a topic :
bin/kafka-topics.sh –create –zookeeper localhost:2181 –replication-factor 1 –partitions 1 –topic test
bin/kafka-topics.sh –list –zookeeper localhost:2181
Send a message (Producer ) :
bin/kafka-console-producer.sh –broker-list localhost:9092 –topic test
start a consumer:
bin/kafka-console-consumer.sh –bootstrap-server localhost:9092 –topic test –from-beginning
Source : https://kafka.apache.org/quickstart
