MS Lesson23: Apache Kafka

                                                            Fundamental bilikler

 

1. Broker - Kafka-nin ishlediyi server prosesidir. Broker poct merkezi kimidir. Broker mesajlari qebul edir, mesajlari diske yazir, consumerler istedikde gonderir. 

Real production-da hecvaxt 1 broker olmur, bir nece broker birlikde Cluster-i teshkil edir. 
Bir nece brokerin olmasi:
* Fault Tolerance : 1 broker cokse digerleri ishleyir
* Scalability : yuk bolunur, performans artir
* Replication : data kopyalanir, itmir

Her brokerin unikal ID-si var. Cluster-de diger brokerlerler emekdashliq edir. 






2. Topic - Kafka-da melumatlarin saxlandigi mentiqi kanal (log)dir. 
Poct merkezinde (Broker) mektublar kateqoriyalara bolunur. 

Topic-in xususiyyetleri:
* Unikal adi olur
* Partitions : nece hisseye bolunur (default 1)
* Replication factor : nece broker-de kopyalanir
* Retention period : melumat nece muddet saxlanir (default 7 gun)


Retention vacib meqamlardan biridir. Kafka diger message brokerlerden (RabbitMQ kimi) bu cehetden ferqlenir.

RabbitMQ: consumer mesaji oxudu -> mesaj silinir
Kafka: consumer mesaji oxudu -> mesaj qalir

Bu ne demekdir? 
Eyni melumati 10 ferqli Consumer oxuya biler.
Yeni bir servis qoshulsa, kecmish melumatlarida oxuya biler


3. Partition - bunu supermarketin kassasi kimi fikirleshek, 1 kassada 1000 nefer novbede gozlese bu zaman sistem cox yavash ishleyer, lakin 10 kassa olsa 1000 neferi 10 bour ve suret artir. 

Topic-i bir nece fiziki hisseye bolursen - her bir hisse musteqil ishleyir. 


Her partition:
* Musteqil bir log faylidir - diskde ayrica saxlanilir
* Ordered - oz icindeki mesajlar ardicildir
* Immutable - yazilan mesaj deyishdirilmir


Bes neye gore partition lazimdir?

* Paralellik (Parallelism)
1 Partition - yalniz 1 Consumer oxuya biler
3 Partition - 3 Consumer parallel oxuya biler.


* Scalibility 
Bugun: 10000 mesaj/saniye  -> 3 partition kifayetdir
Sabah: 100000 mesaj/saniye  -> 30 partition elave et


* Fault tolerance
Her partition ferqli brokerde replikasiya olunur:

Partition 0 → Leader: Broker 1, Replica: Broker 2 Partition 1 → Leader: Broker 2, Replica: Broker 3 Partition 2 → Leader: Broker 3, Replica: Broker 1 Broker 1 çökdü? → Broker 2 dərhal LEADER olur ✅ Data itirilmir!


Bes mesajlar hansi partition-a dushur?

// 3 ssenari var: // 1. Key yoxdur → Round Robin (növbəli bölünmə) producer.send("order-created", message); // msg1 → P0, msg2 → P1, msg3 → P2, msg4 → P0 ... // 2. Key var → eyni key həmişə eyni partition-a düşür producer.send("order-created", "userId-123", message); // "userId-123" olan BÜTÜN mesajlar → həmişə P1-ə // 3. Manual → sən özün seçirsən producer.send("order-created", 2, message); // Bu mesaj → P2-yə


Sual: Niyə eyni key-i eyni partition-a göndəririk? Cavab: ORDERING (sıra) zəmanəti üçün! ❌ Key olmadan: userId-123 → "sifariş verildi" → Partition 0 userId-123 → "ödəniş edildi" → Partition 2 userId-123 → "çatdırıldı" → Partition 1 Consumer bu 3 mesajı FƏRQLI sırada oxuya bilər! ❌ ✅ Key ilə (userId-123): userId-123 → "sifariş verildi" → Partition 1 (offset 0) userId-123 → "ödəniş edildi" → Partition 1 (offset 1) userId-123 → "çatdırıldı" → Partition 1 (offset 2) Consumer həmişə DÜZGÜN sırada oxuyur! ✅


Architect qaydası: Partition sayı = max(gözlənilən Consumer sayı, throughput tələbi) Məsələn: - 6 Consumer paralel işləyəcək → minimum 6 partition - Gələcəkdə böyüyəcək → 12 partition qoy (2x ehtiyat) ⚠️ Partition sayı artırıla bilər ⚠️ Amma AZALDILA BİLMƏZ — bunu həmişə yadda saxla! ⚠️ Çox partition da problem yaradır (metadata yükü)







                                                                    Docker compose yaml fayli


services:

  kafka-1:

    image: apache/kafka:3.9.0

    container_name: kafka-1

    ports:

      - "9092:9092"

      - "29092:29092"

    environment:

      KAFKA_NODE_ID: 1

      CLUSTER_ID: 97exVv29T2icAhT2SNEEDQ

      KAFKA_PROCESS_ROLES: broker,controller

      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka-1:9091,2@kafka-2:9091,3@kafka-3:9091

      KAFKA_LISTENERS: PLAINTEXT://:9090,CONTROLLER://:9091,EXTERNAL://:9092,EXTERNAL_CLIENT://:29092

      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka-1:9090,EXTERNAL://kafka-1:9092,EXTERNAL_CLIENT://localhost:29092

      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,EXTERNAL:PLAINTEXT,PLAINTEXT:PLAINTEXT,EXTERNAL_CLIENT:PLAINTEXT

      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER

      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT

      KAFKA_LOG_DIRS: /var/lib/kafka/data

    healthcheck:

      test: ["CMD", "/opt/kafka/bin/kafka-topics.sh", "--bootstrap-server", "localhost:9090", "--list"]

      interval: 10s

      timeout: 5s

      retries: 5

      start_period: 30s

    volumes:

      - ./volumes/kafka/server-1:/var/lib/kafka/data


  kafka-2:

    image: apache/kafka:3.9.0

    container_name: kafka-2

    ports:

      - "9094:9094"

      - "29094:29094"

    environment:

      KAFKA_NODE_ID: 2

      CLUSTER_ID: 97exVv29T2icAhT2SNEEDQ

      KAFKA_PROCESS_ROLES: broker,controller

      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka-1:9091,2@kafka-2:9091,3@kafka-3:9091

      KAFKA_LISTENERS: PLAINTEXT://:9090,CONTROLLER://:9091,EXTERNAL://:9094,EXTERNAL_CLIENT://:29094

      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka-2:9090,EXTERNAL://kafka-2:9094,EXTERNAL_CLIENT://localhost:29094

      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,EXTERNAL:PLAINTEXT,PLAINTEXT:PLAINTEXT,EXTERNAL_CLIENT:PLAINTEXT

      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER

      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT

      KAFKA_LOG_DIRS: /var/lib/kafka/data

    healthcheck:

      test: ["CMD", "/opt/kafka/bin/kafka-topics.sh", "--bootstrap-server", "localhost:9090", "--list"]

      interval: 10s

      timeout: 5s

      retries: 5

      start_period: 30s

    volumes:

      - ./volumes/kafka/server-2:/var/lib/kafka/data


  kafka-3:

    image: apache/kafka:3.9.0

    container_name: kafka-3

    ports:

      - "9096:9096"

      - "29096:29096"

    environment:

      KAFKA_NODE_ID: 3

      CLUSTER_ID: 97exVv29T2icAhT2SNEEDQ

      KAFKA_PROCESS_ROLES: broker,controller

      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka-1:9091,2@kafka-2:9091,3@kafka-3:9091

      KAFKA_LISTENERS: PLAINTEXT://:9090,CONTROLLER://:9091,EXTERNAL://:9096,EXTERNAL_CLIENT://:29096

      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka-3:9090,EXTERNAL://kafka-3:9096,EXTERNAL_CLIENT://localhost:29096

      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,EXTERNAL:PLAINTEXT,PLAINTEXT:PLAINTEXT,EXTERNAL_CLIENT:PLAINTEXT

      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER

      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT

      KAFKA_LOG_DIRS: /var/lib/kafka/data

    healthcheck:

      test: ["CMD", "/opt/kafka/bin/kafka-topics.sh", "--bootstrap-server", "localhost:9090", "--list"]

      interval: 10s

      timeout: 5s

      retries: 5

      start_period: 30s

    volumes:

      - ./volumes/kafka/server-3:/var/lib/kafka/data


  kafka-ui:

    image: provectuslabs/kafka-ui:v0.7.2

    container_name: kafka-ui

    ports:

      - "9999:8080"

    environment:

      KAFKA_CLUSTERS_0_NAME: local

      KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS: kafka-1:9090,kafka-2:9090,kafka-3:9090

    depends_on:

      kafka-1:

        condition: service_healthy

      kafka-2:

        condition: service_healthy

      kafka-3:

        condition: service_healthy



Konteynerin daxiline girmek ucun: docker exec -it kafka-1 bash
















































Комментарии

Отправить комментарий

Популярные сообщения из этого блога

Interview questions

Изображение
+1. JVM memory areas. Thread safe memory areas, Garbage collector working principle, Java reference types +2. SOLID +3. Collection API: How does HashMap work under the hood, hash collision, Concurrent HashMap +4. Equals and HashCode contract -5. Thread, Volatile vs Synchronized, atomic variables(CAS-compare and swap). How to prevent deadlock. Deadlock vs livelock. Reentrant lock. +6. Error vs Exception +7. Immutable class(how to make object properties immutable, immutable collections) +8. DB index type(clustered, non clustered, composite) +9. DB normalization and denormalization +10. RDBMS vs non RDBMS +11. ACID vs BASE. Lock mechanism +12. Hash eviction policy(TTL, LRU, FIFO, LFU, MRU, random replacement, ARC). Problem: we have 2GB memory, how to store 15GB data in this cache +13. entity lifecycle +14. dirty check, flush +15. cascade types +16. n+1 problem (entity graph, join fetch) +17. lazy initialization exception 18. idempotency, safe in rest methods +19. IOC, DI, Proxy +20. AOP(c...
Далее...

Lesson1: JDK, JVM, JRE

Изображение
1. Compiler - takes the entire programming code written in high language and translates the whole of it into machine code at once. After this process the an executable file will be ready to run. The translation takes some times to be done, but after that execution is so fast. 2. Interpreter - takes each single instructions of the code translates it and then executes it on its own. The translation step is before the program runs is fast, but execution is slow. JDK = JRE + development tools JRE = JVM + library class JDK -> Java Development Kit. A kit which provides the environment to develop and execute (run) Java program. JRE -> Java Runtime Environment. Provides environment to only run. JVM -> Java Virtual Machine f * Compiled: Fortran, C, C++, Pascal, Haskell, Rust, Go * Interpreted: JavaScript, PHP, Python, Ruby Java both compiled and interpreted language. Terminalda : javac Main.java yazdiqda, source fayli(.java) bytecode(.class) faylina cevirir. Terminalda: java Main (bu z...
Далее...

Lesson_2: Operations in Java

1. Capda toplama emeliyyatinin ferqi: package org.example.test ; public class Test { public static void main (String[] args) throws Exception { int one = 1 ; int two = 2 ; System. out .println( "result = " + one + two) ; System. out .println(one + two + " result" ) ; } }   char a = 'a' ; char b = 'b' ; System. out .println(a + b) ; System. out .println( 'a' + 'b' ) ; System. out .println( "" + a + b) ; System. out .println( "" + 'a' + 'b' ) ; 2. float ve double-da bash veren riyazi emaliyyat xeyalari: package org.example.test ; public class Test { public static void main (String[] args) throws Exception { float f = 0.6F ; System. out .println(f + 0.1 ) ; } } Operators in Java Operators in Java can be classified into 6 types: Arithmetic Operators Assignment Operators Relational Operators Logical Operators Unary Operators Bitwise Operator...
Далее...