Cp-ansible: confluent.variables_handlers ์—ญํ• ์˜ ๋ชฉ์ 

์— ๋งŒ๋“  2020๋…„ 04์›” 07์ผ  ยท  9์ฝ”๋ฉ˜ํŠธ  ยท  ์ถœ์ฒ˜: confluentinc/cp-ansible

์ด repo์˜ 5.4.1-post ๋ถ„๊ธฐ๋ฅผ ์‚ฌ์šฉํ•˜์—ฌ ํ•ฉ๋ฅ˜ Kafka์™€ ์‚ฌ์œก์‚ฌ๋ฅผ ์„ค์น˜ํ•˜๋ ค๊ณ  ํ•ฉ๋‹ˆ๋‹ค.
confluent.variables_handlers๋ฅผ confluent.zookeeper ๋ฐ confluent.kafka ์—ญํ• ์—์„œ ์ข…์† ์—ญํ• ๋กœ ์‚ฌ์šฉํ•˜๋Š” ๋ชฉ์ ์€ ๋ฌด์—‡์ž…๋‹ˆ๊นŒ? ์ด ๋ฌด์‹œ confluent.zookeeper / ๊ธฐ๋ณธ๊ฐ’ / main.yaml์—์„œ ์‚ฌ์œก์‚ฌ ๋ณ€์ˆ˜ ์œ ์‚ฌ kafka_broker์—์„œ confluent.kafka /defaults/main.yaml. kafka๋ฅผ ์„ค์น˜ํ•  ๋•Œ ์ด ์˜ค๋ฅ˜๊ฐ€ ๋ฐœ์ƒํ•ฉ๋‹ˆ๋‹ค.

TASK [confluent.kafka_broker : Set Permissions on Data Dirs] ********
 FAILED! => {"msg": "'dict object' has no attribute 'datadir'"}
help wanted

๊ฐ€์žฅ ์œ ์šฉํ•œ ๋Œ“๊ธ€

์‹ค์ œ๋กœ ์‚ฌ์ „์„ ๋ณ‘ํ•ฉํ•ด์•ผ ํ•ฉ๋‹ˆ๋‹ค. cp-ansible์„ ์‹คํ–‰ํ•  ๋•Œ ํ˜„์žฌ ๋””๋ ‰ํ† ๋ฆฌ๊ฐ€ cp-ansible ์ž์ฒด์˜ ๋ฃจํŠธ์ธ์ง€ ํ™•์ธํ•˜์‹ญ์‹œ์˜ค. ์ด๋ ‡๊ฒŒ ํ•˜๋ฉด hash_behaviour=merge sible.cfg ํŒŒ์ผ ์„ ์–ป์„ ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

๋ณ€์ˆ˜์˜ ์œ„์น˜๊ฐ€ ํ˜ผ๋ž€์Šค๋Ÿฝ๋‹ค๋Š” ์ ์„ ์ธ์ •ํ•ฉ๋‹ˆ๋‹ค. ์‹œ๊ฐ„์ด ์žˆ์„ ๋•Œ ๊ฐœ์„ ํ•  ์ˆ˜ ์žˆ๊ธฐ๋ฅผ ๋ฐ”๋ž๋‹ˆ๋‹ค.

๋ชจ๋“  9 ๋Œ“๊ธ€

์‹ค์ œ๋กœ ์‚ฌ์ „์„ ๋ณ‘ํ•ฉํ•ด์•ผ ํ•ฉ๋‹ˆ๋‹ค. cp-ansible์„ ์‹คํ–‰ํ•  ๋•Œ ํ˜„์žฌ ๋””๋ ‰ํ† ๋ฆฌ๊ฐ€ cp-ansible ์ž์ฒด์˜ ๋ฃจํŠธ์ธ์ง€ ํ™•์ธํ•˜์‹ญ์‹œ์˜ค. ์ด๋ ‡๊ฒŒ ํ•˜๋ฉด hash_behaviour=merge sible.cfg ํŒŒ์ผ ์„ ์–ป์„ ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

๋ณ€์ˆ˜์˜ ์œ„์น˜๊ฐ€ ํ˜ผ๋ž€์Šค๋Ÿฝ๋‹ค๋Š” ์ ์„ ์ธ์ •ํ•ฉ๋‹ˆ๋‹ค. ์‹œ๊ฐ„์ด ์žˆ์„ ๋•Œ ๊ฐœ์„ ํ•  ์ˆ˜ ์žˆ๊ธฐ๋ฅผ ๋ฐ”๋ž๋‹ˆ๋‹ค.

๋‚˜๋Š” ๊ทธ๊ฒƒ์„ ์‹œ๋„ํ–ˆ๋‹ค. ์ด์ œ ๋™์ผํ•œ ํ”Œ๋ ˆ์ด๋ถ์„ ์‹คํ–‰ํ•˜๋ฉด

TASK [confluent.kafka_broker : Set Permissions on Data Dirs] ******
 {"msg": "'dict object' has no attribute 'kafka_broker'"}

ํ  ์ด์ƒํ•ฉ๋‹ˆ๋‹ค. ๋‹ค์Œ์„ ์ œ๊ณตํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๊นŒ?

git status
git branch
cat hosts.yml

๋˜๋Š” ์ธ๋ฒคํ† ๋ฆฌ ํŒŒ์ผ์˜ ์ด๋ฆ„์ด ๋ฌด์—‡์ด๋“ 

์ž˜. ๋‚˜๋Š”์ด hosts.yml๊ณผ ํ•จ๊ป˜ ์ผํ•˜๊ฒŒํ–ˆ๋‹ค.

all:
  vars:
    ssl_enabled: false
    ssl_mutual_auth_enabled: false
    sasl_protocol: none
    kafka_broker_configure_additional_brokers: false

kafka_broker:
  vars:
    kafka_broker:
      datadir:
        - /var/lib/kafka/my-data
  hosts:
    172.XX.YY.20:

    172.XX.YY.18:

    172.XX.YY.19:

์ด์— ์•ž์„œ ๋™์  ์ธ๋ฒคํ† ๋ฆฌ์—์„œ ๊ทธ๋ฃน ๋ณ€์ˆ˜์˜ ์ผ๋ถ€๋กœ kafka_broker ๋ฅผ ์ถ”๊ฐ€

์•ผ ์ž˜์ƒ๊ฒผ์–ด! ๋™์  ์ธ๋ฒคํ† ๋ฆฌ๋ฅผ ์‹œ๋„ํ•˜์ง€๋Š” ์•Š์•˜์ง€๋งŒ hash_merging์ด ํ™œ์„ฑํ™”๋˜์–ด ์žˆ๋Š” ํ•œ cp-ansible์ด ์ž‘๋™ํ•ด์•ผ ํ•ฉ๋‹ˆ๋‹ค!

๋˜ ๋‹ค๋ฅธ ์‚ฌํ•ญ - IP ์ฃผ์†Œ๋ฅผ ์‚ฌ์šฉํ•œ ํ˜ธ์ŠคํŠธ ์ด๋ฆ„ ์œ ํšจ์„ฑ ๊ฒ€์‚ฌ์— ๋ฌธ์ œ๊ฐ€ ์žˆ์œผ๋ฏ€๋กœ ์•”ํ˜ธํ™”๋ฅผ ์ผœ๋Š” ๋ฐฉํ–ฅ์œผ๋กœ ๊ฐ€๊ณ  ์žˆ๋‹ค๋ฉด ์•„๋งˆ๋„ ํ˜ธ์ŠคํŠธ ์ด๋ฆ„์„ ์‚ฌ์šฉํ•  ๊ฒƒ์ž…๋‹ˆ๋‹ค.

์ธ๋ฒคํ† ๋ฆฌ์—์„œ Kafka VM์„ kafka_hosts๋กœ ๊ทธ๋ฃนํ™”ํ–ˆ์œผ๋ฏ€๋กœ all.yaml์—์„œ ํ˜ธ์ŠคํŠธ ๊ฐ’์„ ๋ณ€๊ฒฝํ•˜๋ฉด ๋‹ค์Œ๊ณผ ๊ฐ™์Šต๋‹ˆ๋‹ค.

- name: Kafka Broker Provisioning
  # hosts: kafka_broker
  hosts: kafka_hosts

์ด์™€ ๊ฐ™์ด ์ธ๋ฒคํ† ๋ฆฌ ๋””๋ ‰ํ† ๋ฆฌ ์•„๋ž˜์— ๊ทธ๋ฃน ๋ณ€์ˆ˜๋ฅผ ์ •์˜ํ–ˆ์Šต๋‹ˆ๋‹ค.

# File kafka_hosts.yaml
---
ssl_enabled: false
ssl_mutual_auth_enabled: false
sasl_protocol: none
kafka_broker_configure_additional_brokers: false
confluent_server_enabled: false

kafka_broker_service_environment_overrides:
  KAFKA_HEAP_OPTS: "-Xms1g -Xmx1g -XX:MetaspaceSize=96m -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:G1HeapRegionSize=16M -XX:MinMetaspaceFreeRatio=50 -XX:MaxMetaspaceFreeRatio=80"

kafka_broker:
  datadir:
    - /var/lib/kafka/
  auto.create.topics.enable: true
  properties:
    num.partitions: 20
    log.flush.interval.messages: 10000
    log.flush.interval.ms: 1000
    zookeeper.connection.timeout.ms: 6000
    offsets.topic.replication.factor: 2
    min.insync.replicas: 2
    default.replication.factor: 3  

์ด๊ฒƒ์€ kafka_broker์™€ ๊ฐ™์€ ๋งต์„ ์ œ์™ธํ•œ ๋ชจ๋“  ๋ณ€์ˆ˜๋ฅผ ์กด์ค‘ํ•ฉ๋‹ˆ๋‹ค. ๊ทธ๊ฒƒ์€ ๋‚˜์—๊ฒŒ ์˜ค๋ฅ˜๋ฅผ ์ค€๋‹ค

 FAILED! => {"msg": "'dict object' has no attribute 'kafka_broker'"}

๊ทธ๋ฃน ์ด๋ฆ„์„ ํ•ญ์ƒ kafka_broker ๋กœ ์ง€์ •ํ•ด์•ผ ํ•ฉ๋‹ˆ๊นŒ?

์ฃ„์†กํ•ฉ๋‹ˆ๋‹ค, ์ €๋Š” asible์— ๋Œ€ํ•œ ์ดˆ๋ณด์ž์ž…๋‹ˆ๋‹ค. ์–ด์ฉŒ๋ฉด ๋‚˜๋Š” ๋ช‡ ๊ฐ€์ง€ ๊ธฐ๋ณธ ์‚ฌํ•ญ์„ ๋†“์น˜๊ณ  ์žˆ์Šต๋‹ˆ๋‹ค.

@sandeeprapido ์—์„œ ๋™์ผํ•œ ์ž‘์—…์„ ์ˆ˜ํ–‰ํ–ˆ์Šต๋‹ˆ๋‹ค. ์ด ์ž‘์—…์„ ์ˆ˜ํ–‰ํ•˜๋ ค๋ฉด ํ˜ธ์ŠคํŠธ์˜ ์ด๋ฆ„์„ kafka_broker๋กœ ์ง€์ •ํ•ด์•ผ ํ•ฉ๋‹ˆ๋‹ค.

์•„์ง ๋„์›€์ด ํ•„์š”ํ•˜์‹ญ๋‹ˆ๊นŒ?

์ฝ”๋“œ์— ๋””๋ฒ„๊ทธ ์ž‘์—…์„ ์ถ”๊ฐ€ํ•˜์—ฌ vars๊ฐ€ ๋ฌด์—‡์œผ๋กœ ์„ค์ •๋˜์—ˆ๋Š”์ง€ ํ™•์ธํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. ๋‚˜๋Š” ์ด๊ฒƒ์„ ์–ด๋”˜๊ฐ€์— ์ถ”๊ฐ€ ํ•  ๊ฒƒ์ด๋‹ค

- debug: var=kafka_broker

๋˜ํ•œ hash_merging์ด ansible.cfg์™€ ํ•จ๊ป˜ ์‚ฌ์šฉ๋˜๊ณ  ์žˆ๋Š”์ง€ ํ™•์ธํ•˜์‹ญ์‹œ์˜ค.

ํ˜ธ์ŠคํŠธ ๊ทธ๋ฃน์˜ ์ด๋ฆ„์„ kafka_broker์™€ ๋‹ค๋ฅธ ์ด๋ฆ„์œผ๋กœ ์ง€์ •ํ•ด์•ผ ํ–ˆ์Šต๋‹ˆ๋‹ค. ๊ทธ๋ž˜์„œ ํ”Œ๋ ˆ์ด๋ถ์„ ์ˆ˜์ •ํ–ˆ์Šต๋‹ˆ๋‹ค. ํ˜ธ์ŠคํŠธ ๊ทธ๋ฃน ์ด๋ฆ„์„ ๋ณด์œ ํ•˜๋Š” kafka_broker_host_group ๋ณ€์ˆ˜๋ฅผ ์ถ”๊ฐ€ํ–ˆ์Šต๋‹ˆ๋‹ค.
๊ทธ๋ฆฌ๊ณ  ์•„๋ž˜์™€ ๊ฐ™์ด ์—ญํ•  ํŒŒ์ผ์„ ๋ณ€๊ฒฝํ–ˆ์Šต๋‹ˆ๋‹ค.
bootstrap.servers={% for host in groups['kafka_broker'] %} ์ด bootstrap.servers={% for host in groups[kafka_broker_host_group] ๋ณ€๊ฒฝํ–ˆ์Šต๋‹ˆ๋‹ค. ์ด๊ฒƒ์€ ๋‚ด๊ฐ€ ์‚ฌ์šฉ์ž ์ง€์ • ๊ทธ๋ฃน ์ด๋ฆ„์„ ๊ฐ–๋Š” ๋ฐ ๋„์›€์ด ๋˜์—ˆ์Šต๋‹ˆ๋‹ค.
๊ทธ๋ฆฌ๊ณ  ์˜ˆ, hash_behaviour๋ฅผ ๋ณ‘ํ•ฉ์œผ๋กœ ์„ค์ •ํ•˜๋ฉด ์ด ๋ฌธ์ œ๋ฅผ ์ œ๊ธฐํ•œ ์›๋ž˜ ๋ฒ„๊ทธ๋ฅผ ํ•ด๊ฒฐํ•˜๋Š” ๋ฐ ๋„์›€์ด ๋˜์—ˆ์Šต๋‹ˆ๋‹ค.

๋„์™€ ์ฃผ์…”์„œ ๊ฐ์‚ฌํ•ฉ๋‹ˆ๋‹ค :)

์ด ํŽ˜์ด์ง€๊ฐ€ ๋„์›€์ด ๋˜์—ˆ๋‚˜์š”?
0 / 5 - 0 ๋“ฑ๊ธ‰