Cp-ansible: Purpose of confluent.variables_handlers role

It should actually merge the dictionaries, make sure your current directory when running cp-ansible is the root of cp-ansible itself. This will gain you the ansible.cfg file with hash_behaviour=merge

I'll admit, the location of variables is confusing, and I hope to improve it when I have time

I tried that. Now when I run the same playbook I get

TASK [confluent.kafka_broker : Set Permissions on Data Dirs] ******
 {"msg": "'dict object' has no attribute 'kafka_broker'"}

Hmm odd, can you provide:

git status
git branch
cat hosts.yml

or whatever your inventory file is named

Well. I got it working with this hosts.yml

all:
  vars:
    ssl_enabled: false
    ssl_mutual_auth_enabled: false
    sasl_protocol: none
    kafka_broker_configure_additional_brokers: false

kafka_broker:
  vars:
    kafka_broker:
      datadir:
        - /var/lib/kafka/my-data
  hosts:
    172.XX.YY.20:

    172.XX.YY.18:

    172.XX.YY.19:

Prior to this, I added kafka_broker as part of group vars in dynamic inventory. It was somehow not working. I am checking that. Thanks for the prompt help.

Ya that looks good! I haven't tried out dynamic inventories, but as long as hash_merging is enabled cp-ansible should work!

Another thing- I've had problems with hostname validation using IP addresses, so if you are going the direction of turning encryption on, I would probably use hostnames

I have grouped my Kafka VMs as kafka_hosts in my inventory, so changing the hosts values in all.yaml looks like this.

- name: Kafka Broker Provisioning
  # hosts: kafka_broker
  hosts: kafka_hosts

and defined group variables under the inventory directory like this.

# File kafka_hosts.yaml
---
ssl_enabled: false
ssl_mutual_auth_enabled: false
sasl_protocol: none
kafka_broker_configure_additional_brokers: false
confluent_server_enabled: false

kafka_broker_service_environment_overrides:
  KAFKA_HEAP_OPTS: "-Xms1g -Xmx1g -XX:MetaspaceSize=96m -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:G1HeapRegionSize=16M -XX:MinMetaspaceFreeRatio=50 -XX:MaxMetaspaceFreeRatio=80"

kafka_broker:
  datadir:
    - /var/lib/kafka/
  auto.create.topics.enable: true
  properties:
    num.partitions: 20
    log.flush.interval.messages: 10000
    log.flush.interval.ms: 1000
    zookeeper.connection.timeout.ms: 6000
    offsets.topic.replication.factor: 2
    min.insync.replicas: 2
    default.replication.factor: 3  

This is respecting all variables except maps like kafka_broker. It gives me the error

 FAILED! => {"msg": "'dict object' has no attribute 'kafka_broker'"}

Should I always name the group as kafka_broker ?

Sorry, I am a newbie to ansible. Maybe I am missing some basics.

Just went through the same thing here - @sandeeprapido - you do need to namespace your hosts to kafka_broker for this to work.

Do you still need help on this?

You can add debug tasks in the code to see what your vars are set as. I would add this somewhere

- debug: var=kafka_broker

Also confirm hash_merging is being used with your ansible.cfg

I needed to name the host group as something different from kafka_broker. So I made a change in the playbook. I added a variable kafka_broker_host_group that holds the host group name.
And made changes to the roles files like below.
bootstrap.servers={% for host in groups['kafka_broker'] %} changed it to bootstrap.servers={% for host in groups[kafka_broker_host_group]. This helped me to have custom group names.
And yes, setting hash_behaviour to merge helped me to resolve the original bug for which I opened this issue.

Thanks for the help :)