Cp-ansible: Ansible Galaxy support

Any update on it? It will be a lot easier to use these playbooks

Currently there are no plans to publish to ansible galaxy. I will file an internal jira to see about investigating this for a future release.

The general expectation is to clone the repo, and then run the playbook required for the given setup.

Ansible 2.9 introduced a new distribution format called Collections. It seems like a better fit for resolving this issue since a collection can include playbooks, roles, modules, and plugins.

Some pointers:

• https://www.ansible.com/blog/getting-started-with-ansible-collections
• https://docs.ansible.com/ansible/latest/user_guide/collections_using.html
• https://docs.ansible.com/ansible/latest/dev_guide/developing_collections.html

I took a stab at restructuring as a galaxy collection, tested working with the all.yml playbook and some mostly basic config (ssl and sasl/plain listeners). Parts of it are very clunky, primarily that as far as I can tell to reference any filters and modules you have to reference the entire galaxy namespace. It works for my use case though, and I'm happy to open a PR if the team thinks it's worth adding.

https://github.com/aig787/cp-ansible/tree/agriffin/5.4.1-repack

@aig787 I like it! I've always felt the upgrade playbooks should be in a "playbook" directory
The roles have access to whats in the plugins dir right?
Also is the confluentinc_cp dir necessary?

One tricky piece is the ansible.cfg, I've always liked having that right next to the playbooks, because require that the ansible config in that file is use during the install

The roles do all have access to what's in the plugins dir (https://docs.ansible.com/ansible/devel/dev_guide/developing_collections.html#plugins-directory). I think you just have to use the full name to access it, e.g. I had to change KAFKA_OPTS: "{{ kafka_broker_final_java_args | java_arg_build_out }}" to KAFKA_OPTS: "{{ kafka_broker_final_java_args | aig787.confluent_cp.java_arg_build_out }}" to get the filter to work for me. There might some way around that but I wasn't able to find it.

Just tried with the confluentinc_cp directory removed and it worked just fine, so I guess it's not necessary. I initially bootstrapped it with ansible-galaxy collection init and went with the structure that created.

Is there a reason the ansible.cfg couldn't just be moved into the playbooks directory with the rest? I _think_ that in order to use the playbooks people would still need to clone the repo and run as they do now. The advantage of having it in galaxy would more be to give people access to the roles in their own setups than to make playbooks easier. That's my 2 cents anyway.

I personally do not care where the ansible.cfg is placed, i just know if you run the playbook and your current dir is not where ansible.cfg is then there are problems.

I can see the value of adhering to ansible best practices and following the collections directory structure, but I'm curious what the benefits of publishing this collection into galaxy gains us. More visibility?

To me it seems like a lot of overhead to manage versioning in galaxy and in our branches.

It would be an ease of use gain. If I have a playbook that does some base configuration to all of my nodes I could add this to a requirements file and run `ansible-galaxy install -r requirements.yml' and reference the kafka role in my existing playbook rather than having to clone the repo and extend the playbook or copy roles directories around.

As for why you actually have to publish it, I don't believe collections support git references like roles do (https://galaxy.ansible.com/docs/using/installing.html#installing-multiple-roles-from-a-file vs. https://docs.ansible.com/ansible/latest/user_guide/collections_using.html#install-multiple-collections-with-a-requirements-file). I do agree about the amount of overhead for versioning, it seems a little heavy.

I personally do not care to have it in Galaxy or not, but, being able to easily reference the different roles in a requirement file would be really appreciated :smile:

Was about to register a new issue, but I think my issue is somehow related to this issue. So starting with a comment here to get some initial feedback from the maintainers and others watching.

I want to use the roles in this repository, but I want to use them as role dependencies in my custom roles (role extensions). We started off using the playbook (all.yml), but had a lot of trouble getting our extensions applied with correct timing. My first (blocking) issue with this approach was fixed with https://github.com/confluentinc/cp-ansible/pull/442. But after looking closely at our logs for the playbook run, I realize that we might have additional issues:

[DEPRECATION WARNING]: Included file 
'/builds/kafka/provisioner/tasks/failure_handling.yml' not found, however since
 this include is not explicitly marked as 'static: yes', we will try and 
include it dynamically later. In the future, this will be an error unless 
'static: no' is used on the include task. If you do not want missing includes 
to be considered dynamic, use 'static: yes' on the include or set the global 
ansible.cfg options to make all includes static for tasks and/or handlers. This
 feature will be removed from ansible-base in version 2.12. Deprecation 
warnings can be disabled by setting deprecation_warnings=False in ansible.cfg.

Looking at the details, I think the general problem is that the roles are currently not self-contained. Fixing that would be a mandatory step towards eventually publishing the roles or role-collection to Ansible Galaxy. And I think this can be fixed quite easily, since it boils down to references to the following top-level directories /filter_plugins and /tasks from within the roles.

What do you think about the following tasks as pull requests:

1. Move the /filter_plugins folder into the confluent.variables role - making it part of that role? The filter plugins are at least in use in this role.

2. Move the reusable tasks-files in /tasks into one of the existing roles, probably confluent.common is the best fit of the existing roles, or creating a new dedicated role for them (confluent.common_tasks)? And update the references to the reused tasks-files with include_role#tasks_from. Example:

Index: roles/confluent.zookeeper/tasks/health_check.yml
<+>UTF-8
===================================================================
--- roles/confluent.zookeeper/tasks/health_check.yml    (revision 0d369ae20c8215dd2ec469f269dc804539f58190)
+++ roles/confluent.zookeeper/tasks/health_check.yml    (date 1602793473938)
@@ -25,7 +25,9 @@
   ignore_errors: true

 - name: Fetch Files for Debugging Failure
-  include: tasks/failure_handling.yml
+  include_role:
+    name: confluent.common
+    tasks_from: failure_handling.yml
   vars:
     service_name: "{{zookeeper_service_name}}"
     config_file: "{{zookeeper.config_file}}"

@erikgb It's something we've discussed internally in the past. In particular, there was quite a lot of discussion around self contained roles, vs. minimising duplication of code. It looks like we will have to change this for the 2.12. release of Ansible, based on the warning, so I am thinking it may make the most sense to move both the /filter_plugins and /tasks into the confluent.common role, and then use proper import statements in all other roles.

@domenicbove what are your thoughts?

I have no problem with moving file locations as long as functionality stays consistent @erikgb @JumaX.
This seems like large enough work that it should go into the master branch, targeting our next release though.

@domenicbove Agreed about it being work for our next major release. @erikgb We will add an internal JIRA to track this and see if we can get it on the roadmap for our next release.

Or @erikgb If you have the work sitting in a branch, you are welcome to make a PR into the master branch and we will review. I dont think that will solve this issue in its entirety though, we still need to make galaxy.

Next major release? Would that be Confluent Platform 7? This change could be made backwards compatible, I think - as long as one consider the main entrypoints to be the playbooks and the roles. I would prefer to get this fixed before CP 7. 😄

@erikgb it would be for CP 6.1, we tend to not put new functionality, especially if it involves a major refactoring into older releases. We have a lot of enterprise customers that have policies/issues when we change the repo too much for existing version, we try to keep it to bug fixes only.

@domenicbove @JumaX CP 6.1 makes more sense. I will see if I get some time to work on pull request(s) later this week. I would recommend to split the refactoring into meaningful parts. WDYT?

@erikgb If you want to split it into multiple PR's, that's fine assuming that each PR is able to still run without an additional PR's. Not sure I've stated that very well. :). Should aim for the Master Branch.