Awx-operator: Internal Server Error after deployment with new awx-operator version 0.7.0

Created on 23 Mar 2021 · 8Comments · Source: ansible/awx-operator

Yesterday (22nd of march 2021) I deployed AWX fine with awx operator version 0.6.0.
Today I got the new awx-operator version 0.7.0 and all pods start, but the awx-web is full of errors.
the ingress url to AWX now gives me the only line: Internal Server Error

what I also noticed that yesterday I only had 3 containers in the awx deployment, now I have 4. awx-ee is now also the 4th container in the POD.

logging of awx-web:

Traceback (most recent call last):
File "/usr/bin/awx-manage", line 8, in
sys.exit(manage())
File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/awx/__init__.py", line 154, in manage
execute_from_command_line(sys.argv)
File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/core/management/__init__.py", line 381, in execute_from_command_line
utility.execute()
File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/core/management/__init__.py", line 357, in execute
django.setup()
File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/__init__.py", line 19, in setup
configure_logging(settings.LOGGING_CONFIG, settings.LOGGING)
File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/utils/log.py", line 76, in configure_logging
logging_config_func(logging_settings)
File "/usr/lib64/python3.6/logging/config.py", line 802, in dictConfig
dictConfigClass(config).configure()
File "/usr/lib64/python3.6/logging/config.py", line 573, in configure
'%r: %s' % (name, e))
ValueError: Unable to configure handler 'console': Unable to add filter 'guid': 'guid'
2021-03-23 22:05:53,677 INFO exited: wsbroadcast (exit status 1; not expected)
2021-03-23 22:05:53,677 INFO exited: wsbroadcast (exit status 1; not expected)
2021-03-23 22:05:54,753 INFO spawned: 'wsbroadcast' with pid 4180
2021-03-23 22:05:54,753 INFO spawned: 'wsbroadcast' with pid 4180
Traceback (most recent call last):
File "/usr/lib64/python3.6/logging/config.py", line 692, in add_filters
filterer.addFilter(self.config['filters'][f])
File "/usr/lib64/python3.6/logging/config.py", line 326, in __getitem__
value = dict.__getitem__(self, key)
KeyError: 'guid'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/lib64/python3.6/logging/config.py", line 565, in configure
handler = self.configure_handler(handlers[name])
File "/usr/lib64/python3.6/logging/config.py", line 753, in configure_handler
self.add_filters(result, filters)
File "/usr/lib64/python3.6/logging/config.py", line 694, in add_filters
raise ValueError('Unable to add filter %r: %s' % (f, e))
ValueError: Unable to add filter 'guid': 'guid'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/var/lib/awx/venv/awx/bin/daphne", line 8, in
sys.exit(CommandLineInterface.entrypoint())
File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/daphne/cli.py", line 191, in entrypoint
cls().run(sys.argv[1:])
File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/daphne/cli.py", line 252, in run
application = import_by_path(args.application)
File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/daphne/utils.py", line 12, in import_by_path
target = importlib.import_module(module_path)
File "/var/lib/awx/venv/awx/lib64/python3.6/importlib/__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "", line 994, in _gcd_import
File "", line 971, in _find_and_load
File "", line 955, in _find_and_load_unlocked
File "", line 665, in _load_unlocked
File "", line 678, in exec_module
File "", line 219, in _call_with_frames_removed
File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/awx/asgi.py", line 35, in
django.setup()
File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/__init__.py", line 19, in setup
configure_logging(settings.LOGGING_CONFIG, settings.LOGGING)
File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/utils/log.py", line 76, in configure_logging
logging_config_func(logging_settings)
File "/usr/lib64/python3.6/logging/config.py", line 802, in dictConfig
dictConfigClass(config).configure()
File "/usr/lib64/python3.6/logging/config.py", line 573, in configure
'%r: %s' % (name, e))
ValueError: Unable to configure handler 'console': Unable to add filter 'guid': 'guid'
2021-03-23 22:05:55,974 INFO success: wsbroadcast entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2021-03-23 22:05:55,974 INFO success: wsbroadcast entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2021-03-23 22:05:55,974 INFO exited: daphne (exit status 1; not expected)
2021-03-23 22:05:55,974 INFO exited: daphne (exit status 1; not expected)

docs

Source

lingenavd

Most helpful comment

Final test from me, because it works now :-)

again with new clean k8s cluster.
I used the 0.7.0 for the awx-operator
I changed the line in my-awx.yml to get the image from quay.io/ansible/awx:latest
This resulted in version 18.0, so @Zokormazo was right that my k8s cluster got the latest version from dockerhub.

kubectl -n awx get pods
NAME READY STATUS RESTARTS AGE
awx-889f8c78-4bj5l 4/4 Running 0 26m
awx-postgres-0 1/1 Running 0 27m

the awx-ee container is now also deployed in the AWX deployment pod.

AWX was completely deployed without any issues. :-)
Also the secret "awx-admin-password" is created with the awx-operator version 0.7.0 and this works like a charm.

Conclusion:
Everything works as designed if you have the correct combination for the awx-operator and the AWX image version.
So:
- awx-operator version 0.7.0 (lastest) ONLY works successfully in combination with AWX 18.0.0 (latest from quay.io/ansible/awx:latest)
- awx-operator version 0.6.0 in combination with the AWX versions I tested (AWX 15.0.0 and 17.1.0) works fine with the workaround for the AWX admin password.

Maybe this could be mentioned in the install docs more clearly.

Issue can be closed :-)

grtz,
Andre

lingenavd on 25 Mar 2021

👍2

All 8 comments

Did you supply your own tower_image? You'll either need to use the latest, or use an older version of this operator.

shanemcd on 24 Mar 2021

Tower image was the vanilla version 15.0. (I did not confige it in my yaml)
I will give it a go with the latest tower image today.
Thanx for the feedback.
Grtz,
Andre

lingenavd on 24 Mar 2021

Sorry to say it does not make a difference :-(
Clean K8s cluster.
Only line I get on the URL to AWX is: Internal Server Error

kubectl -n awx get pods
NAME READY STATUS RESTARTS AGE
awx-968dd9cc8-85l7n 4/4 Running 0 35m
awx-postgres-0 1/1 Running 0 35m

my-awx.yml:

#######################################################

apiVersion: awx.ansible.com/v1beta1
kind: AWX
metadata:
namespace: awx
name: awx
spec:
tower_admin_user: admin
#tower_admin_password_secret:

tower_image: ansible/awx:latest
tower_image_pull_policy: IfNotPresent

tower_web_resource_requirements:
requests:
cpu: 1000m
memory: 2Gi
limits:
cpu: 2000m
memory: 3Gi

tower_task_resource_requirements:
requests:
cpu: 500m
memory: 1Gi
limits:
cpu: 1000m
memory: 2Gi

tower_postgres_resource_requirements:
requests:
memory: 1Gi
storage: 8Gi
limits:
memory: 2Gi
storage: 16Gi
tower_postgres_storage_class: manual

#

All containers except awx-web and awx_task are fine.
see attached zip file for all container logging.
awx-postgres.zip

lingenavd on 24 Mar 2021

I tested it again with new clean k8s cluster.
I changed the following line in the awx-operator.yaml after download:
cat awx-operator.yaml | grep 0.
image: "quay.io/ansible/awx-operator:0.6.0"
I used the same: my-awx.yml

kubectl -n awx get pods
NAME READY STATUS RESTARTS AGE
awx-8b89d69f-xd259 3/3 Running 0 19m
awx-postgres-0 1/1 Running 0 19m

Now, I don't have the awx-ee container in the POD for AWX.

BUT the MOST important thing is that AWX was completely deployed without issues. :-)
I am now logged into AWX with the new AWX images, looks much better, compliments for that :-)

There is definitely something wrong with the awx-operator 0.7.0 and prob. the combination with awx-ee?

Only issue I have with this awx-operator that it does not create the secret "awx-admin-password".
So you cannot login when AWX is ready up and running.
But that is easly fixed doing the following:

kubectl -n awx exec -it In the container just create you own superuser account:
awx-manage createsuperuser --username

grtz,
Andre

lingenavd on 24 Mar 2021

Wich registry is resolving that tower_image: ansible/awx:latest ? I would say that your k8s is pulling that from dockerhub, where latest is 17.1.0 and not 18.0.0

Try using fqdn: tower_image: quay.io/ansible/awx:latest

Zokormazo on 24 Mar 2021

👍1

I just checked, the installed version is indeed 17.1.0

Grtz,
Andre

lingenavd on 24 Mar 2021

Final test from me, because it works now :-)

kubectl -n awx get pods
NAME READY STATUS RESTARTS AGE
awx-889f8c78-4bj5l 4/4 Running 0 26m
awx-postgres-0 1/1 Running 0 27m

the awx-ee container is now also deployed in the AWX deployment pod.

AWX was completely deployed without any issues. :-)
Also the secret "awx-admin-password" is created with the awx-operator version 0.7.0 and this works like a charm.

Maybe this could be mentioned in the install docs more clearly.

Issue can be closed :-)

grtz,
Andre

lingenavd on 25 Mar 2021

👍2

I agree with @lingenavd here. If version of awx-operator requires specific version of AWX it should be written somewhere.