Awx-operator: Masalah saat membuat penyimpanan persisten untuk /var/lib/awx/projects dan postgresql - menggunakan awx-operator 0.8.0

@PugTheBlack sepertinya ada masalah pada penyedia storage selain awx-operator itu sendiri.

Events:
  Type     Reason       Age                   From     Message
  ----     ------       ----                  ----     -------
  Warning  FailedMount  7m42s (x244 over 9h)  kubelet  MountVolume.MountDevice failed for volume "pvc-ae007b9b-fe8d-42fe-9975-24ad1a79c67a" : rpc error: code = InvalidArgument desc = staging path /var/snap/microk8s/common/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-ae007b9b-fe8d-42fe-9975-24ad1a79c67a/globalmount does not exist on node

Saya akan menyelidiki masalah yang terkait dengan rook-ceph dan pesan di atas yang mungkin mengarahkan Anda ke arah untuk memperbaikinya. Saya juga menerapkan rook-ceph sini dan dalam pengujian saya, itu berfungsi dengan baik.

Ya, itu mungkin masalah benteng-ceph lebih dari awx-operator - alasan mengapa saya mempostingnya di sini sebagian besar adalah bagian di mana pvc terdaftar sebagai terikat. Akan memeriksanya dengan para rook-ceph dan melihat apakah mereka dapat membantu :)

@tchellomello seperti yang Anda katakan masalahnya ada pada implementasi rook-ceph. Untuk beberapa alasan saya harus membuat operator benteng dengan snap-path penuh /var/snap/microk8s/common/var/lib/kubelet dan bukan /var/lib/kubelet, maka awx-operator menjalankannya... . jadi sekarang saya memiliki serangkaian masalah yang berbeda :)

# microk8s kubectl describe svc awx-postgres        
Name:              awx-postgres
Namespace:         default
Labels:            app.kubernetes.io/component=database
                   app.kubernetes.io/managed-by=awx-operator
                   app.kubernetes.io/name=awx-postgres
                   app.kubernetes.io/part-of=awx
Annotations:       <none>
Selector:          app.kubernetes.io/component=database,app.kubernetes.io/managed-by=awx-operator,app.kubernetes.io/name=awx-postgres
Type:              ClusterIP
IP Family Policy:  SingleStack
IP Families:       IPv4
IP:                None
IPs:               None
Port:              <unset>  5432/TCP
TargetPort:        5432/TCP
Endpoints:         <none>
Session Affinity:  None
Events:            <none>

# microk8s kubectl get svc -A -o wide                         
NAMESPACE     NAME                        TYPE           CLUSTER-IP       EXTERNAL-IP      PORT(S)                      AGE     SELECTOR
default       kubernetes                  ClusterIP      10.152.183.1     <none>           443/TCP                      4d23h   <none>
kube-system   kube-dns                    ClusterIP      10.152.183.10    <none>           53/UDP,53/TCP,9153/TCP       4d23h   k8s-app=kube-dns
rook-ceph     rook-ceph-mon-a             ClusterIP      10.152.183.207   <none>           6789/TCP,3300/TCP            4d22h   app=rook-ceph-mon,ceph_daemon_id=a,mon=a,mon_cluster=rook-ceph,rook_cluster=rook-ceph
rook-ceph     rook-ceph-mon-b             ClusterIP      10.152.183.235   <none>           6789/TCP,3300/TCP            4d22h   app=rook-ceph-mon,ceph_daemon_id=b,mon=b,mon_cluster=rook-ceph,rook_cluster=rook-ceph
rook-ceph     rook-ceph-mon-c             ClusterIP      10.152.183.197   <none>           6789/TCP,3300/TCP            4d22h   app=rook-ceph-mon,ceph_daemon_id=c,mon=c,mon_cluster=rook-ceph,rook_cluster=rook-ceph
rook-ceph     rook-ceph-mgr               ClusterIP      10.152.183.119   <none>           9283/TCP                     4d22h   app=rook-ceph-mgr,ceph_daemon_id=a,rook_cluster=rook-ceph
ingress       ingress                     LoadBalancer   10.152.183.229   192.168.50.99    80:31981/TCP,443:31086/TCP   4d19h   name=nginx-ingress-microk8s
default       awx-operator-metrics        ClusterIP      10.152.183.159   <none>           8383/TCP,8686/TCP            4d12h   name=awx-operator
kube-system   metrics-server              ClusterIP      10.152.183.177   <none>           443/TCP                      3d21h   k8s-app=metrics-server
kube-system   dashboard-metrics-scraper   ClusterIP      10.152.183.117   <none>           8000/TCP                     3d21h   k8s-app=dashboard-metrics-scraper
kube-system   kubernetes-dashboard        LoadBalancer   10.152.183.192   192.168.50.100   443:31066/TCP                3d21h   k8s-app=kubernetes-dashboard
rook-ceph     rook-ceph-mgr-dashboard     ClusterIP      10.152.183.165   <none>           8443/TCP                     4d22h   app=rook-ceph-mgr,ceph_daemon_id=a,rook_cluster=rook-ceph
rook-ceph     csi-rbdplugin-metrics       ClusterIP      10.152.183.34    <none>           8080/TCP,8081/TCP            67m     contains=csi-rbdplugin-metrics
rook-ceph     csi-cephfsplugin-metrics    ClusterIP      10.152.183.30    <none>           8080/TCP,8081/TCP            67m     contains=csi-cephfsplugin-metrics
default       awx-postgres                ClusterIP      None             <none>           5432/TCP                     60m     app.kubernetes.io/component=database,app.kubernetes.io/managed-by=awx-operator,app.kubernetes.io/name=awx-postgres



md5-e98be4b4cf684e3e96e0480b80fc1f2a



# microk8s kubectl describe pod awx-86899bfb7b-ccg76
Name:         awx-86899bfb7b-ccg76
Namespace:    default
Priority:     0
Node:         mwg-csm-n03/192.168.50.43
Start Time:   Tue, 04 May 2021 06:42:02 +0000
Labels:       app.kubernetes.io/component=awx
              app.kubernetes.io/managed-by=awx-operator
              app.kubernetes.io/name=awx
              app.kubernetes.io/part-of=awx
              app.kubernetes.io/version=19.0.0
              pod-template-hash=86899bfb7b
Annotations:  cni.projectcalico.org/podIP: 10.1.238.230/32
              cni.projectcalico.org/podIPs: 10.1.238.230/32
Status:       Running
IP:           10.1.238.230
IPs:
  IP:           10.1.238.230
Controlled By:  ReplicaSet/awx-86899bfb7b
Containers:
  redis:
    Container ID:  containerd://aca4d5dbe1670c14371abe31c57ce7525e1216a2c0f6e7182f6ff1953909fd95
    Image:         redis:latest
    Image ID:      docker.io/library/redis<strong i="9">@sha256</strong>:eff56acc5fc7b909183da93236ba09d3b8cb7d6db31d5b25e9a46dac9b5e699b
    Port:          <none>
    Host Port:     <none>
    Args:
      redis-server
      /etc/redis.conf
    State:          Running
      Started:      Tue, 04 May 2021 06:42:14 +0000
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /data from awx-redis-data (rw)
      /etc/redis.conf from awx-redis-config (ro,path="redis.conf")
      /var/run/redis from awx-redis-socket (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-gktwp (ro)
  awx-web:
    Container ID:   containerd://d19efb9e0af7b1f2ef346bc9b355c493d4175de9a7678bbf9339fa87560d6ca7
    Image:          quay.io/ansible/awx:19.0.0
    Image ID:       quay.io/ansible/awx<strong i="10">@sha256</strong>:db165b894507fb520d3f53ac68eb2b49f2a5fd2cc63c7ac7aaa7bd904970b1b2
    Port:           8052/TCP
    Host Port:      0/TCP
    State:          Running
      Started:      Tue, 04 May 2021 06:42:15 +0000
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     2
      memory:  4Gi
    Requests:
      cpu:     1
      memory:  2Gi
    Environment:
      MY_POD_NAMESPACE:  default (v1:metadata.namespace)
    Mounts:
      /etc/nginx/nginx.conf from awx-nginx-conf (ro,path="nginx.conf")
      /etc/tower/SECRET_KEY from awx-secret-key (ro,path="SECRET_KEY")
      /etc/tower/conf.d/ from awx-application-credentials (ro)
      /etc/tower/settings.py from awx-settings (ro,path="settings.py")
      /var/lib/awx/rsyslog from rsyslog-dir (rw)
      /var/run/awx-rsyslog from rsyslog-socket (rw)
      /var/run/redis from awx-redis-socket (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-gktwp (ro)
      /var/run/supervisor from supervisor-socket (rw)
  awx-task:
    Container ID:  containerd://c79f56ec95c46ea947f3aa954a71f0a97a532397025b3983c1f8f9f55331940b
    Image:         quay.io/ansible/awx:19.0.0
    Image ID:      quay.io/ansible/awx<strong i="11">@sha256</strong>:db165b894507fb520d3f53ac68eb2b49f2a5fd2cc63c7ac7aaa7bd904970b1b2
    Port:          <none>
    Host Port:     <none>
    Args:
      /usr/bin/launch_awx_task.sh
    State:          Running
      Started:      Tue, 04 May 2021 06:42:15 +0000
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     1
      memory:  2Gi
    Requests:
      cpu:     500m
      memory:  1Gi
    Environment:
      SUPERVISOR_WEB_CONFIG_PATH:  /etc/supervisord.conf
      AWX_SKIP_MIGRATIONS:         1
      MY_POD_UID:                   (v1:metadata.uid)
      MY_POD_IP:                    (v1:status.podIP)
      MY_POD_NAMESPACE:            default (v1:metadata.namespace)
    Mounts:
      /etc/tower/SECRET_KEY from awx-secret-key (ro,path="SECRET_KEY")
      /etc/tower/conf.d/ from awx-application-credentials (ro)
      /etc/tower/settings.py from awx-settings (ro,path="settings.py")
      /var/lib/awx/projects from awx-projects (rw)
      /var/lib/awx/rsyslog from rsyslog-dir (rw)
      /var/run/awx-rsyslog from rsyslog-socket (rw)
      /var/run/receptor from receptor-socket (rw)
      /var/run/redis from awx-redis-socket (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-gktwp (ro)
      /var/run/supervisor from supervisor-socket (rw)
  awx-ee:
    Container ID:  containerd://06a3f2d9b7bc55c992c95d09af7c148798cd0bf1ac0eaa54cdf76ae0ecd099cf
    Image:         quay.io/ansible/awx-ee:0.1.1
    Image ID:      quay.io/ansible/awx-ee<strong i="12">@sha256</strong>:76ecd20bd375c1cc4c1fd2fc9f43e7a321136905b45f3c0fdd0304de59467b93
    Port:          <none>
    Host Port:     <none>
    Args:
      receptor
      --config
      /etc/receptor.conf
    State:          Running
      Started:      Tue, 04 May 2021 06:42:16 +0000
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /etc/receptor.conf from awx-receptor-config (ro,path="receptor.conf")
      /var/lib/awx/projects from awx-projects (rw)
      /var/run/receptor from receptor-socket (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-gktwp (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  awx-application-credentials:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  awx-app-credentials
    Optional:    false
  awx-secret-key:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  awx-secret-key
    Optional:    false
  awx-settings:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      awx-awx-configmap
    Optional:  false
  awx-nginx-conf:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      awx-awx-configmap
    Optional:  false
  awx-redis-config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      awx-awx-configmap
    Optional:  false
  awx-redis-socket:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  awx-redis-data:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  supervisor-socket:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  rsyslog-socket:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  receptor-socket:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  rsyslog-dir:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  awx-receptor-config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      awx-awx-configmap
    Optional:  false
  awx-projects:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  awx-projects-claim
    ReadOnly:   false
  kube-api-access-gktwp:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:                      <none>



md5-e98be4b4cf684e3e96e0480b80fc1f2a



# microk8s kubectl logs -f awx-86899bfb7b-ccg76 -c awx-web


2021-05-04 07:49:40,911 ERROR    [-] awx.conf.settings Database settings are not available, using defaults.
Traceback (most recent call last):
  File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
    self.connect()
  File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/django/db/backends/base/base.py", line 195, in connect
    self.connection = self.get_new_connection(conn_params)
  File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
    connection = Database.connect(**conn_params)
  File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/psycopg2/__init__.py", line 126, in connect
    conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not translate host name "awx-postgres" to address: Name or service not known


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/awx/conf/settings.py", line 81, in _ctit_db_wrapper
    yield
  File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/awx/conf/settings.py", line 416, in __getattr__
    value = self._get_local(name)
  File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/awx/conf/settings.py", line 360, in _get_local
    setting = Setting.objects.filter(key=name, user__isnull=True).order_by('pk').first()
  File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/django/db/models/query.py", line 653, in first
    for obj in (self if self.ordered else self.order_by('pk'))[:1]:
  File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/django/db/models/query.py", line 274, in __iter__
    self._fetch_all()
  File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/django/db/models/query.py", line 1242, in _fetch_all
    self._result_cache = list(self._iterable_class(self))
  File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/django/db/models/query.py", line 55, in __iter__
    results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
  File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/django/db/models/sql/compiler.py", line 1140, in execute_sql
    cursor = self.connection.cursor()
  File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/django/db/backends/base/base.py", line 256, in cursor
    return self._cursor()
  File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/django/db/backends/base/base.py", line 233, in _cursor
    self.ensure_connection()
  File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
    self.connect()
  File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/django/db/utils.py", line 89, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
    self.connect()
  File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/django/db/backends/base/base.py", line 195, in connect
    self.connection = self.get_new_connection(conn_params)
  File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
    connection = Database.connect(**conn_params)
  File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/psycopg2/__init__.py", line 126, in connect
    conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not translate host name "awx-postgres" to address: Name or service not known

Traceback (most recent call last):
  File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
    self.connect()
  File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/django/db/backends/base/base.py", line 195, in connect
^C    self.connection = self.get_new_connection(conn_params)
  File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection



md5-381ef2ba10e7440d36a992f36511ad41



# microk8s kubectl get pvc -A -o wide                     
NAMESPACE   NAME                 STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS      AGE   VOLUMEMODE
default     awx-projects-claim   Bound    pvc-5e7a4861-7432-4c64-abf2-05b5480ac3d6   20Gi       RWO            rook-ceph-block   72m   Filesystem

Pada log yang disediakan, Anda tidak menunjukkan apakah statefulset PostgreSQL sedang berjalan. Jadi apa yang terjadi jika Anda melakukan:

kubectl get pods -o wide  awx-postgres-0

Kemudian jika Anda mengambil alamat IP dari pod tersebut, dan dari wadah awx-task , apakah Anda dapat mencapai IP itu?

Selanjutnya, apa yang terjadi jika Anda menerapkan versi operator 0.9.0 terbaru dan mencoba membuat instance awx dengan nama yang berbeda?

# microk8s kubectl get pods -o wide awx-postgres-0
Error from server (NotFound): pods "awx-postgres-0" not found

Jadi tidak, itu tidak berjalan.

Saya akan mencoba men-deploy instance awx baru dengan nama yang berbeda menggunakan operator 0.9.0 nanti hari ini :)

Tapi saya melakukan "penghapusan" dari penyebaran awx lama

# microk8s kubectl delete -f my-awx.yaml 
# microk8s kubectl delete -f awx-operator.yaml

Kemudian mengkloning awx-operator baru dari github - dan sekarang semuanya bekerja lebih baik :)

# microk8s kubectl get pods -o wide
NAME                            READY   STATUS    RESTARTS   AGE     IP             NODE          NOMINATED NODE   READINESS GATES
awx-operator-5d9d764bcd-hdfxk   1/1     Running   0          8m56s   10.1.238.235   mwg-csm-n03   <none>           <none>
awx-postgres-0                  1/1     Running   0          6m30s   10.1.238.236   mwg-csm-n03   <none>           <none>

# microk8s kubectl get pvc -o wide
NAME                      STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS      AGE     VOLUMEMODE
postgres-awx-postgres-0   Bound    pvc-7b7cdd59-3f39-4f23-a76c-f01e3fe4fba0   8Gi        RWO            rook-ceph-block   7m43s   Filesystem
awx-projects-claim        Bound    pvc-5afc16bb-5089-4b70-92fd-778f3c7d2b60   20Gi       RWO            rook-ceph-block   7m18s   Filesystem

Jadi mudah-mudahan saya akan memiliki penyebaran yang berfungsi hari ini :)

# microk8s kubectl describe pods awx-postgres-0        
Name:         awx-postgres-0
Namespace:    default
Priority:     0
Node:         mwg-csm-n03/192.168.50.43
Start Time:   Thu, 06 May 2021 05:51:48 +0000
Labels:       app.kubernetes.io/component=database
              app.kubernetes.io/managed-by=awx-operator
              app.kubernetes.io/name=awx-postgres
              app.kubernetes.io/part-of=awx
              controller-revision-hash=awx-postgres-c64cb47d9
              statefulset.kubernetes.io/pod-name=awx-postgres-0
Annotations:  cni.projectcalico.org/podIP: 10.1.238.236/32
              cni.projectcalico.org/podIPs: 10.1.238.236/32
Status:       Running
IP:           10.1.238.236
IPs:
  IP:           10.1.238.236
Controlled By:  StatefulSet/awx-postgres
Containers:
  postgres:
    Container ID:   containerd://fbde7aa71f53344fe30b214de85b40b16088773db41398249bacc7554ca096f8
    Image:          postgres:12
    Image ID:       docker.io/library/postgres<strong i="5">@sha256</strong>:06277995d7028f4455e56f21864146ee2425c83308dc96283c7a96e6881cc826
    Port:           5432/TCP
    Host Port:      0/TCP
    State:          Running
      Started:      Thu, 06 May 2021 05:52:29 +0000
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     1
      memory:  4Gi
    Requests:
      cpu:     500m
      memory:  2Gi
    Environment:
      POSTGRESQL_DATABASE:        <set to the key 'database' in secret 'awx-postgres-configuration'>  Optional: false
      POSTGRESQL_USER:            <set to the key 'username' in secret 'awx-postgres-configuration'>  Optional: false
      POSTGRESQL_PASSWORD:        <set to the key 'password' in secret 'awx-postgres-configuration'>  Optional: false
      POSTGRES_DB:                <set to the key 'database' in secret 'awx-postgres-configuration'>  Optional: false
      POSTGRES_USER:              <set to the key 'username' in secret 'awx-postgres-configuration'>  Optional: false
      POSTGRES_PASSWORD:          <set to the key 'password' in secret 'awx-postgres-configuration'>  Optional: false
      PGDATA:                     /var/lib/postgresql/data/pgdata
      POSTGRES_INITDB_ARGS:       --auth-host=scram-sha-256
      POSTGRES_HOST_AUTH_METHOD:  scram-sha-256
    Mounts:
      /var/lib/postgresql/data from postgres (rw,path="data")
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-z4bwr (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  postgres:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  postgres-awx-postgres-0
    ReadOnly:   false
  kube-api-access-z4bwr:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason                  Age   From                     Message
  ----     ------                  ----  ----                     -------
  Warning  FailedScheduling        13m   default-scheduler        0/4 nodes are available: 4 pod has unbound immediate PersistentVolumeClaims.
  Normal   Scheduled               13m   default-scheduler        Successfully assigned default/awx-postgres-0 to mwg-csm-n03
  Normal   SuccessfulAttachVolume  13m   attachdetach-controller  AttachVolume.Attach succeeded for volume "pvc-7b7cdd59-3f39-4f23-a76c-f01e3fe4fba0"
  Normal   Pulling                 12m   kubelet                  Pulling image "postgres:12"
  Normal   Pulled                  12m   kubelet                  Successfully pulled image "postgres:12" in 17.444457891s
  Normal   Created                 12m   kubelet                  Created container postgres
  Normal   Started                 12m   kubelet                  Started container postgres

# microk8s kubectl describe pods awx-operator-5d9d764bcd-hdfxk
Name:         awx-operator-5d9d764bcd-hdfxk
Namespace:    default
Priority:     0
Node:         mwg-csm-n03/192.168.50.43
Start Time:   Thu, 06 May 2021 05:49:21 +0000
Labels:       name=awx-operator
              pod-template-hash=5d9d764bcd
Annotations:  cni.projectcalico.org/podIP: 10.1.238.235/32
              cni.projectcalico.org/podIPs: 10.1.238.235/32
Status:       Running
IP:           10.1.238.235
IPs:
  IP:           10.1.238.235
Controlled By:  ReplicaSet/awx-operator-5d9d764bcd
Containers:
  awx-operator:
    Container ID:   containerd://932ed30d4d9f776c2f3fb75d8274bfa78f658e75e42979ae5d3a12bad1812b9b
    Image:          quay.io/ansible/awx-operator:0.9.0
    Image ID:       quay.io/ansible/awx-operator<strong i="5">@sha256</strong>:00bb025d3607f072cb1967f72817e8c31deb485b91503b4fb3b1ca62ff857103
    Port:           <none>
    Host Port:      <none>
    State:          Running
      Started:      Thu, 06 May 2021 05:50:01 +0000
    Ready:          True
    Restart Count:  0
    Liveness:       http-get http://:6789/healthz delay=15s timeout=1s period=20s #success=1 #failure=3
    Environment:
      WATCH_NAMESPACE:    
      POD_NAME:           awx-operator-5d9d764bcd-hdfxk (v1:metadata.name)
      OPERATOR_NAME:      awx-operator
      ANSIBLE_GATHERING:  explicit
    Mounts:
      /tmp/ansible-operator/runner from runner (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-krlz9 (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  runner:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  kube-api-access-krlz9:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type    Reason     Age   From               Message
  ----    ------     ----  ----               -------
  Normal  Scheduled  17m   default-scheduler  Successfully assigned default/awx-operator-5d9d764bcd-hdfxk to mwg-csm-n03
  Normal  Pulling    17m   kubelet            Pulling image "quay.io/ansible/awx-operator:0.9.0"
  Normal  Pulled     17m   kubelet            Successfully pulled image "quay.io/ansible/awx-operator:0.9.0" in 30.290555877s
  Normal  Created    17m   kubelet            Created container awx-operator
  Normal  Started    17m   kubelet            Started container awx-operator

Tampaknya masih ada yang mencurigakan di sini ... pod "awx-" biasa tidak dibuat, dan operator awx

{"level":"error","ts":1620281677.3637104,"logger":"controller-runtime.controller","msg":"Reconciler error","controller":"awx-controller","request":"default/awx","error":"event runner on failed","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\tpkg/mod/github.com/go-logr/[email protected]/zapr.go:128\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tpkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:258\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tpkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:232\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\tpkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:211\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\tpkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\tpkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\tpkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.Until\n\tpkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:90"}

--------------------------- Ansible Task StdOut -------------------------------

 TASK [Apply Resources] ******************************** 
fatal: [localhost]: FAILED! => {"msg": "The task includes an option with an undefined variable. The error was: 'tower_loadbalancer_annotations' is undefined\n\nThe error appears to be in '/opt/ansible/roles/installer/tasks/resources_configuration.yml': line 20, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n- name: Apply Resources\n  ^ here\n"}

Saya dapat mereproduksi masalah dengan membuat spesifikasi berikut:

apiVersion: awx.ansible.com/v1beta1
kind: AWX
metadata:
  name: awx-lb-annotations
  namespace: default
spec:
  kind: AWX
  tower_admin_user: admin
  tower_ingress_type: LoadBalancer <===== added this

Inilah kesalahannya:

[awx-operator-84694f9865-kvbc2] ok: [localhost] => (item=tower_persistent) => {"ansible_loop_var": "item", "changed": false, "item": "tower_persistent", "result": {"results": []}} 
[awx-operator-84694f9865-kvbc2] fatal: [localhost]: FAILED! => {"msg": "The task includes an option with an undefined variable. The error was: 'tower_loadbalancer_annotations' is undefined\n\nThe error appears to be in '/opt/ansible/roles/installer/tasks/resources_configuration.yml': line 20, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n- name: Apply Resources\n  ^ here\n"} 
 awx-operator-84694f9865-kvbc2] 
 awx-operator-84694f9865-kvbc2] PLAY RECAP *********************************************************************
 awx-operator-84694f9865-kvbc2] localhost                  : ok=29   changed=0    unreachable=0    failed=1    skipped=26   rescued=0    ignored=0   

Itu adalah bug yang harus diperbaiki.

Mengubah file "my-awx.yaml" menjadi:

---
apiVersion: awx.ansible.com/v1beta1
kind: AWX
metadata:
  name: awx
spec:
  tower_ingress_type: LoadBalancer
  tower_loadbalancer_annotations: ''    <===== Added this
  tower_loadbalancer_protocol: http
  tower_postgres_resource_requirements:
    requests:
      cpu: 500m
      memory: 2Gi
    limits:
      cpu: 1000m
      memory: 4Gi
  tower_postgres_storage_requirements:
    requests:
      storage: 8Gi
    limits:
      storage: 50Gi
  tower_postgres_storage_class: rook-ceph-block
  tower_web_resource_requirements:
    requests:
      cpu: 1000m
      memory: 2Gi
    limits:
      cpu: 2000m
      memory: 4Gi
  tower_task_resource_requirements:
    requests:
      cpu: 500m
      memory: 1Gi
    limits:
      cpu: 1000m
      memory: 2Gi
  tower_projects_persistence: true
  tower_projects_storage_class: rook-ceph-block
  tower_projects_storage_access_mode: ReadWriteOnce
  tower_projects_storage_size: 20Gi

Dan kemudian penyebaran berjalan seperti yang saya harapkan :)

# microk8s kubectl get pods -o wide   
NAME                            READY   STATUS    RESTARTS   AGE   IP             NODE          NOMINATED NODE   READINESS GATES
awx-operator-5d9d764bcd-hdfxk   1/1     Running   0          5d    10.1.238.235   mwg-csm-n03   <none>           <none>
awx-postgres-0                  1/1     Running   0          77m   10.1.238.207   mwg-csm-n03   <none>           <none>
awx-6f94b56cdf-jddq8            4/4     Running   0          76m   10.1.238.208   mwg-csm-n03   <none>           <none>

# microk8s kubectl get svc -o wide   
NAME                   TYPE           CLUSTER-IP       EXTERNAL-IP      PORT(S)             AGE   SELECTOR
kubernetes             ClusterIP      10.152.183.1     <none>           443/TCP             11d   <none>
awx-operator-metrics   ClusterIP      10.152.183.55    <none>           8383/TCP,8686/TCP   5d    name=awx-operator
awx-postgres           ClusterIP      None             <none>           5432/TCP            73m   app.kubernetes.io/component=database,app.kubernetes.io/managed-by=awx-operator,app.kubernetes.io/name=awx-postgres
awx-service            LoadBalancer   10.152.183.224   192.168.50.101   80:30350/TCP        73m   app.kubernetes.io/component=awx,app.kubernetes.io/managed-by=awx-operator,app.kubernetes.io/name=awx

# microk8s kubectl get pvc -o wide   
NAME                      STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS      AGE   VOLUMEMODE
postgres-awx-postgres-0   Bound    pvc-4f4e9caa-961e-42ea-926d-daa3007f12bd   8Gi        RWO            rook-ceph-block   75m   Filesystem
awx-projects-claim        Bound    pvc-cb3634da-362f-4953-af0d-6b8f6be0e195   20Gi       RWO            rook-ceph-block   74m   Filesystem

Setuju bahwa ini adalah bug yang perlu diperbaiki, tetapi solusinya tampaknya berfungsi dengan baik, jadi saya senang :)