Awx-operator: 관리되는 postgres - /var/lib/postgresql/data: 권한이 거부되었습니다.

이것 좀 봐.
https://github.com/kurokobo/awx-on-k3s

위에 링크된 @marwel 의 정확한 지침을

현재 실험실에서는 재현할 수 없으므로 오늘 오후에 k3s 를 사용해 보겠습니다.

안녕하세요 여러분, https://rancher.com/docs/k3s/latest/en/quick-start/#install -script에 설명된 대로 테스트 머신에 단일 노드로 k3s 를 배포했습니다.

$ kubectl get nodes                                                            
NAME              STATUS   ROLES                  AGE     VERSION
storm.tatu.home   Ready    control-plane,master   3m39s   v1.21.3+k3s1

$ kubectl get pods -A                                                                                          23:01:09
NAMESPACE     NAME                                      READY   STATUS      RESTARTS   AGE
kube-system   local-path-provisioner-5ff76fc89d-4d7bn   1/1     Running     0          9m51s
kube-system   metrics-server-86cbb8457f-9fkt2           1/1     Running     0          9m51s
kube-system   coredns-7448499f4d-9t87w                  1/1     Running     0          9m51s
kube-system   helm-install-traefik-crd-mlrtg            0/1     Completed   0          9m51s
kube-system   helm-install-traefik-v5n5s                0/1     Completed   1          9m51s
kube-system   svclb-traefik-c9cgh                       2/2     Running     0          9m28s
kube-system   traefik-97b44b794-6dz4g                   1/1     Running     0          9m28s

그런 다음 최신 devel 운영자 이미지를 생성하고 배포했습니다.

kubectl apply -f deploy/awx-operator.yaml                                                                     23:08:40
customresourcedefinition.apiextensions.k8s.io/awxs.awx.ansible.com created
customresourcedefinition.apiextensions.k8s.io/awxbackups.awx.ansible.com created
customresourcedefinition.apiextensions.k8s.io/awxrestores.awx.ansible.com created
clusterrole.rbac.authorization.k8s.io/awx-operator created
clusterrolebinding.rbac.authorization.k8s.io/awx-operator created
serviceaccount/awx-operator created
deployment.apps/awx-operator created

운영자가 예상대로 시작했습니다.

kubectl get pods -A -w                                                                                       23:07:32
NAMESPACE     NAME                                      READY   STATUS              RESTARTS   AGE
kube-system   local-path-provisioner-5ff76fc89d-4d7bn   1/1     Running             0          11m
kube-system   metrics-server-86cbb8457f-9fkt2           1/1     Running             0          11m
kube-system   coredns-7448499f4d-9t87w                  1/1     Running             0          11m
kube-system   helm-install-traefik-crd-mlrtg            0/1     Completed           0          11m
kube-system   helm-install-traefik-v5n5s                0/1     Completed           1          11m
kube-system   svclb-traefik-c9cgh                       2/2     Running             0          10m
kube-system   traefik-97b44b794-6dz4g                   1/1     Running             0          10m
default       awx-operator-88b886454-9pq7w              0/1     ContainerCreating   0          15s
default       awx-operator-88b886454-9pq7w              1/1     Running             0          16s

이제 문제 해결을 위해 다음과 같이 이전에 제공된 유사한 AWX 사양을 사용하고 있습니다. 보시다시피 AWX 사양에 따라 존재할 것으로 예상되는 PVC awx-projects-claim 생성할 수 있도록 확장해야 합니다.

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: awx-projects-claim
  namespace: default
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: local-path
  resources:
    requests:
      storage: 2Gi
---
apiVersion: awx.ansible.com/v1beta1
kind: AWX
metadata:
  name: awx
spec:
  ingress_type: Ingress
  route_tls_termination_mechanism: edge
  hostname: localhost
  postgres_storage_requirements:
    requests:
      storage: 3Gi
  projects_persistence: true
  projects_existing_claim: awx-projects-claim
  web_resource_requirements:
    requests:
      cpu: 250m
      memory: 2Gi
    limits:
      cpu: 750m
      memory: 4Gi
  task_resource_requirements:
    requests:
      cpu: 250m
      memory: 1Gi
    limits:
      cpu: 500m
      memory: 2Gi
  ee_resource_requirements:
    requests:
      cpu: 250m
      memory: 1Gi
    limits:
      cpu: 500m
      memory: 2Gi

$ kubectl apply -f pg-k3s.yml                                                                                                   23:13:08
persistentvolumeclaim/awx-projects-claim created
awx.awx.ansible.com/awx created

# still pending because POD has not started yet
$ kubectl get pvc                                                                                                               23:14:05
NAME                      STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
awx-projects-claim        Pending                                                                        local-path     23s
postgres-awx-postgres-0   Bound     pvc-3b9e6563-9085-4d79-90ba-fa6c88431c6c   3Gi        RWO            local-path     4s

그런 다음 포드를 보면 충돌이 발생했습니다.

$ sh kubectl get pods -A -w                                                                                       23:07:32
NAMESPACE     NAME                                      READY   STATUS              RESTARTS   AGE
kube-system   local-path-provisioner-5ff76fc89d-4d7bn   1/1     Running             0          11m
kube-system   metrics-server-86cbb8457f-9fkt2           1/1     Running             0          11m
kube-system   coredns-7448499f4d-9t87w                  1/1     Running             0          11m
kube-system   helm-install-traefik-crd-mlrtg            0/1     Completed           0          11m
kube-system   helm-install-traefik-v5n5s                0/1     Completed           1          11m
kube-system   svclb-traefik-c9cgh                       2/2     Running             0          10m
kube-system   traefik-97b44b794-6dz4g                   1/1     Running             0          10m
default       awx-operator-88b886454-9pq7w              0/1     ContainerCreating   0          15s
default       awx-operator-88b886454-9pq7w              1/1     Running             0          16s



default       awx-postgres-0                            0/1     Pending             0          0s
kube-system   helper-pod-create-pvc-3b9e6563-9085-4d79-90ba-fa6c88431c6c   0/1     Pending             0          0s
kube-system   helper-pod-create-pvc-3b9e6563-9085-4d79-90ba-fa6c88431c6c   0/1     ContainerCreating   0          0s
kube-system   helper-pod-create-pvc-3b9e6563-9085-4d79-90ba-fa6c88431c6c   0/1     Completed           0          3s
kube-system   helper-pod-create-pvc-3b9e6563-9085-4d79-90ba-fa6c88431c6c   0/1     Terminating         0          3s
kube-system   helper-pod-create-pvc-3b9e6563-9085-4d79-90ba-fa6c88431c6c   0/1     Terminating         0          3s
default       awx-postgres-0                                               0/1     Pending             0          4s
default       awx-postgres-0                                               0/1     ContainerCreating   0          4s
default       awx-76bdfc954c-jxvll                                         0/4     Pending             0          0s
kube-system   helper-pod-create-pvc-85b1b705-43b3-42a6-a96b-1e79943e99d5   0/1     Pending             0          0s
kube-system   helper-pod-create-pvc-85b1b705-43b3-42a6-a96b-1e79943e99d5   0/1     ContainerCreating   0          0s
default       awx-postgres-0                                               1/1     Running             0          15s
kube-system   helper-pod-create-pvc-85b1b705-43b3-42a6-a96b-1e79943e99d5   0/1     Completed           0          6s
kube-system   helper-pod-create-pvc-85b1b705-43b3-42a6-a96b-1e79943e99d5   0/1     Terminating         0          7s
kube-system   helper-pod-create-pvc-85b1b705-43b3-42a6-a96b-1e79943e99d5   0/1     Terminating         0          7s
default       awx-postgres-0                                               0/1     Error               0          16s
default       awx-76bdfc954c-jxvll                                         0/4     Pending             0          7s
default       awx-76bdfc954c-jxvll                                         0/4     Init:0/1            0          8s
default       awx-postgres-0                                               0/1     Error               1          18s
default       awx-postgres-0                                               0/1     CrashLoopBackOff    1          18s
default       awx-76bdfc954c-jxvll                                         0/4     PodInitializing     0          18s
default       awx-postgres-0                                               1/1     Running             2          35s
default       awx-postgres-0                                               0/1     Error               2          35s
default       awx-postgres-0                                               0/1     CrashLoopBackOff    2          48s
default       awx-postgres-0                                               0/1     Error               3          64s
default       awx-postgres-0                                               0/1     CrashLoopBackOff    3          77s
default       awx-76bdfc954c-jxvll                                         4/4     Running             0          111s
default       awx-postgres-0                                               0/1     CrashLoopBackOff    4          2m11s

따라서 기본적으로 postgres statefulset이 작동하지 않았지만 awx 는 제대로 작동했습니다(물론 데이터베이스 누락으로 인해 작동하지 않음).

$ kubectl get pvc
NAME                      STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
postgres-awx-postgres-0   Bound    pvc-3b9e6563-9085-4d79-90ba-fa6c88431c6c   3Gi        RWO            local-path     2m37s
awx-projects-claim        Bound    pvc-85b1b705-43b3-42a6-a96b-1e79943e99d5   2Gi        RWO            local-path     2m56s

$ kubectl get pods                                                                                       23:17:32
NAME                           READY   STATUS    RESTARTS   AGE
awx-operator-88b886454-9pq7w   1/1     Running   0          8m47s
awx-76bdfc954c-jxvll           4/4     Running   0          3m21s
awx-postgres-0                 0/1     Error     5          3m30s

그런 다음 컨테이너를 보면 로컬 경로 제공자를 사용하여 k3s 에서 동일한 오류가 발생했습니다. https://github.com/ansible/awx-operator/pull/413 과 비슷하게 보이지만 postgresql statefulset에 대해 주소를 지정해야 합니다.

$ kubectl logs awx-postgres-0                                                                       23:26:52
mkdir: cannot create directory ‘/var/lib/postgresql/data’: Permission denied

작업 중입니다.

따라서 기본적으로 데이터베이스를 생성할 수 있도록 권한을 수정하기 위해 `initContainer 접근 방식을 활용해야 합니다. 이 스니펫은 다음 작업을 수행합니다.

diff --git a/roles/installer/tasks/database_configuration.yml b/roles/installer/tasks/database_configuration.yml
index 2e99be5..470530a 100644
--- a/roles/installer/tasks/database_configuration.yml
+++ b/roles/installer/tasks/database_configuration.yml
@@ -80,8 +80,9 @@
 - block:
     - name: Create Database if no database is specified
       k8s:
-        apply: true
+        apply: yes
         definition: "{{ lookup('template', 'postgres.yaml.j2') }}"
+        wait: yes
       register: create_statefulset_result

   rescue:
diff --git a/roles/installer/templates/postgres.yaml.j2 b/roles/installer/templates/postgres.yaml.j2
index d17ee12..f87c842 100644
--- a/roles/installer/templates/postgres.yaml.j2
+++ b/roles/installer/templates/postgres.yaml.j2
@@ -37,10 +37,27 @@ spec:
       imagePullSecrets:
         - name: {{ image_pull_secret }}
 {% endif %}
+      initContainers:
+        - name: init-chmod-data
+          image: '{{ postgres_image }}:{{ postgres_image_version }}'
+          imagePullPolicy: '{{ image_pull_policy }}'
+          command:
+            - /bin/sh
+            - -c
+            - |
+              if [ ! -f {{ postgres_data_path }}/PG_VERSION ]; then
+                chown postgres:root {{ postgres_data_path | dirname }}
+              fi
+          volumeMounts:
+            - name: postgres
+              mountPath: '{{ postgres_data_path | dirname }}'
+              subPath: '{{ postgres_data_path | dirname | basename }}'
       containers:
         - image: '{{ postgres_image }}:{{ postgres_image_version }}'
           imagePullPolicy: '{{ image_pull_policy }}'
           name: postgres
+          securityContext:
+            fsGroup: 999
           env:
             # For postgres_image based on rhel8/postgresql-12
             - name: POSTGRESQL_DATABASE

패치가 적용되면 작동 상태가 됩니다.

$ ubectl get pods -w                              00:38:58
NAME                            READY   STATUS    RESTARTS   AGE
awx-operator-5bc776b4d4-d9ww2   1/1     Running   0          4m41s
awx-postgres-0                  1/1     Running   0          4m3s
awx-d67898cd9-k6jrc             4/4     Running   0          3m48s

$ kubectl iexec awx-postgres-0 /bin/bash                                                                                                                         00:57:00
root@awx-postgres-0:/# namei  -xmolv /var/lib/postgresql/data/pgdata/
f: /var/lib/postgresql/data/pgdata/
Drwxr-xr-x root     root     /
drwxr-xr-x root     root     var
drwxr-xr-x root     root     lib
drwxr-xr-x postgres postgres postgresql
Drwx------ postgres root     data
drwx------ postgres root     pgdata

PR을 작성하겠습니다. @flisak-robert 및 @scott-vick 문제를 보고해 주셔서 감사합니다.

업데이트 전에 임시 해결책이 있습니까?

업데이트 전에 임시 해결책이 있습니까?

귀하의 요구에 맞는지 모르겠지만 도커 컨테이너에서 postgres를 실행하고 대신 해당 postgres 인스턴스를 사용하도록 awx를 지정했습니다.
내 구성은 다음과 같습니다.

apiVersion: v1
kind: Secret
metadata:
  name: awx-postgres-configuration
  namespace: awx
stringData:
  host: <postgres address>
  port: "5432"
  database: awx
  username: postgres
  password: <postgres password>
  type: unmanaged
type: Opaque

awx 구성에 postgres_configuration_secret: awx-postgres-configuration 를 포함하는 것을 잊지 마십시오. 그렇지 않으면 예를 들어 awx 노드를 다시 시작할 때 AWX가 postgres 데이터베이스의 내용을 해독할 수 없습니다. 거기에 있었어, 해냈어 :(

내가 찾은 해결 방법은 PV를 만드는 것입니다.

apiVersion: v1
kind: PersistentVolume
metadata:
  name: task-pv-volume
  labels:
    type: local
spec:
  storageClassName: <className>
  capacity:
    storage: 10Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: "<path>"

경로에는 chmod를 통해 777 권한이 부여됩니다.
storageClassName과 동일한 값으로 postgres_storage_class 속성을 추가해야 합니다.