Kubernetes: Failed to pull image with "x509: certificate signed by unknown authority" error

I have asked the question here: http://stackoverflow.com/q/43150437/969784

Assuming you're using a self signed certificate, your CA still needs to get added in your local trust store even if you're using --skip-tls-verify.

@rushilpaul

• First --insecure-skip-tls-verify is not a valid argument for kubectl create;
• Actually x509 error is on docker side. The daemon failed to pull image from that insecure registry. You can refer to insecure docker registry on how to trust/skip the registry security.

@dixudx I forgot to mention it. I installed the server certificate globally on this kubernetes master node and then restarted the docker service running on it. After that I am successfully able to manually pull that image using docker pull 10.78.0.228:5000/monitorms. Before that I was getting this error message while doing a manual pull of that image.

Is the error coming because the Kubernetes nodes don't have the certificate installed?

@dixudx Also, the kubectl options lists --insecure-skip-tls-verify as one of the "global" options and says that it can be passed to any Kubernetes command

--insecure-skip-tls-verify just skips the server's certificate verification, not docker registry, so it can not solve the problem. The error is from Docker daemon while pulling image.

I installed the server certificate globally on this kubernetes master node and then restarted the docker service running on it.

Maybe you should try the command docker pull 10.78.0.228:5000/monitorms on the k8s nodes which hold the pod, not the k8s master.

That is a valid arg to kubectl create but just controls trust between kubectl and the API server

The pull error is between the node and the docker registry. The node either needs to trust the certificate or treat that registry as an untrusted registry (which makes the node tolerate TLS verification errors)

@supereagle I am going to add the insecure registry option to the docker configuration file on the k8s nodes. Hopefully that should do the trick

You would think that this is solved by now.

CA Certificates

Actual recorded cases of preventing unauthorized access : ZERO
Amount of countless of developer time wasted because of tooling that don't integrate CA certs into their tooling properly: gasmillions of man hours.

Moral of the story. Ditch CA certs. Such a ballache every time you have try to get tooling to work together. Nobody knows how it works. Nobody. Software that use it never work. In the end you just copy all the certs to every machine and your toaster just so you don't get that god damn x509: certificate signed by unknown authority bullshit error every bloody time you try to do any tooling.

Now I have to go and drill right to the core of this cluster to get those certs installed, because kubernetes's secrets for handling docker is just plain useless.

Just use the money that would have been spent trying to get the bloody CA certs to work and hire a henchman with an axe that cut the hardlines when the hacker comes. This CA certs it's not security if it does not let authorized people in because the entire field is just one giant BUG that will JAM your tooling.

/sig auth

should anyone face it while using directly the gcr.io, one possible situation is that CA certificates on your machine are too old.

docker pull gcr.io/google_containers/kube-apiserver-amd64:v1.7.2
Trying to pull repository gcr.io/google_containers/kube-apiserver-amd64 ...
Get https://gcr.io/v1/_ping: x509: certificate signed by unknown authority '

solution that worked for me on RH/CentOS:

yum check-update ca-certificates; (($?==100)) && yum update ca-certificates || yum reinstall ca-certificates
update-ca-trust extract

cc @kubernetes/sig-node-bugs for image pulling issue

if you go to the node in question and try curl -v https://gcr.io/v1/_ping, do you see a successful response? if so, then there may be an issue with the way the node is pulling the images. if not, then you simply need to update the root certs on that node

Any update on this issue? this is hitting us now.

@srossross-tableau

As far as I remember this was a docker issue, not a kubernetes one. Docker does not use linux's ca certs. Nobody knows why.

You have to install those certs manually (on every node that could spawn those pods) so that docker can use them:

/etc/docker/certs.d/mydomain.com:1234/ca.crt

This is a highly annoying issue as you have to butcher your nodes after bootstrapping to get those certs in there. And kubernetes spawns nodes all the time. How this issue has not been solved yet is a mystery to me. It's a complete showstopper IMO.

This should really be solved using the secrets mechanism of kubernetes. But somehow it is not. Who knows!?

@pompomJuice, could this be a minikube image issue? I am not able to even curl this site

minikube ssh -- curl -I https://storage.googleapis.com
curl: (60) SSL certificate problem: self signed certificate in certificate chain

$minikube logs
...
Nov 08 18:19:06 minikube localkube[3032]: E1108 18:19:06.788101    3032 remote_image.go:108] PullImage "gcr.io/google_containers/heapster:v1.3.0" from image service failed: rpc error: code = 2 desc = error pulling image configuration: Get https://storage.googleapis.com/artifacts.google-containers.appspot.com/containers/images/sha256:f9d33bedfed3f1533f734a73718c15976fbd37f04f383087f35e5ebd91b18d1e: x509: certificate signed by unknown authority
..

Exactly my point. That curl error is just plain wrong. It is telling you that you have the certificates but they are self signed. I find that highly unlikely. (Unless you hacked them in there somehow)

That means that that error message is just plain wrong. This connects with my previous point that almost nobody implements this stuff correctly.

Try to update the certs on that box like ReSearchITEng suggested above.

I'm running into the same issue. Certs are from digicert, kubernetes cluster running in GCE, certs installed through the host and put in /etc/docker/certs.d/, and still x509 error.

Docker logs:
TLS handshake error from XXXXXXXXXX: remote error: tls: bad certificate

Kub version:
Client Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.4", GitCommit:"9befc2b8928a9426501d3bf62f72849d5cbcd5a3", GitTreeState:"clean", BuildDate:"2017-11-20T05:28:34Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}

host:
NAME="Ubuntu"
VERSION="16.04.3 LTS (Xenial Xerus)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 16.04.3 LTS"
VERSION_ID="16.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
VERSION_CODENAME=xenial
UBUNTU_CODENAME=xenial

Paste the entire folder name in '/etc/docker/certs.d/' please. And the filenames of the certs.

It should work if all your nodes have that cert installed.

root@kubernetes-minion-group-96k7:/etc/docker/certs.d/"foo.bar.com":5000# ll
total 16
drwxr-xr-x 2 root root 4096 Dec 2 20:43 ./
drwxr-xr-x 3 root root 4096 Dec 2 20:07 ../
-rw-r--r-- 1 root root 3332 Dec 2 20:23 domain.crt
-rw-r--r-- 1 root root 1675 Dec 2 20:43 domain.key

So far only one node in the cluster :)

Change those to ca.crt and client.key

Like here: https://docs.docker.com/engine/security/certificates/#creating-the-client-certificates

Changed them to ca.crt and ca.key both in the directory, as well as updated the files called out in the secret . I restarted docker service on the node and redeployed the pods and still, same error.

Here's more info from curl:

curl -vvI https://foo.bar.com:5000/v2/

• Trying XXX.XXX.XXX.XXX...
• TCP_NODELAY set
• Connected to foo.bar.com (XXX.XXX.XXX.XXX) port 5000 (#0)
• ALPN, offering h2
• ALPN, offering http/1.1
• Cipher selection: PROFILE=SYSTEM
• successfully set certificate verify locations:
• CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
• TLSv1.2 (OUT), TLS handshake, Client hello (1):
• TLSv1.2 (IN), TLS handshake, Server hello (2):
• TLSv1.2 (IN), TLS handshake, Certificate (11):
• TLSv1.2 (OUT), TLS alert, Server hello (2):
• SSL certificate problem: unable to get local issuer certificate
• stopped the pause stream!
• Closing connection 0
  curl: (60) SSL certificate problem: unable to get local issuer certificate
  More details here: https://curl.haxx.se/docs/sslcerts.html

curl performs SSL certificate verification by default, using a "bundle"
of Certificate Authority (CA) public keys (CA certs). If the default
bundle file isn't adequate, you can specify an alternate file
using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
the bundle, the certificate verification probably failed due to a
problem with the certificate (it might be expired, or the name might
not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
the -k (or --insecure) option.
HTTPS-proxy has similar options --proxy-cacert and --proxy-insecure.

I made a mistake, the one should be client.key not ca.key.

It should work.

Double check by trying to start the image manually on the box.

That didn't seem to work either :( same error

What happens when you try to start the docker container manually from command line?

should I use kubectl run on one of the nodes or docker run? docker run, the container starts but I get connection refused. If I use kubctl, error: failed to discover supported resources: an error on the server ("") has prevented the request from succeeding

If I used kubectl on my local machine that's leveraging kubectl proxy, the container starts but I get the following:
http: server gave HTTP response to HTTPS client

kubectl command: kubectl run --image=registry:2 devreg-test2 --port=5000 --env="DOMAIN=cluster, REGISTRY_HTTP_ADDR=0.0.0.0:5000, REGISTRY_HTTP_TLS_CERTIFICATE=/certs/ca.crt, REGISTRY_HTTP_TLS_KEY=/certs/client.key" --expose=true

Try the following.

Make your own docker registry. Use gitlab for this it is free.

Host some images on it on http. Try to start a pod with this image. Then verify that the docker you are looking at is in fact running that pod. If it is then you know you have the correct node.

Then like before docker run and explain to me what you mean by connection refused.

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten
/remove-lifecycle stale

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

So whats the workaround/fix on this? Im still getting it after upgrading from 3.9 to 3.10. Failed to pull image "docker-registry.default.svc:5000/openshift/mysql@sha256:dfd9f18f47caf290... and with errormessage: v2/: x509: certificate signed by unknown authority. Im agreeing with @pompomJuice. A permanent fix that doesnt break after install/upgrades is needed or reengineer this completly. If not this is not ready for production workloads.

working solution for pulling docker image on ubuntu from Artifactory (certificate is selfsigned):

• put all used (if there is more than one root ca) ca certs in /usr/local/share/ca-certificates
• run update-ca-certificates
• restart docker daemon (sudo service docker restart)

should anyone face it while using directly the gcr.io, one possible situation is that CA certificates on your machine are too old.

docker pull gcr.io/google_containers/kube-apiserver-amd64:v1.7.2
Trying to pull repository gcr.io/google_containers/kube-apiserver-amd64 ...
Get https://gcr.io/v1/_ping: x509: certificate signed by unknown authority '

solution that worked for me on RH/CentOS:

yum check-update ca-certificates; (($?==100)) && yum update ca-certificates || yum reinstall ca-certificates
update-ca-trust extract

This actually worked for me.

I run kubernetes on RancherOS as part of Rancher 2.x setup and have private registry that is not internet facing, so I have to use self-signed certificate on it, resulting in x509 error. I read this thread and few others and the solved the issue - sharing in case it may help someone, if not directly then by suggesting possible path.

This worked for me - https://www.ctrl-alt-del.cc/2018/11/solution-rancher-2-k8s-private-registry.html

2020 and still the same issue.
private harbor registry.
docker pull works with no problem.
ls /etc/docker/certs.d/registry.myharbor.com/ shows up the certificate.
kubernetes fails to pull images with imagepullbackoff error.
It is 3 years and kubernetes still has this issue. Very disappointing.

Solved

1. Make sure you are able to do a docker pull IMAGENAME from the machine where you are running the deployments (yaml files, helm packages etc.,)
2. On all the kubernetes nodes make sure the following is present /etc/docker/certs.d/my-private-registry.com/my-private-registry.com.crt

I am working in my local dev environment

    OS:
       Ubuntu (bionic) 18.0.4 LTS    
    Minikube Version:
       v1.11.0
    Docker Version:
       19.03.10

I am using Jfrog Container Registry as registry to my minikube. I am able do the following:

1. docker login localhost:443 | or | ip-add:443
2. docker push ip-add:443/docker-local/test:latest
3. docker pull ip-add:443/docker-local/test:latest

I have configured Jfrog Container Registry to run behind Nginx Reverse Proxy listening on port 443. Created self-signed certs and Jfrog is using these certs.

Configured docker to use the self-signed certs as follows.

1. Create certs, copy it to /usr/local/share/ca-certificates/
2. sudo update-ca-certificates
3. copy the certificate to /etc/docker/cert.d/192.168.0.114:443/ca.crt
4. restarted the docker, just be sure

Configure K8 to use the docker login secret by .yaml file as following:

1. base64 encode ~/.docker/config.json
2. use it in the following template
   apiVersion: v1 kind: Secret metadata: name: myregistrykey namespace: awesomeapps data: .dockerconfigjson: UmVhbGx5IHJlYWxseSByZWVlZWVlZWVlZWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWxsbGxsbGxsbGxsbGxsbGxsbGxsbGxsbGxsbGxsbGx5eXl5eXl5eXl5eXl5eXl5eXl5eSBsbGxsbGxsbGxsbGxsbG9vb29vb29vb29vb29vb29vb29vb29vb29vb25ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubmdnZ2dnZ2dnZ2dnZ2dnZ2dnZ2cgYXV0aCBrZXlzCg== type: kubernetes.io/dockerconfigjson

In the deployment.yaml, I use ImagePullSecrets and the name flag.

Now after all this setup where the docker pull is working on terminal, I get error on the pods saying x509 IP Sans.

I went through lot of documentation and K8 issues, replicated the step of https://github.com/kubernetes/kubernetes/issues/43924#issuecomment-631533150

replicated the steps didn't work out. Can anyone let me know what I am doing wrong? and how can I correct it.

I am also having the same issues, but on EKS in this case. I am using a daemonset to deploy a privileged workload to attempt the fixes above on all nodes to solve this problem (images are on a private signed registry).