Kubeadm: Can not upgrade kube-apiserver from private registry.

Created on 2 Oct 2017  ·  4Comments  ·  Source: kubernetes/kubeadm

I checked this issue, #71 , it seems I can not add credentials if kube-apiserver is not running, but after manually pull the private image, and restart kubelet I still get the error.

PullImage "docker.kesci.com:5000/kube-apiserver:latest" from image service failed: rpc error: code = Unknown desc = Error response from daemon: Get https://**my.private.image.registry**/v2/kube-apiserver/manifests/latest: no basic auth credentials

Versions

kubeadm version (use kubeadm version):

Environment:

  • Kubernetes version (use kubectl version):
    Client Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.5", GitCommit:"17d7182a7ccbb167074be7a87f0a68bd00d58d97", GitTreeState:"clean", BuildDate:"2017-08-31T09:14:02Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}
  • Cloud provider or hardware configuration:
    aws
  • OS (e.g. from /etc/os-release):
NAME="Ubuntu"
VERSION="16.04.3 LTS (Xenial Xerus)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 16.04.3 LTS"
VERSION_ID="16.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
VERSION_CODENAME=xenial
UBUNTU_CODENAME=xenial
  • Kernel (e.g. uname -a):
    Linux ip-172-31-28-254 4.4.0-92-generic #115-Ubuntu SMP Thu Aug 10 09:04:33 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
  • Others:

What happened?

I manually replace the image of /etc/kubernetes/manifests/kube-apiserver.yaml configuration, and kubelet got error container start failed: ErrImagePull: rpc error: code = Unknown desc = Error response from daemon: Get https://my.private.image.registry/v2/kube-apiserver/manifests/latest: no basic auth credentials，it seems no credentials configured, so I manually docker pull the image
docker pull my.private.image.registry/kube-apiserver:latest and run service kubelet restart, still got this error.

What you expected to happen?

kubelet support manually configure credentials or, it can recognize local pulled images.

How to reproduce it (as minimally and precisely as possible)?

Replace the /etc/kubernetes/manifests/kube-apiserver.yaml image with a private registry image.

kinsupport
Source
picture ggaaooppeenngg

Most helpful comment

@ggaaooppeenngg you should try this:

  1. Pre pull images with docker pull
  2. Use kubeadm init --config yourconfigfile, having care to set the imageRepository option in the config file

Please note that:

  • you have to pre pull images, because there is no way to create an imagePullSecrets before the cluster exists 😄
  • the procedure above works in kubeadm v1.8; I don't remeber if this was supported as well in previous releases :confused:
  • this procedure sets your repo in yaml files without requiring manual changes 👍; however, it is expected that all local images follow a standard naming convention (e.g. componentname-architecture:version)

For more info you can have a look at https://kubernetes.io/docs/admin/kubeadm/

picture fabriziopandini on 7 Oct 2017
👍4

All 4 comments

My solution: docker pull the image and set imagePullPolicy to Never, then restart kubelet.

ggaaooppeenngg picture ggaaooppeenngg on 5 Oct 2017

How do you expect kubeadm to act in this situation? We use official images and do not support custom repos. What is your use case for using a custom image?

jamiehannaford picture jamiehannaford on 6 Oct 2017

@ggaaooppeenngg you should try this:

  1. Pre pull images with docker pull
  2. Use kubeadm init --config yourconfigfile, having care to set the imageRepository option in the config file

Please note that:

  • you have to pre pull images, because there is no way to create an imagePullSecrets before the cluster exists 😄
  • the procedure above works in kubeadm v1.8; I don't remeber if this was supported as well in previous releases :confused:
  • this procedure sets your repo in yaml files without requiring manual changes 👍; however, it is expected that all local images follow a standard naming convention (e.g. componentname-architecture:version)

For more info you can have a look at https://kubernetes.io/docs/admin/kubeadm/

fabriziopandini picture fabriziopandini on 7 Oct 2017
👍4

@jamiehannaford I want to dig some performance issues, and add some tracing logs, so that I can have a check.

ggaaooppeenngg picture ggaaooppeenngg on 9 Oct 2017
Was this page helpful?
0 / 5 - 0 ratings

Related issues

yagonobre picture yagonobre  ·  4Comments
danderson picture danderson  ·  3Comments
bruceauyeung picture bruceauyeung  ·  4Comments
ep4eg picture ep4eg  ·  3Comments
shenshouer picture shenshouer  ·  3Comments