/cc @lukemarsden @luxas @mikedanese

If we revert #43474 completely, we are in a situation again where we break 0.2.0 CNI plugins (see https://github.com/kubernetes/kubernetes/issues/43014)

Should we consider doing something like https://github.com/kubernetes/kubernetes/pull/43284?

Also /cc @thockin

/cc @kubernetes/sig-network-bugs

Ref: https://github.com/kubernetes/kubernetes/issues/43397#issuecomment-289978351

@jbeda can I get some kubelet logs with --loglevel=5?

@yujuhong -- you mention that you think that this is working as intended. Regardless, kubeadm was depending on this behavior. We introduced a breaking change with #43474. We can talk about the right way to fix this for 1.7 but, for now, we need to get kubeadm working again.

Slack discussion ongoing now -- https://kubernetes.slack.com/archives/C09QYUH5W/p1490803144368246

It looks like DaemonSets will still get scheduled even if the node is not ready. This is really, in this case, kubeadm being a little too paranoid.

The current plan that we are going to test out is to have kubeadm no longer wait for the master node to be ready but instead just have it be registered. This should be good enough to let a CNI DaemonSet be scheduled to set up CNI.

@kensimon is testing this out.

@jbeda yeah, looks like the DaemonSet controller will still enqueue them mainly because it's completely ignorant of network-iness. We should really fix this more generally. Is there anything immediate to do in kube or is it all in kubeadm for now?

I'm trying to install kubernetes with kubeadm on Ubuntu 16.04. Is there a quick fix for this?

@jbeda if you have a patched version happy to test it..

I have kubeadm getting past the node's NotReady status, but the dummy deployment it creates isn't working due to the node.alpha.kubernetes.io/notReady taint preventing it from running. Adding tolerations doesn't seem to help, I'm not exactly sure how to proceed at this point. Can anybody shed some light on how to deploy a pod that tolerates the notReady taint?

I'm exploring some other options like not marking the node as notReady, but it's not clear that's what we want to do.

we worked around it by removing KUBELET_NETWORK_ARGS from kubelet command line. after that kubeadm init worked fine and we were able to install canal cni plugin.

@sbezverk would you please describe how to do that?

Can confirm @sbezverk (good find :) ) findings, adjusting /etc/systemd/system/10-kubeadm.conf and removing KUBELET_NETWORK_ARGS will make it run on centos. Tested with weave.

@overip you need to edit /etc/systemd/system/kubelet.service.d/10-kubeadm.conf

ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_SYSTEM_PODS_ARGS $KUBELET_NETWORK_ARGS $KUBELET_DNS_ARGS $KUBELET_AUTHZ_ARGS $KUBELET_EXTRA_ARGS

remove $KUBELET_NETWORK_ARGS

and then restart kubelet after that kubeadm init should work.

this is what i did

kubeadm reset

remove ENV entries from:

/etc/systemd/system/kubelet.service.d/10-kubeadm.conf

reload systemd and kube services

systemctl daemon-reload
systemctl restart kubelet.service

re-run init

kubeadm init

All correct, and while we're at it

If you see this:
kubelet: error: failed to run Kubelet: failed to create kubelet: misconfiguration: kubelet cgroup driver: "cgroupfs" is different from docker cgroup driver: "systemd"

you have to edit your /etc/systemd/system/kubelet.service.d/10-kubeadm.conf and add the flag --cgroup-driver="systemd"

and do as above

kuebadm reset
systemctl daemon-reload
systemctl restart kubelet.service
kubeadm init.

I'd be careful removing --network-plugin=cni from the kubelet CLI flags, this causes kubelet to use the no_op plugin by default... I would be surprised if common plugins like calico/weave would even work in this case (but then again my understanding of how these plugins operate underneath is a bit limited.)

@kensimon hm, have not seen any issues on my setup, I deployed canal cni plugin and it worked fine..

@sbezverk Is cross host networking also working well?

@resouer cannot confirm, I have 1.6.0 only as All-In-One.

@resouer @sbezverk I succesfully joined a machine.

 [root@deploy-01 x86_64]# kubectl get nodes
 NAME        STATUS    AGE       VERSION
 deploy-01   Ready     51m       v1.6.0
 master-01   Ready     4m        v1.6.0

``` [root@deploy-01 x86_64]# kubectl get pods -n kube-system
NAME READY STATUS RESTARTS AGE
etcd-deploy-01 1/1 Running 0 50m
kube-apiserver-deploy-01 1/1 Running 0 51m
kube-controller-manager-deploy-01 1/1 Running 0 50m
kube-dns-3913472980-6plgh 3/3 Running 0 51m
kube-proxy-mbvdh 1/1 Running 0 4m
kube-proxy-rmp36 1/1 Running 0 51m
kube-scheduler-deploy-01 1/1 Running 0 50m
kubernetes-dashboard-2396447444-fm8cz 1/1 Running 0 24m
weave-net-3t487 2/2 Running 0 44m
weave-net-hhcqp 2/2 Running 0 4m

```

workaround works but can't get flannel going...

@stevenbower worst case scenario, you can put back this setting and restart kubelet when you are done with kubeadm business..

I got a three node cluster with weave working. Not sure how stable this might be with this hack, but thanks anyway! :smiley:

On a side node, you can put back the $KUBELET_NETWORK_ARGS, after the init on the master passes. I actually did not remove it on the machine I joined, only the cgroup-driver, otherwise kubelet and docker won't work together.

But you don't have to kubeadm reset, just change /etc/systemd/system/kubelet.service.d/10-kubeadm.conf and do the systemctl dance:

systemctl daemon-reload
systemctl restart kubelet.service

To those of you who are dropping KUBELET_NETWORK_ARGS and reporting it works for you:

For me, I have 2 clusters up: one where I applied the patch from https://github.com/kubernetes/kubernetes/pull/43824 and let kubeadm proceed normally on initialization, and one with KUBELET_NETWORK_ARGS deleted. On the cluster with KUBELET_NETWORK_ARGS removed, any traffic between pods fails.

On a cluster with KUBELET_NETWORK_ARGS removed:

$ kubectl run nginx --image=nginx
deployment "nginx" created
$ kubectl expose deployment nginx --port 80
service "nginx" exposed
$ kubectl run --rm -i -t ephemeral --image=busybox -- /bin/sh -l
If you don't see a command prompt, try pressing enter.
/ # wget nginx.default.svc.cluster.local
wget: bad address 'nginx.default.svc.cluster.local'

On a cluster with normal KUBELET_NETWORK_ARGS but with a patched kubeadm:

$ kubectl run nginx --image=nginx          
deployment "nginx" created
$ kubectl expose deployment nginx --port 80
service "nginx" exposed
$ kubectl run --rm -i -t ephemeral --image=busybox -- /bin/sh -l
If you don't see a command prompt, try pressing enter.
/ # wget nginx.default.svc.cluster.local
Connecting to nginx.default.svc.cluster.local (10.109.159.41:80)
index.html           100% |********************************************************************************************|   612   0:00:00 ETA

If you're one of those who disabled KUBELET_NETWORK_ARGS, check if the above works for you.

I suggest that we drop both the node ready and the dummy deployment check
altogether for 1.6 and move them to a validation phase for 1.7.

On Mar 29, 2017 10:13 AM, "Dan Williams" notifications@github.com wrote:

@jbeda https://github.com/jbeda yeah, looks like the DaemonSet
controller will still enqueue them mainly because it's completely ignorant
of network-iness. We should really fix this more generally. Is there
anything immediate to do in kube or is it all in kubeadm for now?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/kubernetes/kubernetes/issues/43815#issuecomment-290158416,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABtFIUw8GIJVfHrecB3qpTLT8Q4AyLVjks5rqpFKgaJpZM4MtMRe
.

Anyone else running Ubuntu 16.04? I've removed the KUBELET_NETWORK_ARGSfrom the systemd service and reloaded the systemd daemon. I can turnup a master node but cannot join a node. It fails with the error The requested resource is unavailable

KUBELET_NETWORK_ARGS removed, any traffic between pods fails.

I wouldn't be surprised since KUBELET_NETWORK_ARGS tells kubelet what plugin to use, where to look for the configuration and binaries. You NEED them.

I recommend #43835 (and the 1.6 cherry pick #43837) as the fix we make for 1.6. I tested both and they work. I've assigned @jbeda and @luxas to review when they wake up.

Both of those PRs look reasonable. ButI think we should look at going with https://github.com/kubernetes/kubernetes/pull/43824 instead. While it is a bit more complicated, it does preserve that code path so that users that preconfigure CNI outside of using a Daemonset (I do this in https://github.com/jbeda/kubeadm-gce-tf though I haven't updated it to 1.6) still wait for nodes to be ready.

As a bonus, this is @kensimon's first PR to Kubernetes and he has pulled out the stops to test this stuff out. But, to be honest, they are both workable and I really want to see it fixed. :)

Sorry I missed https://github.com/kubernetes/kubernetes/pull/43824. I'm also happy with either if they both work.

I'm also happy with either if they both work too

@kensimon It works for me, when I only disable KUBELET_NETWORK_ARGS during kubadm init. Thanks to you instructions I could verify that.

Confirmed @webwurst for it working when you only disable KUBELET_NETWORK_ARGS during kubadm init. I had to restart kube-dns though for it to pick it up. The check from @kensimon works, dns resolves.

Although I agree that this a terrible hack, and too horrible for most people to follow, looking at the slack channels.

A better solution is presented by the patches from @kensimon or @mikedanese.

@coeki how exactly did you restart kube-dns. I tried kubectl delete pod kube-dns-3913472980-l771x --namespace=kube-system and now kube-dns stays on pending kube-system kube-dns-3913472980-7jrwm 0/3 Pending 0 4m
It did exactly as described: remove KUBELET_NETWORK_ARGS, sudo systemctl daemon-reload && sudo systemctl restart kubelet.service, kubeadm init, add KUBELET_NETWORK_ARGS, again sudo systemctl daemon-reload && sudo systemctl restart kubelet.service
But then my master stays in NotReady. In describe I get

Conditions:
  Type          Status  LastHeartbeatTime           LastTransitionTime          Reason              Message
  ----          ------  -----------------           ------------------          ------              -------
...
KubeletNotReady         runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized

I tried the kube-dns restart as described above, but no success. Any idea? I'm dying on this, trying to get our cluster running again after a failed 1.6.0 upgrade this morging :(

@patte So I just delete pods kube-dns-3913472980-3ljfx -n kube-system And then kube-dns comes up again.

Did you after kubeadm init, add KUBELET_NETWORK_ARGS, again sudo systemctl daemon-reload && sudo systemctl restart kubelet.service install a pod network, like weave or calico? Add that first, you should be able to get it to work.

I tried and tested on centos7 and just did it on ubuntu/xenial, so should work.

To recap what I did:

remove KUBELET_NETWORK_ARGS
sudo systemctl daemon-reload && sudo systemctl restart kubelet.service

kubeadm init --token=$TOKEN --apiserver-advertise-address=$(your apiserver ip address)

add KUBELET_NETWORK_ARGS again
sudo systemctl daemon-reload && sudo systemctl restart kubelet.service

kubectl apply -f https://git.io/weave-kube-1.6

Joined a machine, typically have to add a static route to 10.96.0.0 (cluster ip) as I am on vagrant.
Use with the $TOKEN, but this step is extra.

Then:

delete pods kube-dns-3913472980-3ljfx -n kube-system

Wait for it

kubectl run nginx --image=nginx
kubectl expose deployment nginx --port 80
kubectl run --rm -i -t ephemeral --image=busybox -- /bin/sh -l
/ # wget nginx.default.svc.cluster.local Connecting to nginx.default.svc.cluster.local (10.101.169.36:80) index.html 100% |***********************************************************************| 612 0:00:00 ETA

Works for me, although it's a horrible hack ;)

I am really surprised that kubernetes development community has not provided any ETA for an official fix. I mean this is a horrible bug which should be easily get caught during the code testing. Since it has not, at the very least, 1.6.1 should be pushed asap with the fix so people would stop hacking their clusters and start doing productive things ;). Am I wrong here?

Hey all,

I was a little distracted this week, and this is a long thread full of
kubeadm stuff I don't know that well. Can someone summarize for me? I
think I get the gist of the bug, but what are the proposed solutions and
what makes them horrible?

On Thu, Mar 30, 2017 at 8:13 AM, Serguei Bezverkhi <[email protected]

wrote:

I am really surprised that kubernetes development community has not
provided any ETA for an official fix. I mean this is a horrible bug which
should be easily get caught during the code testing. Since it has not, at
the very least, 1.6.1 should be pushed asap with the fix so people would
stop hacking their clusters and start doing productive things ;). Am I
wrong here?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/kubernetes/kubernetes/issues/43815#issuecomment-290442315,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AFVgVEoKuUf28VazmUApsnyfGhuAhZIqks5rq8aLgaJpZM4MtMRe
.

A change in kubelet (#43474) caused kubelet to start correctly reporting network not ready before the cni plugin was initialized. This broke some ordering that we were depending on and caused a deadlock in kubeadm master initialization. We didn't catch it because kubeadm e2e tests had been broken for a few days leading up to this change.

Current proposed fixes are #43824 and #43835.

ok, That was what I understood. The interlock between network plugin
coming up and node readiness is a little awful right now.

On Thu, Mar 30, 2017 at 8:28 AM, Mike Danese notifications@github.com
wrote:

A change in kubelet (#43474
https://github.com/kubernetes/kubernetes/pull/43474) caused kubelet to
start correctly reporting network not ready before the cni plugin was
initialized. This broke some ordering that we were depending on and caused
a deadlock in kubeadm master initialization. We didn't catch it because
kubeadm e2e tests had been broken for a few days leading up to this change.

Current proposed fixes are #43824
https://github.com/kubernetes/kubernetes/pull/43824 and #43835
https://github.com/kubernetes/kubernetes/pull/43835.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/kubernetes/kubernetes/issues/43815#issuecomment-290447309,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AFVgVLzQLxeV6qp1Rw1fNALDDaf-Sktyks5rq8o2gaJpZM4MtMRe
.

I still prefer #43835. It's a simpler change, I don't think the e2e checks should be done where they are, and there are reports of #43824 not working still. I'm going to push to get this resolved today.

+1 to resolve it today as lots of efforts are wasted on dealing with collateral from the workaround.

I can't believe nobody really tried kubeadm 1.6.0 before 1.6.0 was released....

And, kubelet 1.5.6 + kubeadm 1.5.6 are also broken, /etc/systemd/system/kubelet.service.d/10-kubeadm.conf references /etc/kubernetes/pki/ca.crt but kubeadm doesn't generate ca.crt, there is ca.pem although.

Currently 1.6.0 and 1.5.6 are the only left releases in k8s apt repository...

"broken out of the box", words learned today.

I still prefer #43835. It's a simpler change, I don't think the e2e checks should be done where they are, and there are reports of #43824 not working still. I'm going to push to get this resolved today.

Agree with Mike on this one. #43835 is the simpler change, and validation (if needed) can be done in a separate phase.

@thockin we really need finer-grained conditions and status for networking, especially with hostNetwork:true. Nodes can be ready for some pods, but not others.

We can't use nodeNetworkUnavailable, becuase that's specific to cloud providers. We probably need another one, or a way for the scheduler to allow hostNetwork:true pods on nodes with NetworkReady:false, or making the taints work for unready nodes. And working kubeadm e2e tests :(

Agree. I have been delaying the problem because I had no great answers,
but we need to get this in 1.7

On Thu, Mar 30, 2017 at 10:02 AM, Dan Williams notifications@github.com
wrote:

@thockin https://github.com/thockin we really need finer-grained
conditions and status for networking, especially with hostNetwork:true.
Nodes can be ready for some pods, but not others.

We can't use nodeNetworkUnavailable, becuase that's specific to cloud
providers. We probably need another one, or a way for the scheduler to
allow hostNetwork:true pods on nodes with NetworkReady:false, or making the
taints work for unready nodes.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/kubernetes/kubernetes/issues/43815#issuecomment-290475480,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AFVgVAaO9c76_R8me9PDo96AQ1ZrplMxks5rq-A1gaJpZM4MtMRe
.

@thockin any preferred solution? We can either block node readiness and find out who uses IsNodeReady() like we've already done, or we could add another node condition and then fix up who-knows-how-many bits of the codebase elsewhere for it. Or maybe there's another option.

We can't use nodeNetworkUnavailable, becuase that's specific to cloud providers. We probably need another one, or a way for the scheduler to allow hostNetwork:true pods on nodes with NetworkReady:false, or making the taints work for unready nodes.

I think the scheduler can be fixed to respect taints and tolerations, instead of ruling out all unready nodes completely.
Someone will have to apply the toleration on the hostNetwork:true pods though. I don't know who but it should not be the scheduler since teaching scheduler to understand the pod spec seems way too much.

/cc @davidopp @dchen1107

Also, to anyone who has tried my or michael's patches and is getting hung up on the control plane coming up, it appears as though building a kubeadm from master doesn't work when having it manage a v1.6.0 cluster, due to kubeadm trying to launch kube-apiserver with invalid args:

unknown flag: --proxy-client-cert-file

If you want to test mine or michael's changes against a v1.6.0 cluster, cherry-pick them against v1.6.0 instead of building on top of master for now.

@yujuhong the scheduler already auto-applies tolerations to pods (see https://github.com/kubernetes/kubernetes/pull/41414), this seems like a similar enough use case

I don't have a clear picture of all of the taints and readiness wrt
network, at this point. Dan, do you have a half hour to write up the
current state, so we can start to tease it apart? I think you know it
best. I do feel like we have a number of granular issues that we have just
blanket-covered so far.

On Thu, Mar 30, 2017 at 10:22 AM, Ken Simon notifications@github.com
wrote:

@yujuhong https://github.com/yujuhong the scheduler already
auto-applies tolerations to pods (see #41414
https://github.com/kubernetes/kubernetes/pull/41414), this seems like a
similar enough use case

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/kubernetes/kubernetes/issues/43815#issuecomment-290481017,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AFVgVMas2IP5X5YA0V5626UlGbaS98jtks5rq-TCgaJpZM4MtMRe
.

@thockin I can do that, where should I put it?

there are a couple related issues, can we re-title one of those, post a
summary of current state and go from there?

On Thu, Mar 30, 2017 at 10:50 AM, Dan Williams notifications@github.com
wrote:

@thockin https://github.com/thockin I can do that, where should I put
it?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/kubernetes/kubernetes/issues/43815#issuecomment-290489157,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AFVgVCXJYk9J0QmjTqQA6PPVuhzMHlLZks5rq-t-gaJpZM4MtMRe
.

Do we have anytime line as to when this fixed will be ported to the CentOS repository ?

In 1.6 by default core Kubernetes does not automatically apply any taints to nodes. The scheduler does respect taints that are added by deployment systems like kubeadm, kops, etc. or added manually by users (and respects the corresponding tolerations that are added to pods).

In 1.6 by default the Kubernetes scheduler excludes nodes from consideration if they are reporting certain node conditions, including network unavailable. The code for that is here. It has nothing to do with taints -- it's just looking at the node condition.

In 1.6 you can enable an alpha feature that causes core Kubernetes (the NodeController) to add a NoExecute taint whenever there is NodeReady == "False" or NodeReady == "Unknown". This will cause pods running on the node to be evicted (if they don't tolerate the taint) and prevent new pods from scheduling onto the node (if they don't tolerate the taint).

The long-term plan is to move all of the "how core Kubernetes reacts to node conditions, with respect to scheduling and eviction" logic to use taints and tolerations (e.g. #42001). But we aren't there yet and won't be for a while. So for the near-term there is no way for a pod to "tolerate" NetworkUnavailable or any other node condition you might decide to add.

If I understand what @davidopp said right, then the function is NodeReady, which now returns true or false or unknown, ticks a list with what is needed. If it could return a list of values not met, you can test against that.

In case of kubeadm, if then NetworkUnavailable, is the only one. kubeadm can return true. That is in the case we don't want the networking/cni plugin passed in at kubeadm init time.

Cause if I understand well, the taint's and toleration are set based on this function, at least for kubeadm.

Correct me if I'm wrong ;)

Jumping on to this thread because I hit this issue and it is annoying:

Why can't network readiness become a taint that the kubelet adds and removes based on the state of the network on the node?
That way the node can be "Ready" for all pods that tolerate the network readiness taint (pod network apps).

As for host network pods, an admission controller could inject tolerations to pod network readiness taint.

I think that aligns with the long term plan in davidopp's comment.

I think that aligns with the long term plan in davidopp's comment.

Yes, exactly. The long-term plan is to have a taint for every node condition and stop having anything internal in Kubernetes pay attention to node conditions. We started that work with the 1.6 alpha feature I mentioned.

Ack. Is there a reason not to switch to taints in v1.6 patch release?

The code to automatically add taints based on node conditions just went into alpha in 1.6. It's not ready to be relied upon. (And it only handles node ready condition currently, not network).

So, if I understand correctly, for now, since https://github.com/kubernetes/features/issues/166 is taking some longer time to be able to taint network availability correctly, we have to go with a work around. If we can push a fix ASAP for kubeadm, like #43835, with a comment to fix this with https://github.com/kubernetes/features/issues/166, a lot ppl are going to be happy.

@davidopp I assume you meant to say "not relied upon". I'm still missing the relationship between the automatic condition to taint logic you mentioned with the of the kubelet posting taints directly for network issues.

Yes thanks, that was a typo, fixed now

Yes, you could have Kubelet add the taint. It may not be ideal to have Kubelet add the taints for some node conditions and NodeController add them for other node conditions (the latter is necessary because only the master can detect node unreachable), but that's just a software engineering argument, not anything fundamental. My plan had been for NodeController to add the taints for all node conditions but there's no strong requirement to do it that way.

Ack. In this case, a single Node condition has several attributes associated with it which the Node Controller may not be able to discern.
@mikedanese I feel the existing behavior of kubeadm init resulting in a functioning node is appealing. I hope the requirement of adding a validation phase is not primarily driven based on this issue.
@dcbw @thockin @yujuhong any thoughts on using taints to represent node configuration issues in the kubelet?

Note that if you want your new taint to replace the current automatic filtering-out of nodes with NetworkUnavailable, you will need to modify the function I mentioned earlier (i.e. remove it from the set of conditions that filter out a node). I do not know what other side-effects that will have, since that function may be used in node listers other than just the scheduler.

Is there a way to install 1.5.6 which should be working on Ubuntu? If there is could you please explain how to do it? Thanks!

Note that if you want your new taint to replace the current automatic filtering-out of nodes with NetworkUnavailable, you will need to modify the function I mentioned earlier (i.e. remove it from the set of conditions that filter out a node). I do not know what other side-effects that will have, since that function may be used in node listers other than just the scheduler.

@davidopp we need to be careful about NetworkUnavailable vs. NodeReady. There are two separate conditions that should really be cleaned up:

nodeNetworkUnavailable - this is specific to cloud routes, and only cloud route controllers clear this condition when cloud routes between nodes have been set up. Network plugins that create overlays or that do not do L3 routing between nodes do not want or use this condition. This condition is not added by any network plugin; it's added by kubelet specifically when GCE or AWS are selected as cloud providers. Doesn't affect NodeReady since it's a separate condition.

NodeReady - affected directly by the network plugins (eg, CNI or kubenet or remote runtimes like dockershim/CRI-O).

There are three separate issues to consider:

1. Cloud Routes - if the cloud infrastructure and network plugin require cloud routes, then lack of cloud routes (eg, NodeNetworkUnavailable=true) needs to block scheduling hostNetwork=false pods
2. Network plugins - if the plugin is not yet ready, that needs to block scheduling of hostNetwork=false pods. This is separate from NodeNetworkUnavailable because the plugin may have no direct interaction with a Routes Controller because it doesn't depend on cloud routes. Eg, kubenet can be used locally on the node if you have something else (cloud routes, flannel, etc) set up node routes.
3. hostNetwork=true pods should ignore all these conditions and be scheduled, but subject to any other applicable conditions (disk space, etc)

NodeReady is just too big of a hammer. I'm thinking we probably want an additional NetworkPluginReady condition for the network plugin readiness (separate from cloud routes readiness!) and then we have to plumb that through to places that care :( Or, really, new taints not conditions.

Another work around I found.

While kubeadm keeps printing '[apiclient] First node has registered, but is not ready yet', I deployed 'kube-flannel.yml' which installs flanneld. And it worked without changing configuration files.

1) kubeadm init --pod-network-cidr=10.244.0.0/16 &
2) cp /etc/kubernetes/admin.conf ~/.kube/config
3) kubectl apply -f kube-flannel-rbac.yml (Necessary in kubeadm 1.6)
4) kubectl apply -f kube-flannel.yaml
Use kube-flannel.yaml in https://github.com/coreos/flannel/blob/master/Documentation/kube-flannel.yml

I could make the master node 'Ready'. But, kube-dns failed with the message,

Error syncing pod, skipping: failed to "CreatePodSandbox" for "kube-dns-2759646324-nppx6_kube-system(bd585951-15cb-11e7-95c3-1c98ec12245c)" with CreatePodSandboxError: "CreatePodSandbox for pod \"kube-dns-2759646324-nppx6_kube-system(bd585951-15cb-11e7-95c3-1c98ec12245c)\" failed: rpc error: code = 2 desc = NetworkPlugin cni failed to set up pod \"kube-dns-2759646324-nppx6_kube-system\" network: \"cni0\" already has an IP address different from 10.244.0.1/24"

After I change the network plug-in to weave-net, it worked.

@dcbw I don't know enough about the specific problem discussed in this issue to tell you exactly what to do. I just wanted to explain the current status of taints and how they might apply here.

PLEASE TEST THE PATCHED DEBS

The kubernetes-xenial-unstable now has a patched build 1.6.1-beta.0.5+d8a384c1c5e35d-00 that @pipejakob and I have been testing today. The nodes remain not ready until a pod network is created (e.g. by applying weave/flannel configs). Conformance test pass. PTAL.

# cat <<EOF >/etc/apt/sources.list.d/kubernetes.list
deb http://apt.kubernetes.io/ kubernetes-xenial-unstable main
EOF

cc @luxas @jbeda

Is the general direction that taints can express pretty much everything
conditions can, and are therefore better in every way?

I agree we need more granular signal. We still need to sort out the
interplay between kubelet, net driver, cloud controller, and cloud provider
to unlock hostNetwork=false pods, but hostNetwork=true should not be
blocked.

On Thu, Mar 30, 2017 at 7:24 PM, Dan Williams notifications@github.com
wrote:

Note that if you want your new taint to replace the current automatic
filtering-out of nodes with NetworkUnavailable, you will need to modify the
function I mentioned earlier (i.e. remove it from the set of conditions
that filter out a node). I do not know what other side-effects that will
have, since that function may be used in node listers other than just the
scheduler.

@davidopp https://github.com/davidopp we need to be careful about
NetworkUnavailable vs. NodeReady. There are two separate conditions that
should really be cleaned up:

nodeNetworkUnavailable - this is specific to cloud routes, and only cloud
route controllers clear this condition when cloud routes between nodes have
been set up. Network plugins that create overlays or that do not do L3
routing between nodes do not want or use this condition. This condition is
not added by any network plugin; it's added by kubelet specifically when
GCE or AWS are selected as cloud providers. Doesn't affect NodeReady since
it's a separate condition.

NodeReady - affected directly by the network plugins (eg, CNI or kubenet
or remote runtimes like dockershim/CRI-O).

There are three separate issues to consider:

1. Cloud Routes - if the cloud infrastructure and network plugin
   require cloud routes, then lack of cloud routes (eg,
   NodeNetworkUnavailable=true) needs to block scheduling hostNetwork=false
   pods
2. Network plugins - if the plugin is not yet ready, that needs to
   block scheduling of hostNetwork=false pods
3. hostNetwork=true pods should ignore all these conditions and be
   scheduled, but subject to any other applicable conditions (disk space, etc)

NodeReady is just too big of a hammer. I'm thinking we probably want an
additional NetworkPluginReady condition for the network plugin readiness
(separate from cloud routes readiness!) and then we have to plumb that
through to places that care :(

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/kubernetes/kubernetes/issues/43815#issuecomment-290597988,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AFVgVGIn0CpVkcHd2SVaoRsp2RJNEgXFks5rrGPqgaJpZM4MtMRe
.

For anyone still trying the temporary fix by removing the kubelet KUBELET_NETWORK_ARGS config line, @jc1arke found a simpler workaround - have two sessions to the new master and, whilst waiting for the first node to become ready, apply a node-network config in the second session:
First session after running kubeadmin init:

...
[apiclient] Created API client, waiting for the control plane to become ready
[apiclient] All control plane components are healthy after 24.820861 seconds
[apiclient] Waiting for at least one node to register and become ready
[apiclient] First node has registered, but is not ready yet
[apiclient] First node has registered, but is not ready yet
[apiclient] First node has registered, but is not ready yet
[apiclient] First node has registered, but is not ready yet
[apiclient] First node has registered, but is not ready yet
...

Second session (using Calico. Your choice may of course vary):

root@kube-test-master-tantk:~# kubectl apply -f http://docs.projectcalico.org/v2.0/getting-started/kubernetes/installation/hosted/kubeadm/calico.yaml
configmap "calico-config" created
daemonset "calico-etcd" created
service "calico-etcd" created
daemonset "calico-node" created
deployment "calico-policy-controller" created
job "configure-calico" created
root@kube-test-master-tantk:~#

Back to first session:

...
[apiclient] First node has registered, but is not ready yet
[apiclient] First node has registered, but is not ready yet
[apiclient] First node has registered, but is not ready yet
[apiclient] First node is ready after 118.912515 seconds
[apiclient] Test deployment succeeded
[token] Using token: <to.ken>
[apiconfig] Created RBAC rules
...

Is the general direction that taints can express pretty much everything
conditions can, and are therefore better in every way?

Pretty much. We probably can't get rid of any of the node conditions due to backward compatibility concerns, but we can get rid of all the logic in the master that takes action based on them (like blocking scheduling or evicting pods). Besides making the behavior more obvious (no need to memorize which conditions block scheduling, which cause eviction, how long we wait for eviction, etc.), the reaction to the conditions is configurable on a per-pod basis.

Just tested the debs from @mikedanese and @pipejakob, these are working fine.

Tested on ubuntu/xenial:

root@kube1:/home/ubuntu# kubeadm version
kubeadm version: version.Info{Major:"1", Minor:"6+", GitVersion:"v1.6.1-beta.0.5+d8a384c1c5e35d", GitCommit:"d8a384c1c5e35d5118f79808deb7bab41f3f7964", GitTreeState:"clean", BuildDate:"2017-03-31T04:20:36Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"linux/amd64"}

Just tested the debs from @mikedanese and @pipejakob, it still freeze at
[apiclient] Created API client, waiting for the control plane to become ready

I have waited about five minutes, but nothing changed.
And yesterday it kept repeating
[apiclient] First node has registered, but is not ready yet

T T I think the problem is still no solution?

Tested on Ubuntu 16.04:
kubeadm version: version.Info{Major:"1", Minor:"6+", GitVersion:"v1.6.1-beta.0.5+d8a384c1c5e35d", GitCommit:"d8a384c1c5e35d5118f79808deb7bab41f3f7964", GitTreeState:"clean", BuildDate:"2017-03-31T04:20:36Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"linux/amd64"}

@myqq0000 can you post the version you using?

kubeadm version

@coeki Oh, I forget it. I have edited just now and post the version in my previous comment. :)

@mikedanese Do you have any plan to update centos yum repo ? Or is it already deployed to yum repo ?

Just tried 1.6.1-beta.0.5+d8a384c1c5e35d-00 (the 1.6.1-beta.0.5+48c6a53cf4ff7d-00 one mentioned in https://github.com/kubernetes/kubernetes/issues/43815#issuecomment-290616036 doesn't seem to exist).

It seems to work fine.
Unrelated: note that if you're using weave you have to apply https://github.com/weaveworks/weave/releases/download/latest_release/weave-daemonset-k8s-1.6.yaml instead of the 'default' https://git.io/weave-kube

@mausch Could you tell me how to get these debs?

@srinat999 https://github.com/kubernetes/kubernetes/issues/43815#issuecomment-290616036

@mikedanese patched debs work for me. thanks for all working on this! :smile:

root@kube-test0:~# kubeadm version
kubeadm version: version.Info{Major:"1", Minor:"6+", GitVersion:"v1.6.1-beta.0.5+d8a384c1c5e35d", GitCommit:"d8a384c1c5e35d5118f79808deb7bab41f3f7964", GitTreeState:"clean", BuildDate:"2017-03-31T04:20:36Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"linux/amd64"}

@mausch Thanks. They work for me too with weave network provider.

Any chance to build the same fix for Centos as well? Our gating system mostly uses centos for kubernetes cluster base. If I have centos version I can guarantee approx. 100 runs of kubeadm init a day as testing.

@mikedanese patched debs are partially work for me. kubeadm is notifying that cluster is ready, but node itself is not.

@squall0gd nodes should become ready when a pod network is applied, e.g. kubectl apply -f https://git.io/weave-kube-1.6

In my test-environment (in vagrant, based on centos/7 machines), kubeadm just hangs. Using strace i can see it trying to connect to the kubelet server on the master , and doing FUTEX_WAIT, epoll_wait retry loops. But there is no single line ouf output coming from it.

kubelet fails to start, which seems to be the reason.
(But i can't see reasons for kublet failing to start..)

Is this the problem of this issue?

I get binaries from the http://yum.kubernetes.io/repos/kubernetes-el7-x86_64 repo.
I also try updating the binaries from https://storage.googleapis.com/kubernetes-release/release/v1.6.0/bin/linux/amd64/ .

==> Where can I get a patched version of kubeadm for testing? <==

Note: I found one (claimed) patched version of kubeadm referenced in this issue at https://github.com/kensimon/aws-quickstart/commit/9ae07f8d9de29c6cbca4624a61e78ab4fe69ebc4 (patched version is this: https://heptio-aws-quickstart-test.s3.amazonaws.com/heptio/kubernetes/ken-test/kubeadm), but that does not work for me. Behavior is still the same (no output whatsoever)...

@squall0gd Can you show us the exact message you're seeing? Based on what you describe, I wonder if you actually picked up the new .debs or were using older cached versions.

@thockin @davidopp so per Tim's suggestion I'll commandeer an existing issue or file a new one to track disentangling network readiness conditions and to convert them all over to taints instead of actual conditions, and copy in most of https://github.com/kubernetes/kubernetes/issues/43815#issuecomment-290597988 into it.

Here's what is seemingly working for me with the unstable repo (only tested the master itself):

"sudo apt-get update && sudo apt-get install -y apt-transport-https",
"echo 'deb http://apt.kubernetes.io/ kubernetes-xenial-unstable main' | sudo tee /etc/apt/sources.list.d/kubernetes.list",
"curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -",
"sudo apt-get update",
"sudo apt-get install -y docker.io",
"sudo apt-get install -y kubelet kubeadm kubectl kubernetes-cni",
"sudo service kubelet stop",
"sudo service kubelet start",
"sudo kubeadm init",
"sudo cp /etc/kubernetes/admin.conf $HOME/",
"sudo chown $(id -u):$(id -g) $HOME/admin.conf",
"export KUBECONFIG=$HOME/admin.conf",
"kubectl taint nodes --all dedicated-",
"kubectl apply -f https://github.com/weaveworks/weave/releases/download/latest_release/weave-daemonset-k8s-1.6.yaml",

This does spit out error: taint "dedicated:" not found at one point, but it seems to carry on regardless.

Do we not run these tests on the 1.6 branch?

@davidopp Node conditions are intended to provide additional information for humans, for understanding what is going on, such as the reason, message, and transition time.

kubeadm doesn't look like it is tested on the release branch:
https://k8s-testgrid.appspot.com/release-1.6-all

@bgrant0607 apparently the kubeadm e2e tests were disabled/non-functional for about a week leading up to the 1.6 release, IIRC

Node conditions are intended to provide additional information for humans, for understanding what is going on, such as the reason, message, and transition time.

@bgrant0607 are they intended primarily for human info, or is that just a happy side-effect? If primarily for humans, should they be used for scheduling decisions like they currently are or should we move away from that?

@dcbw Taints are intended to become the GUT of unschedulability and disruption. Eventually, the scheduler should ignore conditions.

https://github.com/kubernetes/kubernetes/pull/18263#issuecomment-169220144
https://github.com/kubernetes/kubernetes/issues/29178

Note that if we're going to add a taint to nodes, we need to consider

a) Backward compatibility, as with the master taint: https://github.com/kubernetes/kubernetes/pull/43537

b) Version skew. When upgrading clusters to 1.6, kubelets will be of mixed releases, and some may be as old as 1.4.

so to recap or as I parse this:

based on @dcbw

nodeNetworkUnavailable is a cloud provider thing, not tied to kubeadm atm, we might run into this.

But NodeReady, which is the root cause for the loop, needs to more granular. This is what @davidopp wants to be a taint, based on a list that's ticking off boxes, in regard to probe/liveness, CNI ready and etc.

fine, although you could argue, why not label?

But @dcbw did you find a thread more suitable for this discussion? Cause this one becomes a bucket for issues with deployment and not really the root of things.

I know, I was part of not really getting to the problem, and posting hacks around it :)

But anyway, we should discuss the fundamentals in another place, make sure an ETA on fix is placed here, and move on.

Not being snappy, but well :)

PS yes @bgrant0607 we should have tested this more
PS2 If I see this wrong, you can blame me ;)

@coeki I'd also add a request for N-1 versions to be kept in rpm/deb repos. All major releases eventually end up with a problem or two. Operators have long avoided N.0 releases for production for that very reason. It works well if previous versions are left around for a while. But this time 1.5.x was removed entirely before 1.6 was made stable. That puts operators that weren't very prepared (local repo mirroring, etc) from making forward progress while the issue is sorted out. The pain of a bumpy N+1 release can often be dealt with by simply keeping N around for a while.

@kfox1111 gating is a problem too....we need a better strategy on that :)

Why even delete old releases at all? That could easily break people's automation and prevent repeatable installs.

Node conditions are intended to provide additional information for humans, for understanding what is going on, such as the reason, message, and transition time.

Agreed. UX is definitely a major reason why we probably can't get rid of node conditions. However, I do believe we can get rid of all uses of node conditions as a trigger for master actions (like eviction and blocking scheduling), and use taints for that instead.

In the long run (like v2 API long-run) it's conceivable we could add reason and message to taints and then get rid of node conditions, but I haven't really thought about it and am definitely not formally proposing to do so. (We already have the equivalent of transition time in one direction, namely TimeAdded.)

@coeki no, what I was talking about has nothing to do with gating. Gating can't find all problems unless you have a gate that tests for everything that could possibly ever be done. Operators like to do weird stuff in their environments. :) Prohibitively expensive to test all things possible.

I'm asking for not deleting the old release before the new one has made it at least through the first bugfix release. For this example, 1.6.1. At minimum. Keeping more older versions around is even better. This is a best practice to allow operators to continue to make progress while bugs in the new version is worked out.

@kfox1111, that is what gating does, not go with something that does seem good, and throwing out the old good trusted way, what happened now.

@davidopp I do agree that labels and taints might have a different distinction from a api way of looking at it, UX and machinery, but well it's diffuse right now. Too me, that is :)

Anyway, you lure me into a discussion, we really not to have in this issue, that was my point.

I'd like to ask again: if I see "kubeadm init" just hanging there with no output (trying to connect to kubelet server, which has failed to start), am I suffering of this issue? And if this is a case, is #43835 a fix for me?

Now where can I get:

• either older (pre 1.6.0) versions of kubeadm / kubectl / kubelet
• or a patched version of kubeadm (including #43835 or whatever other fix)?

Thanks!

(note: the patched version of kubeadm referenced in the mentioned commit above does not work for me...)

@obnoxxx try the tip of the release-1.6 branch.

$ gsutil ls gs://kubernetes-release-dev/ci/v1.6.1-beta.0.12+018a96913f57f9

https://storage.googleapis.com/kubernetes-release-dev/ci/v1.6.1-beta.0.12+018a96913f57f9/bin/linux/amd64/kubeadm

@mikedanese thanks! trying...

If you run kubeadm without installing the os package you need to

$ cat <<EOF > /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
[Service]
Environment="KUBELET_KUBECONFIG_ARGS=--kubeconfig=/etc/kubernetes/kubelet.conf --require-kubeconfig=true"
Environment="KUBELET_SYSTEM_PODS_ARGS=--pod-manifest-path=/etc/kubernetes/manifests --allow-privileged=true"
Environment="KUBELET_NETWORK_ARGS=--network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin"
Environment="KUBELET_DNS_ARGS=--cluster-dns=10.96.0.10 --cluster-domain=cluster.local"
Environment="KUBELET_AUTHZ_ARGS=--authorization-mode=Webhook --client-ca-file=/etc/kubernetes/pki/ca.crt"
ExecStart=
ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_SYSTEM_PODS_ARGS $KUBELET_NETWORK_ARGS $KUBELET_DNS_ARGS $KUBELET_AUTHZ_ARGS $KUBELET_EXTRA_ARGS
EOF
$ systemctl daemon-reload
$ systemctl restart kubelet

from https://github.com/kubernetes/release/blob/master/debian/xenial/kubeadm/channel/stable/etc/systemd/system/kubelet.service.d/10-kubeadm.conf

@mikedanese thanks. I am installing the os package (from kube repo) and then installing the binaries from the web url over the binaries from the rpms...

the 1.6.1-beta.0.12 version did not work for me though:
kubeadm init still hangs without any output. (trying to connect to kubelet)

Dont forget the systemd driver too if centos.

systemd driver?

https://github.com/kubernetes/release/issues/306

gosh, thanks, that is better! :-)

@pipejakob I don't have access to this logs right now, but I've checked kubeadm version and it was the same version as @webwurst has got. Moreover I've checked possible kubeadm versions with apt-cache policy kubeadm. And there was only one entry - from kubernetes-xenial-unstable.
@mikedanese I've tried with flannel before posting ;)
EDIT: I've rebuild whole platform and kubadm is working fine :+1:

Guys what is the status quo for the fix? is it gonna move to the stable repository anytime soon?

@mikedanese patched debs are doing the same "network plugin is not ready" thing.

Can anybody tell me how to build a patched version of kubeadm for rhel (rpm) ?

@mikedanese with patched debs the init reports success, but the "network plugin is not ready: cni config uninitialized" message persists.

The master node is in NotReady.

The weave-net (weaveworks/weave-kube:1.9.4 - https://github.com/weaveworks/weave/releases/download/latest_release/weave-daemonset-k8s-1.6.yaml) is in 1/2 CrashLoopBackOff:

FailedSync  Error syncing pod, skipping: failed to "StartContainer" for "weave"

The kube-dns pod is in 0/3 Pending.

@ChipmunkV You probably need to configure kube-proxy to run in userspace. This has been an issue in 1.5 as well.

command:
  - /usr/local/bin/kube-proxy
  - --kubeconfig=/var/lib/kube-proxy/kubeconfig.conf
  # add this line:
  - --proxy-mode=userspace

For what it's worth, kubernetes-xenial-unstable is working well for me.

@pstadler, thanks to @errordeveloper in chat who has pointed out that I've got overlapping networks, I've reconfigured the IPALLOC_RANGE in weave.

Thanks @thenayr. I could bring up the Kubernetes master but also could connect one worker to it using kubeadm join. Will post more updates soon

@mikedanese it took me forever to find out what version to use, until I read all comments here. We really need to make it much easier to find binaries for a given version. I've tried using what ci-cross/latest.txt points at (namely v1.7.0-alpha.0.1982+70684584beeb0c), and that happen to introduce new kube-apiserver flag (namely --proxy-client-key-file in d8be13fee85075abfc087632b3dbc586a10897ad), which doesn't work with any of the recents tags in gcr.io/google_containers/kube-apiserver-amd64...

How can we make it easier to figure out what revisions have binaries in the bucket? Being able to list directories would be nice, or have a simple map file or just anything that doesn't require having to know or having to guess.

@errordeveloper we are trying to do a release so we can push a fix to the stable branch, should be done in O(days) I hope. There's a link in the description of this issue to the patched debs in unstable.

Previous release 1.5.6 was working for me on CentOs 7 but now I can't even roll back because the only version of kubeadm in http://yum.kubernetes.io/repos/kubernetes-el7-x86_64 is the broken 1.6.0. Any suggestions on how to get kubeadm 1.5.6 RPM?

Indeed, it's a pity. Old versions seem to have been entirely removed. :-(

My results are these on centos 7:

• I have not been able to get kubadm init to give me any result at all, even with a patched , latest kubeadm, without doing something about kubectl.
• I have been able to get kubectl start with the instructions of @coeki , and then kubeadm even passed. But after that, nothing works for me. kubectl says

The connection to the server localhost:8080 was refused - did you specify the right host or port?

Any hint what's going on?

@obnoxxx by default apiserver does not listen on 8080, it listens on secure 6443 port, you can check with netstat -tunlp.
You need to copy admin.conf from /etc/kubernetes to you $HOME/.kube/config
make sure you change permissions on the file in your $HOME/.kube/, after that your kubectl should be able to contact apiserver properly. HTH. Serguei

@apsinha Are you aware of this thread? It might be good to have some product folks following, as I think there will be some important takeaways for the future.

Off the top of my head:

• 1.6.0 never went through automated testing: https://github.com/kubernetes/kubernetes/issues/43815#issuecomment-290809481
• Binaries/packages for older versions were removed, so people cannot roll back safely; broke any installation automation that was assuming they continued to be available: https://github.com/kubernetes/kubernetes/issues/43815#issuecomment-290861903
• No public announcement (that I've seen) of the fact that this is broken
• No timeline given about a fix (I'm a developer, so I know how obnoxious being asked "when will it be fixed?" is, but nevertheless, people will ask this, so it's good to at least offer an estimated time period)
• Users continuing to join the conversation confused about the status of things, workarounds, and fix schedule
• Lots of technical discussion among contributors, many of which are about long term strategy, combined in the same thread as users trying to figure out what's going on and how to deal with the immediate problem

No disrespect intended to all the great people that make Kubernetes what it is. I'm just hoping there are some "teachable moments" here moving forward, as this looks bad in terms of the public perception of Kubernetes being reliable/stable. (Granted kubeadm is alpha/beta, but it's still got lots of visibility.)

I built kubeadm-1.5.6 using kubernetes/release.rpm only to to find (to my dismay) that

# rpm -Uvh kubectl-1.5.6-0.x86_64.rpm kubeadm-1.5.6-0.x86_64.rpm 
error: Failed dependencies:
        kubectl >= 1.6.0 is needed by kubeadm-1.5.6-0.x86_64
        kubelet >= 1.6.0 is needed by kubeadm-1.5.6-0.x86_64

@bostone you need to adjus the .spec here.

@sbezverk thanks for the hint! It is better now...

This is curious:

• I don't remember copying the admin.conf was necessary for me in earlier releases.
• I tried with kubectl -s localhost:6443 before but that was not sufficient.

Anyways, now I can continue. Thanks again

v1.6.1 is in the process of being released. It will be done by EOD.

Some notes:

1. The issue with old packages being deleted was tracked here: https://github.com/kubernetes/release/issues/234. Due to some wacky versioning of kubeadm and how these things are sorted alphabetically, apparently it wasn't possible to keep the 1.5.x version of kubeadm in the repo. (@bostone your issue is referenced there too)
2. kubeadm e2e testing on PRs being tracked here: https://github.com/kubernetes/kubeadm/issues/190
3. As for public announcement -- I posted on twitter but that is hardly official. Not clear what the correct channels are for critical issues like this.

These are all good issues to bring up in a PM. @pipejakob has graciously volunteered to pull that post mortem together.

@obnoxxx The requirement to copy/chown the admin.conf was necessary before because with 1.5 we had the API server exposing an insecure port. We changed that in 1.6. Now you must use secure credentials to access the API server (see https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG.md#kubeadm-2)

@mikedanese that sounds great, thanks!
Just for the dummy I am - what will be the effect of this release 1.6.1 with respect to this issue?
Will kubelet start without the change to /etc/systemd/system/kubelet.service.d/10-kubeadm.conf ( @coeki 's workaround) or will it contain the fix of #43835 (which did not seem to have any effect for me)?

@jbeda thanks for the explanation!

@jbeda I think the confusion comes from the official kubadm installation doc setting YUM repo to http://yum.kubernetes.io/repos/kubernetes-el7-x86_64. That repo only has kubeadm-1.6.0

And again to my disappointment building 1.5.6 rpms and installing ended up with same exact problem. Running kubeadm init is hanging at the very same line:

# kubeadm init
Running pre-flight checks
<master/tokens> generated token: "5076be.38581a71134596d0"
<master/pki> generated Certificate Authority key and certificate:
Issuer: CN=kubernetes | Subject: CN=kubernetes | CA: true
Not before: 2017-04-03 21:28:18 +0000 UTC Not After: 2027-04-01 21:28:18 +0000 UTC
Public: /etc/kubernetes/pki/ca-pub.pem
Private: /etc/kubernetes/pki/ca-key.pem
Cert: /etc/kubernetes/pki/ca.pem
<master/pki> generated API Server key and certificate:
Issuer: CN=kubernetes | Subject: CN=kube-apiserver | CA: false
Not before: 2017-04-03 21:28:18 +0000 UTC Not After: 2018-04-03 21:28:18 +0000 UTC
Alternate Names: [10.25.47.21 10.96.0.1 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local]
Public: /etc/kubernetes/pki/apiserver-pub.pem
Private: /etc/kubernetes/pki/apiserver-key.pem
Cert: /etc/kubernetes/pki/apiserver.pem
<master/pki> generated Service Account Signing keys:
Public: /etc/kubernetes/pki/sa-pub.pem
Private: /etc/kubernetes/pki/sa-key.pem
<master/pki> created keys and certificates in "/etc/kubernetes/pki"
<util/kubeconfig> created "/etc/kubernetes/kubelet.conf"
<util/kubeconfig> created "/etc/kubernetes/admin.conf"
<master/apiclient> created API client configuration
<master/apiclient> created API client, waiting for the control plane to become ready

I guess I'll just wait for 1.6.1 release and hope...

1.6.1 is out.

@mikedanese did you test kubeadm before closing this bug? Right now I'm staring at [apiclient] Created API client, waiting for the control plane to become ready for 10 min already. On CentOS 7 with brand new 1.6.1 installation

@bostone you might be hitting this issue: #43805

Try editing /etc/systemd/system/kubelet.service.d/10-kubeadm.conf and adding --cgroup-driver=systemd

Remember to run systemctl daemon-reload; systemctl restart kubelet

@gtirloni with our suggestion I got to the end of kubeadm init however any attempt to run kubectl produces this error: The connection to the server localhost:8080 was refused - did you specify the right host or port? I'm not sure where and how change that or what is the right port at this point?

Not sure if this is related but here's what I see when running systemctl status kubelet:
``` systemctl status kubelet -l
● kubelet.service - kubelet: The Kubernetes Node Agent
Loaded: loaded (/etc/systemd/system/kubelet.service; enabled; vendor preset: disabled)
Drop-In: /etc/systemd/system/kubelet.service.d
└─10-kubeadm.conf
Active: active (running) since Mon 2017-04-03 17:31:07 PDT; 11min ago
Docs: http://kubernetes.io/docs/
Main PID: 10382 (kubelet)
CGroup: /system.slice/kubelet.service
├─10382 /usr/bin/kubelet --kubeconfig=/etc/kubernetes/kubelet.conf --require-kubeconfig=true --pod-manifest-path=/etc/kubernetes/manifests --allow-privileged=true --network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin --cluster-dns=10.96.0.10 --cluster-domain=cluster.local --authorization-mode=Webhook --client-ca-file=/etc/kubernetes/pki/ca.crt --cgroup-driver=systemd
└─10436 journalctl -k -f

Apr 03 17:41:56 sdl02269 kubelet[10382]: W0403 17:41:56.645299 10382 cni.go:157] Unable to update cni config: No networks found in /etc/cni/net.d
Apr 03 17:41:56 sdl02269 kubelet[10382]: E0403 17:41:56.645407 10382 kubelet.go:2067] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized```

Yes we tested. e2e tests are passing on the release branch. You may be hitting other issues.

@bostone maybe you're missing these steps after kubeadm init?

  sudo cp /etc/kubernetes/admin.conf $HOME/
  sudo chown $(id -u):$(id -g) $HOME/admin.conf
  export KUBECONFIG=$HOME/admin.conf

You also need to follow step 3 described here. That seems related to the cni config error you're getting.

Also staring at [apiclient] Created API client, waiting for the control plane to become ready for 10 min already with Ubuntu 16.04 and a fresh install of 1.6.1

@pingthings I'd recommend joining the Slack Kubernetes and the kubeadm channel. We can help you debug there.

I successfully setup my Kubernetes cluster on centos-release-7-3.1611.el7.centos.x86_64 by taking the following steps (I assume Docker is already installed):

1) (from /etc/yum.repo.d/kubernetes.repo) baseurl=http://yum.kubernetes.io/repos/kubernetes-el7-x86_64-unstable
=> To use the unstable repository for the latest Kubernetes 1.6.1
2) yum install -y kubelet kubeadm kubectl kubernetes-cni
3) (/etc/systemd/system/kubelet.service.d/10-kubeadm.conf) add "--cgroup-driver=systemd" at the end of the last line.
=> This is because Docker uses systemd for cgroup-driver while kubelet uses cgroupfs for cgroup-driver.
4) systemctl enable kubelet && systemctl start kubelet
5) kubeadm init --pod-network-cidr 10.244.0.0/16
=> If you used to add --api-advertise-addresses, you need to use --apiserver-advertise-address instead.
6) cp /etc/kubernetes/admin.conf $HOME/
sudo chown $(id -u):$(id -g) $HOME/admin.conf
export KUBECONFIG=$HOME/admin.conf
=> Without this step, you might get an error with kubectl get
=> I didn't do it with 1.5.2
7) kubectl create -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel-rbac.yml
=> 1.6.0 introduces a role-based access control so you should add a ClusterRole and a ClusterRoleBinding before creating a Flannel daemonset
8) kubectl create -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
=> Create a Flannel daemonset
9) (on every slave node) kubeadm join --token (your token) (ip):(port)
=> as shown in the result of kubeadm init

All the above steps are a result of combining suggestions from various issues around Kubernetes-1.6.0, especially kubeadm.

Hope this will save your time.

This doesn't seem to be resolved (Ubuntu LTS, kubeadm 1.6.1).

First, I also experienced kubeadm hanging on "Created API client, waiting for the control plane to become ready" when using --apiserver-advertise-address flag.. The journal logs say:

let.go:2067] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
Apr 04 12:27:04 xxx kubelet[57592]: E0404 12:27:04.352780   57592 eviction_manager.go:214] eviction manager: unexpected err: failed GetNode: node 'xxx' not found
Apr 04 12:27:05 xxx kubelet[57592]: E0404 12:27:05.326799   57592 kubelet_node_status.go:101] Unable to register node "xxx" with API server: Post https://x.x.x.x:6443/api/v1/nodes: dial tcp x.x.x.x:6443: i/o timeout
Apr 04 12:27:06 xxx kubelet[57592]: E0404 12:27:06.745674   57592 reflector.go:190] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46: Failed to list *v1.Pod: Get https://x.x.x.x:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dxxx&resourceVersion=0: dial tcp x.x.x.x:6443: i/o timeout
Apr 04 12:27:06 xxx kubelet[57592]: E0404 12:27:06.746759   57592 reflector.go:190] k8s.io/kubernetes/pkg/kubelet/kubelet.go:390: Failed to list *v1.Node: Get https://x.x.x.x:6443/api/v1/nodes?fieldSelector=metadata.name%3Dxxx&resourceVersion=0: dial tcp x.x.x.x:6443: i/o timeout
Apr 04 12:27:06 xxx kubelet[57592]: E0404 12:27:06.747749   57592 reflector.go:190] k8s.io/kubernetes/pkg/kubelet/kubelet.go:382: Failed to list *v1.Service: Get https://x.x.x.x:6443/api/v1/services?resourceVersion=0: dial tcp x.x.x.x:6443: i/o timeou

If I don't provide this flag, kubeadm passes, but even then I get following error for kubelet starting:

Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized

Kubelet refuses to properly start, and I cannot connect to cluster with kubectl in any way

It seems 1.5.x kubeadm versions have been removed not only from centos repository, but also from Ubuntu LTS: https://packages.cloud.google.com/apt/dists/kubernetes-xenial/main/binary-amd64/Packages It makes it difficult to downgrade, and actually breaks backward compatibility

@bostone

Did you figure out the hang at "waiting for control plane to become ready"? I see the same thing after trying all the changes with 1.6.1.

@gtirloni @srzjulio adding these steps after kubeadm init did not help. It did make kubelet service change from failed to active (running) but I still have The connection to the server localhost:8080 was refused - did you specify the right host or port? message. And if I look at services listening I can't see 8080 anywhere. What I can see is that kubelet is listening on these ports:

kubelet    6397  root   12u  IPv6 611842      0t0  TCP *:4194 (LISTEN)
kubelet    6397  root   15u  IPv4 611175      0t0  TCP sdl02269.labs.teradata.com:49597->sdl02269.labs.teradata.com:sun-sr-https (ESTABLISHED)
kubelet    6397  root   16u  IPv4 611890      0t0  TCP localhost:10248 (LISTEN)
kubelet    6397  root   18u  IPv6 611893      0t0  TCP *:10250 (LISTEN)
kubelet    6397  root   19u  IPv6 611895      0t0  TCP *:10255 (LISTEN)

So there's some wrong (changed?) setting that wrongly points to 8080?

@bostone it is not kubelet port you need, it is kube-apiserver, it should be listening on 6443 port.

@sbezverk I did not modify any defaults as far as ports (it's not in any instructions) What do I need to do to switch from 8080 to 6443?

@bostone if you did not modify anything in apiserver manifest then by default it will listen on 6443 port. You just need to copy /etc/kubernetes/admin.conf into $HOME/.kube/config, change permissions on config file and you should be all set.

@sbezverk BTW - I'm running all the installation steps as root but these 3 instruction lines from kubeadm init output suggest using sudo user. What is the recommendation on this? Should run all installation steps as a sudo as well?

Ok, I did all the steps from scratch and it seems to be better. Here are the steps that worked for me thus far and I'm running as root on CentOS 7.

# cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=http://yum.kubernetes.io/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg
        https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
EOF
# setenforce 0
# yum install -y docker kubelet kubeadm kubectl kubernetes-cni
# vi /etc/systemd/system/kubelet.service.d/10-kubeadm.conf

Add -cgroup-driver=systemd to 10-kubeadm.conf and save

# systemctl enable docker && systemctl start docker
# systemctl enable kubelet && systemctl start kubelet
# sysctl -w net.bridge.bridge-nf-call-iptables=1
# systemctl stop firewalld; systemctl disable firewalld
# kubeadm init
# cp /etc/kubernetes/admin.conf $HOME/
# chown $(id -u):$(id -g) $HOME/admin.conf
# export KUBECONFIG=$HOME/admin.conf
# kubectl apply -f https://git.io/weave-kube

At this point I can run kubectl get nodes and see my master node in the list.
Repeat all the steps for minion except kubeadm init and instead running kubeadm join --token a21234.c7abc2f82e2219fd 12.34.567.89:6443 as generated by kubeadm init
This step completes and I can see master and minion(s) nodes

And now - the problem:

# kubectl get nodes
NAME       STATUS     AGE       VERSION
abc02918   NotReady   42m       v1.6.1
abc04576   NotReady   29m       v1.6.1

# kubectl describe node abc02918
Events:
  FirstSeen LastSeen    Count   From            SubObjectPath   Type        Reason          Message
  --------- --------    -----   ----            -------------   --------    ------          -------
  43m       43m     1   kubelet, sdl02918           Normal      Starting        Starting kubelet.
  43m       43m     1   kubelet, sdl02918           Warning     ImageGCFailed       unable to find data for container /
  43m       43m     29  kubelet, sdl02918           Normal      NodeHasSufficientDisk   Node sdl02918 status is now: NodeHasSufficientDisk
  43m       43m     29  kubelet, sdl02918           Normal      NodeHasSufficientMemory Node sdl02918 status is now: NodeHasSufficientMemory
  43m       43m     29  kubelet, sdl02918           Normal      NodeHasNoDiskPressure   Node sdl02918 status is now: NodeHasNoDiskPressure
  42m       42m     1   kube-proxy, sdl02918            Normal      Starting        Starting kube-proxy.

So looks like the nodes are never become ready. Any suggestions?

I imagine your weave isn't deploying properly because you are using the
pre-1.6 yaml file.

Try "kubectl apply -f https://git.io/weave-kube-1.6"

On Tue, Apr 4, 2017 at 12:24 PM, Bo Stone notifications@github.com wrote:

Ok, I did all the steps from scratch and it seems to be better. Here are
the steps that worked for me thus far and I'm running as root.

cat <

[kubernetes]
name=Kubernetes
baseurl=http://yum.kubernetes.io/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg http://yum.kubernetes.io/repos/kubernetes-el7-x86_64enabled=1gpgcheck=1repo_gpgcheck=1gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg
https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
EOF

setenforce 0

yum install -y docker kubelet kubeadm kubectl kubernetes-cni

vi /etc/systemd/system/kubelet.service.d/10-kubeadm.conf

Add -cgroup-driver=systemd to 10-kubeadm.conf and save

systemctl enable docker && systemctl start docker

systemctl enable kubelet && systemctl start kubelet

sysctl -w net.bridge.bridge-nf-call-iptables=1

systemctl stop firewalld; systemctl disable firewalld

kubeadm init

cp /etc/kubernetes/admin.conf $HOME/

chown $(id -u):$(id -g) $HOME/admin.conf

export KUBECONFIG=$HOME/admin.conf

kubectl apply -f https://git.io/weave-kube

At this point I can run kubectl get nodes and see my master node in the
list.
Repeat all the steps for minion except of course running kubeadm join
--token a21234.c7abc2f82e2219fd 12.34.567.89:6443 as generated by kubeadm
init
This step completes and I can see master and minion(s) nodes

And now - the problem:

kubectl get nodes

NAME STATUS AGE VERSION
abc02918 NotReady 42m v1.6.1
abc04576 NotReady 29m v1.6.1

kubectl describe node abc02918

Events:
FirstSeen LastSeen Count From SubObjectPath Type Reason Message
--------- -------- ----- ---- ------------- -------- ------ -------
43m 43m 1 kubelet, sdl02918 Normal Starting Starting kubelet.
43m 43m 1 kubelet, sdl02918 Warning ImageGCFailed unable to find data for container /
43m 43m 29 kubelet, sdl02918 Normal NodeHasSufficientDisk Node sdl02918 status is now: NodeHasSufficientDisk
43m 43m 29 kubelet, sdl02918 Normal NodeHasSufficientMemory Node sdl02918 status is now: NodeHasSufficientMemory
43m 43m 29 kubelet, sdl02918 Normal NodeHasNoDiskPressure Node sdl02918 status is now: NodeHasNoDiskPressure
42m 42m 1 kube-proxy, sdl02918 Normal Starting Starting kube-proxy.

So looks like the nodes are never become ready. Any suggestions?

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/kubernetes/kubernetes/issues/43815#issuecomment-291571437,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAeJQ6OFBV3s6OHmfOkUdwqQsJ1sjg23ks5rsnzMgaJpZM4MtMRe
.

Using CentOS, and 1.6.1-1 version of kubeadm, and following the steps above, i get slightly different behavior: the cluster is reported as ready but then I get stuck at this message:

[apiclient] Temporarily unable to list nodes (will retry)

In the logs, though, we still have the same error:

Apr 04 13:30:30 csc-q-docker-1.colinx.com kubelet[108575]: E0404 13:30:30.491150  108575 pod_workers.go:182] Error syncing pod 2193220cb6f65d66c0d8762189232e64 ("kube-apiserver-csc-q-docker-1.colinx.com_kube-system(2193220cb6f65d66c0d8762189232e64)"), skipping: failed to "StartContainer" for "kube-apiserver" with CrashLoopBackOff: "Back-off 20s restarting failed container=kube-apiserver pod=kube-apiserver-csc-q-docker-1.colinx.com_kube-system(2193220cb6f65d66c0d8762189232e64)"
Apr 04 13:30:30 csc-q-docker-1.colinx.com kubelet[108575]: W0404 13:30:30.524051  108575 cni.go:157] Unable to update cni config: No networks found in /etc/cni/net.d
Apr 04 13:30:30 csc-q-docker-1.colinx.com kubelet[108575]: E0404 13:30:30.524243  108575 kubelet.go:2067] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized

@sjenning That's it! I still need to deploy some images to see if everything works but I can see all nodes with Ready status! Thank you all, guys (for now :)

@bostone I followed the same recipe you used, but got different results-- i didnt get past kubeadm init. What docker version are you using? perhaps thats the difference in my case?

@dcowden It's whatever yum picks on my system Docker version 1.12.6, build 96d83a5/1.12.6. Also what helped me was to reprovision all VMs I was using instead of trying to fix on top of my previous installs

@bostone thanks. i'll downgrade to that version to see if i can get a working setup. on my system the latest is a weird 17.03.1.ce version ( evidently the latest greatest)

Not sure if it is generally useful, but I have an ansible playbook that
does all the steps for CentOS 7

https://github.com/sjenning/kubeadm-playbook

YMMV, but it at least documents the process. I also do a few things like
switch docker to use json-file logging and overlay storage.

Might be useful as a reference even if you don't actually run the playbook.

On Tue, Apr 4, 2017 at 12:55 PM, Dave Cowden notifications@github.com
wrote:

@bostone https://github.com/bostone thanks. i'll downgrade to that
version to see if i can get a working setup. on my system the latest is a
weird 17.03.1.ce version ( evidently the latest greatest)

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/kubernetes/kubernetes/issues/43815#issuecomment-291580822,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAeJQ7CUA4vxhF3T7nMc9wRu47rbWe-Kks5rsoQmgaJpZM4MtMRe
.

@sjenning Thanks! that's very helpful!

Ok, here's one complication. after running kubeadm reset on master and minion and restarting docker and kubelet services kubeadm init hangs on Created API client, waiting for the control plane to become ready once again. Can someone provide complete steps for resetting kubeadm?
@sjenning have you tried to rerun your playbook after you run it initially?

@bostone in my case moving /var/lib/etcd/ did the trick.

@autostatic the directory was empty and renaming it didn't help. @sjenning you playbook hangs, I created a ticket for you

Has anyone tried to run the dashboard on v1.6.1? I'm getting an error that states the following:

`Forbidden (403)

User "system:serviceaccount:kube-system:default" cannot list namespaces at the cluster scope. (get namespaces)
`

I guessing I need to config more RBAC, like you have to for running Flannel?

@srzjulio Can you file a new issue for that? My guess is that is something that SIG Auth and the Dashboard team should work together on. It is probably a simple-ish YAML change.

@srzjulio you need to update RBAC rules, we used these to get us going:

apiVersion: rbac.authorization.k8s.io/v1alpha1
kind: ClusterRoleBinding
metadata:
name: cluster-admin
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
subjects:

• kind: Group
  name: system:masters
• kind: Group
  name: system:authenticated
• kind: Group
  name: system:unauthenticated

Be careful -- The binding that @sbezverk has there is essentially turning off RBAC. You will have a super insecure cluster if you do that.

kind: Group
name: system:unauthenticated

This is particularly bad. It means anyone who can contact your API server, even without any credentials, can run arbitrary commands.

@jbeda it matches behavior which we had with 1.5.6 right out of the box. It gives people opportunity to review and gradually adjust security config instead of being unable to do anything with new security rules.

Actually, making system:unauthenticated a cluster admin is far worse than 1.5.6

If you want no authorization, use the AlwaysAllow authorizer

If you want to allow all while you audit, use a combination of RBAC,AlwaysAllow

That will disable anonymous requests, log RBAC denials in the API server log, but continue to allow all authenticated requests to do whatever they want

Sorry, and I iterate on this again, this has nothing to do with the original issue. And although valid problems and issues, we should move the conversation elsewhere.

Again sorry, hit enter too soon, but as things stand:

1 - release 1.6.0 cause problems
2 - not are all fixed
3 - moving to RBAC (good in my opinion) but is aproblem, why? see point 4
4 - This was not very well communicated, and there's not a lot docs/blogs or whatever to explain it
5 - I bow to all ppl trying to salvage, but we need a better way to do this

https://kubernetes.io/docs/admin/authorization/rbac/#service-account-permissions is a good guide to the most-to-least granular options you have for opening up permissions.

adding "--cgroup-driver=systemd" in the kublet causes new issue on Centos/RHEL 7.3 (fully up to date - aka docker 1.10.3):

Apr 12 14:23:25 machine01 kubelet[3026]: W0412 14:23:25.542322    3026 docker_service.go:196] No cgroup driver is set in Docker
Apr 12 14:23:25 machine01 kubelet[3026]: W0412 14:23:25.542343    3026 docker_service.go:197] Falling back to use the default driver: "cgroupfs"
Apr 12 14:23:25 machine01 kubelet[3026]: error: failed to run Kubelet: failed to create kubelet: misconfiguration: kubelet cgroup driver: "systemd" is different from docker cgroup driver: "cgroupfs"

while we can see clearly that native.cgroupdriver=systemd is set in the docker daemon:

 ps -ef|grep -i docker
root      4365     1  3 14:30 ?        00:00:33 /usr/bin/docker-current daemon --authorization-plugin=rhel-push-plugin --exec-opt native.cgroupdriver=systemd --selinux-enabled --log-driver=journald --insecure-registry 172.30.0.0/16 --storage-driver devicemapper --storage-opt dm.fs=xfs --storage-opt dm.thinpooldev=/dev/mapper/vg.docker--pool --storage-opt dm.use_deferred_removal=true --storage-opt dm.use_deferred_deletion=true

@ReSearchITEng why do not you update docker to 1.12.6? Works like a charm with this version.

@sbezverk: I just updated to 1.12.5 and now it's working! Many thanks!

Thanks all for help.
Finally fully working k8s 1.6.1 with flannel. Everything is now in ansible playbooks.
Tested on Centos/RHEL. Preparations started for Debian based also (e.g. Ubuntu), but there might needs some refining.

https://github.com/ReSearchITEng/kubeadm-playbook/blob/master/README.md

PS: work based on sjenning/kubeadm-playbook - Many thanks @sjenning !

@sjenning @ReSearchITEng
Hi, I have a vagrant+ansible playbook [0] very similar to what you have created, but I'm still unable to get it working, altought for me it is failing on the networking setup. I have tried with calico, weave and flannel, and all three fail (although with different symptoms).

I'm running these versions:
[vagrant@master ~]$ docker --version
Docker version 1.12.6, build 3a094bd/1.12.6
[vagrant@master ~]$ kubelet --version
Kubernetes v1.6.1
[vagrant@master ~]$ kubeadm version
kubeadm version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.1", GitCommit:"b0b7a323cc5a4a2019b2e9520c21c7830b7f708e", GitTreeState:"clean", BuildDate:"2017-04-03T20:33:27Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"linux/amd64"}

errors:

[vagrant@master ~]$ kubectl get all --all-namespaces
NAMESPACE     NAME                                           READY     STATUS             RESTARTS   AGE
kube-system   po/calico-etcd-gvrhd                           1/1       Running            0          47m
kube-system   po/calico-node-7jvs8                           1/2       CrashLoopBackOff   12         45m
kube-system   po/calico-node-7ljpn                           2/2       Running            0          47m
kube-system   po/calico-node-w15z3                           1/2       CrashLoopBackOff   12         45m
kube-system   po/calico-node-zq3zx                           1/2       CrashLoopBackOff   12         45m
kube-system   po/calico-policy-controller-1777954159-13x01   1/1       Running            0          47m
kube-system   po/etcd-master                                 1/1       Running            0          46m
kube-system   po/kube-apiserver-master                       1/1       Running            0          46m
kube-system   po/kube-controller-manager-master              1/1       Running            0          46m
kube-system   po/kube-dns-3913472980-16m01                   3/3       Running            0          47m
kube-system   po/kube-proxy-70bmf                            1/1       Running            0          45m
kube-system   po/kube-proxy-9642h                            1/1       Running            0          45m
kube-system   po/kube-proxy-jhtvm                            1/1       Running            0          45m
kube-system   po/kube-proxy-nb7q5                            1/1       Running            0          47m
kube-system   po/kube-scheduler-master                       1/1       Running            0          46m

NAMESPACE     NAME              CLUSTER-IP      EXTERNAL-IP   PORT(S)         AGE
default       svc/kubernetes    10.96.0.1       <none>        443/TCP         47m
kube-system   svc/calico-etcd   10.96.232.136   <none>        6666/TCP        47m
kube-system   svc/kube-dns      10.96.0.10      <none>        53/UDP,53/TCP   47m

NAMESPACE     NAME                              DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
kube-system   deploy/calico-policy-controller   1         1         1            1           47m
kube-system   deploy/kube-dns                   1         1         1            1           47m

NAMESPACE     NAME                                     DESIRED   CURRENT   READY     AGE
kube-system   rs/calico-policy-controller-1777954159   1         1         1         47m
kube-system   rs/kube-dns-3913472980                   1         1         1         47m

[vagrant@master ~]$ kubectl -n kube-system describe po/calico-node-zq3zx
Name:       calico-node-zq3zx
Namespace:  kube-system
Node:       node1/192.168.10.101
Start Time: Wed, 26 Apr 2017 19:37:35 +0000
Labels:     k8s-app=calico-node
        pod-template-generation=1
Annotations:    kubernetes.io/created-by={"kind":"SerializedReference","apiVersion":"v1","reference":{"kind":"DaemonSet","namespace":"kube-system","name":"calico-node","uid":"844cd287-2ab7-11e7-b184-5254008815b6","ap...
        scheduler.alpha.kubernetes.io/critical-pod=
Status:     Running
IP:     192.168.10.101
Controllers:    DaemonSet/calico-node
Containers:
  calico-node:
    Container ID:   docker://ca00b0a73a073a2d2e39cb0cc315b8366eaa20e2e479002dd16134b0d1e94f0b
    Image:      quay.io/calico/node:v1.1.3
    Image ID:       docker-pullable://quay.io/calico/node@sha256:8e62eee18612a6ac7bcae90afaba0ed95265baba7bf3c0ab632b7b40ddfaf603
    Port:       
    State:      Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Mon, 01 Jan 0001 00:00:00 +0000
      Finished:     Wed, 26 Apr 2017 20:21:09 +0000
    Ready:      False
    Restart Count:  12
    Requests:
      cpu:  250m
    Environment:
      ETCD_ENDPOINTS:               <set to the key 'etcd_endpoints' of config map 'calico-config'> Optional: false
      CALICO_NETWORKING_BACKEND:        <set to the key 'calico_backend' of config map 'calico-config'> Optional: false
      CALICO_DISABLE_FILE_LOGGING:      true
      FELIX_DEFAULTENDPOINTTOHOSTACTION:    ACCEPT
      CALICO_IPV4POOL_CIDR:         192.168.0.0/16
      CALICO_IPV4POOL_IPIP:         always
      FELIX_IPV6SUPPORT:            false
      FELIX_LOGSEVERITYSCREEN:          info
      IP:                   
    Mounts:
      /lib/modules from lib-modules (ro)
      /var/run/calico from var-run-calico (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from calico-cni-plugin-token-5wnmg (ro)
  install-cni:
    Container ID:   docker://442c3adfa908f76654bb54070ef5ff638e4b68e0331ea0555ae877ce583ce858
    Image:      quay.io/calico/cni:v1.7.0
    Image ID:       docker-pullable://quay.io/calico/cni@sha256:3612ffb0bff609d65311b45f4bae57fa80a05d25e1580ceb83ba4162e2ceef9f
    Port:       
    Command:
      /install-cni.sh
    State:      Running
      Started:      Wed, 26 Apr 2017 19:38:29 +0000
    Ready:      True
    Restart Count:  0
    Environment:
      ETCD_ENDPOINTS:       <set to the key 'etcd_endpoints' of config map 'calico-config'>     Optional: false
      CNI_NETWORK_CONFIG:   <set to the key 'cni_network_config' of config map 'calico-config'> Optional: false
    Mounts:
      /host/etc/cni/net.d from cni-net-dir (rw)
      /host/opt/cni/bin from cni-bin-dir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from calico-cni-plugin-token-5wnmg (ro)
Conditions:
  Type      Status
  Initialized   True 
  Ready     False 
  PodScheduled  True 
Volumes:
  lib-modules:
    Type:   HostPath (bare host directory volume)
    Path:   /lib/modules
  var-run-calico:
    Type:   HostPath (bare host directory volume)
    Path:   /var/run/calico
  cni-bin-dir:
    Type:   HostPath (bare host directory volume)
    Path:   /opt/cni/bin
  cni-net-dir:
    Type:   HostPath (bare host directory volume)
    Path:   /etc/cni/net.d
  calico-cni-plugin-token-5wnmg:
    Type:   Secret (a volume populated by a Secret)
    SecretName: calico-cni-plugin-token-5wnmg
    Optional:   false
QoS Class:  Burstable
Node-Selectors: <none>
Tolerations:    CriticalAddonsOnly=:Exists
        node-role.kubernetes.io/master=:NoSchedule
        node.alpha.kubernetes.io/notReady=:Exists:NoExecute for 300s
        node.alpha.kubernetes.io/unreachable=:Exists:NoExecute for 300s
Events:
  FirstSeen LastSeen    Count   From        SubObjectPath           Type        Reason      Message
  --------- --------    -----   ----        -------------           --------    ------      -------
  46m       46m     1   kubelet, node1  spec.containers{calico-node}    Normal      Pulling     pulling image "quay.io/calico/node:v1.1.3"
  45m       45m     1   kubelet, node1  spec.containers{calico-node}    Normal      Pulled      Successfully pulled image "quay.io/calico/node:v1.1.3"
  45m       45m     1   kubelet, node1  spec.containers{calico-node}    Normal      Created     Created container with id e035a82202b2c8490e879cb9647773158ff05def6c60b31a001e23e6d288a977
  45m       45m     1   kubelet, node1  spec.containers{calico-node}    Normal      Started     Started container with id e035a82202b2c8490e879cb9647773158ff05def6c60b31a001e23e6d288a977
  45m       45m     1   kubelet, node1  spec.containers{install-cni}    Normal      Pulling     pulling image "quay.io/calico/cni:v1.7.0"
  45m       45m     1   kubelet, node1  spec.containers{install-cni}    Normal      Pulled      Successfully pulled image "quay.io/calico/cni:v1.7.0"
  45m       45m     1   kubelet, node1  spec.containers{install-cni}    Normal      Created     Created container with id 442c3adfa908f76654bb54070ef5ff638e4b68e0331ea0555ae877ce583ce858
  45m       45m     1   kubelet, node1  spec.containers{install-cni}    Normal      Started     Started container with id 442c3adfa908f76654bb54070ef5ff638e4b68e0331ea0555ae877ce583ce858
  44m       44m     1   kubelet, node1  spec.containers{calico-node}    Normal      Created     Created container with id 163a9073070aa52ce7ee98c798ffe130a581e4fdbbc503540ed5d3b79651c549
  44m       44m     1   kubelet, node1  spec.containers{calico-node}    Normal      Started     Started container with id 163a9073070aa52ce7ee98c798ffe130a581e4fdbbc503540ed5d3b79651c549
  44m       44m     1   kubelet, node1                  Warning     FailedSync  Error syncing pod, skipping: failed to "StartContainer" for "calico-node" with CrashLoopBackOff: "Back-off 10s restarting failed container=calico-node pod=calico-node-zq3zx_kube-system(c983e5d0-2ab7-11e7-b184-5254008815b6)"

  44m   44m 1   kubelet, node1  spec.containers{calico-node}    Normal  Started     Started container with id 07453d944dfb9a4ebae57c83158e4b51f8870bcab94b4f706239f6c0b93bb62d
  44m   44m 1   kubelet, node1  spec.containers{calico-node}    Normal  Created     Created container with id 07453d944dfb9a4ebae57c83158e4b51f8870bcab94b4f706239f6c0b93bb62d
  43m   43m 2   kubelet, node1                  Warning FailedSync  Error syncing pod, skipping: failed to "StartContainer" for "calico-node" with CrashLoopBackOff: "Back-off 20s restarting failed container=calico-node pod=calico-node-zq3zx_kube-system(c983e5d0-2ab7-11e7-b184-5254008815b6)"

  43m   43m 1   kubelet, node1  spec.containers{calico-node}    Normal  Started     Started container with id 00f363848c16ff66743d54b87948133a87a97bfd32fbde2338622904d0990601
  43m   43m 1   kubelet, node1  spec.containers{calico-node}    Normal  Created     Created container with id 00f363848c16ff66743d54b87948133a87a97bfd32fbde2338622904d0990601
  42m   42m 3   kubelet, node1                  Warning FailedSync  Error syncing pod, skipping: failed to "StartContainer" for "calico-node" with CrashLoopBackOff: "Back-off 40s restarting failed container=calico-node pod=calico-node-zq3zx_kube-system(c983e5d0-2ab7-11e7-b184-5254008815b6)"

  41m   41m 1   kubelet, node1  spec.containers{calico-node}    Normal  Created     Created container with id a5aad1f1a57a361fafcaa2ee6aba244bf19925f56c5b46771cfd45e5e7fd884e
  41m   41m 1   kubelet, node1  spec.containers{calico-node}    Normal  Started     Started container with id a5aad1f1a57a361fafcaa2ee6aba244bf19925f56c5b46771cfd45e5e7fd884e
  41m   40m 6   kubelet, node1                  Warning FailedSync  Error syncing pod, skipping: failed to "StartContainer" for "calico-node" with CrashLoopBackOff: "Back-off 1m20s restarting failed container=calico-node pod=calico-node-zq3zx_kube-system(c983e5d0-2ab7-11e7-b184-5254008815b6)"

  40m   40m 1   kubelet, node1  spec.containers{calico-node}    Normal  Created     Created container with id 520ee97fe986fd726a0347cab6de5b2a8fba91f73df2d601e8b7625531ed2117
  40m   40m 1   kubelet, node1  spec.containers{calico-node}    Normal  Started     Started container with id 520ee97fe986fd726a0347cab6de5b2a8fba91f73df2d601e8b7625531ed2117
  39m   36m 12  kubelet, node1                  Warning FailedSync  Error syncing pod, skipping: failed to "StartContainer" for "calico-node" with CrashLoopBackOff: "Back-off 2m40s restarting failed container=calico-node pod=calico-node-zq3zx_kube-system(c983e5d0-2ab7-11e7-b184-5254008815b6)"

  36m   36m 1   kubelet, node1  spec.containers{calico-node}    Normal  Created     Created container with id 90be4da6fd2e8c111c3e2a91256d60656db80316c1497c29c4155b8f009f241f
  36m   36m 1   kubelet, node1  spec.containers{calico-node}    Normal  Started     Started container with id 90be4da6fd2e8c111c3e2a91256d60656db80316c1497c29c4155b8f009f241f
  31m   31m 1   kubelet, node1  spec.containers{calico-node}    Normal  Created     Created container with id bf0d93f45d5ffa2d2c42487851f80048757da5c767491f673bfecfa37fe76e48
  31m   31m 1   kubelet, node1  spec.containers{calico-node}    Normal  Started     Started container with id bf0d93f45d5ffa2d2c42487851f80048757da5c767491f673bfecfa37fe76e48
  44m   3m  12  kubelet, node1  spec.containers{calico-node}    Normal  Pulled      Container image "quay.io/calico/node:v1.1.3" already present on machine
  25m   3m  5   kubelet, node1  spec.containers{calico-node}    Normal  Started     (events with common reason combined)
  25m   3m  5   kubelet, node1  spec.containers{calico-node}    Normal  Created     (events with common reason combined)
  36m   15s 149 kubelet, node1                  Warning FailedSync  Error syncing pod, skipping: failed to "StartContainer" for "calico-node" with CrashLoopBackOff: "Back-off 5m0s restarting failed container=calico-node pod=calico-node-zq3zx_kube-system(c983e5d0-2ab7-11e7-b184-5254008815b6)"

  44m   15s 173 kubelet, node1  spec.containers{calico-node}    Warning BackOff Back-off restarting failed container

This looks like key information, but I'm not sure how to fix it:

[vagrant@master ~]$ kubectl -n kube-system logs calico-node-zq3zx calico-node
Skipping datastore connection test
time="2017-04-26T20:20:39Z" level=info msg="NODENAME environment not specified - check HOSTNAME" 
time="2017-04-26T20:20:39Z" level=info msg="Loading config from environment" 
ERROR: Unable to access datastore to query node configuration
Terminating
time="2017-04-26T20:21:09Z" level=info msg="Unhandled error: client: etcd cluster is unavailable or misconfigured; error #0: client: endpoint http://10.96.232.136:6666 exceeded header timeout
" 
time="2017-04-26T20:21:09Z" level=info msg="Unable to query node configuration" Name=node1 error="client: etcd cluster is unavailable or misconfigured; error #0: client: endpoint http://10.96.232.136:6666 exceeded header timeout
" 
Calico node failed to start

Any help would be greatly appreciated...

[0]- https://github.com/thiagodasilva/kubernetes-swift/tree/master/roles

I could not identify what's wrong on your end.
I strongly suggest you try to create a separate installation using the playbooks here: https://github.com/ReSearchITEng/kubeadm-playbook and try to see what's the difference.
PS: my last tests are with 1.6.2 , both kube* and images and seems fine.

@kelseyhightower

@ReSearchITEng sorry I forgot to report back, but I eventually got it to work, my vagrant+ansible files are here: https://github.com/thiagodasilva/kubernetes-swift

I also got the same issue,but I just copy the cni config on the master node to the corresponding location of the worker node,then it became OK.

systemctl status kubelet.service -l

● kubelet.service - kubelet: The Kubernetes Node Agent
Loaded: loaded (/etc/systemd/system/kubelet.service; enabled; vendor preset: disabled)
Drop-In: /etc/systemd/system/kubelet.service.d
└─10-kubeadm.conf
Active: active (running) since Tue 2017-06-06 10:42:00 CST; 18min ago
Docs: http://kubernetes.io/docs/
Main PID: 4414 (kubelet)
Memory: 43.0M
CGroup: /system.slice/kubelet.service
├─4414 /usr/bin/kubelet --kubeconfig=/etc/kubernetes/kubelet.conf --require-kubeconfig=true --pod-manifest-path=/etc/kubernetes/manifests --network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin --cluster-dns=10.96.0.10 --cluster-domain=cluster.local --authorizatio-ca-file=/etc/kubernetes/pki/ca.crt --cgroup-driver=cgroupfs
└─4493 journalctl -k -f

Jun 06 10:59:46 contiv1.com kubelet[4414]: W0606 10:59:46.215827 4414 cni.go:157] Unable to update cni config: No networks found in /etc/cni/net.d
Jun 06 10:59:46 contiv1.com kubelet[4414]: E0606 10:59:46.215972 4414 kubelet.go:2067] Container runtime network not ready: NetworkReady=false ready message:docker: network plugin is not ready: cni config uninitialized
Jun 06 10:59:51 contiv1.com kubelet[4414]: W0606 10:59:51.216843 4414 cni.go:157] Unable to update cni config: No networks found in /etc/cni/net.d
Jun 06 10:59:51 contiv1.com kubelet[4414]: E0606 10:59:51.216942 4414 kubelet.go:2067] Container runtime network not ready: NetworkReady=false ready message:docker: network plugin is not ready: cni config uninitialized
Jun 06 10:59:56 contiv1.com kubelet[4414]: W0606 10:59:56.217923 4414 cni.go:157] Unable to update cni config: No networks found in /etc/cni/net.d
Jun 06 10:59:56 contiv1.com kubelet[4414]: E0606 10:59:56.218113 4414 kubelet.go:2067] Container runtime network not ready: NetworkReady=false ready message:docker: network plugin is not ready: cni config uninitialized
Jun 06 11:00:01 contiv1.com kubelet[4414]: W0606 11:00:01.219251 4414 cni.go:157] Unable to update cni config: No networks found in /etc/cni/net.d
Jun 06 11:00:01 contiv1.com kubelet[4414]: E0606 11:00:01.219382 4414 kubelet.go:2067] Container runtime network not ready: NetworkReady=false ready message:docker: network plugin is not ready: cni config uninitialized
Jun 06 11:00:06 contiv1.com kubelet[4414]: W0606 11:00:06.220396 4414 cni.go:157] Unable to update cni config: No networks found in /etc/cni/net.d
Jun 06 11:00:06 contiv1.com kubelet[4414]: E0606 11:00:06.220575 4414 kubelet.go:2067] Container runtime network not ready: NetworkReady=false ready message:docker: network plugin is not ready: cni config uninitialized

The status of all nodes:
[root@swarm net.d]# kubectl get node
NAME STATUS AGE VERSION
contiv1.com Ready 1h v1.6.4
contiv2.com Ready 1h v1.6.4
swarm.com Ready 1h v1.6.4

Any resolution on this ? I was not able to do this even after trying all the mentioned resolutions.

Being new to setting up Kubernetes I get superconfused. I tried following https://medium.com/@SystemMining/setup-kubenetes-cluster-on-ubuntu-16-04-with-kubeadm-336f4061d929 that use weave-kube for network but I'm also stuck with the same issue. Any way to solve this?
Ready False Mon, 12 Jun 2017 16:55:16 +0200 Mon, 12 Jun 2017 12:22:45 +0200 KubeletNotReady runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized

Why is this still an issue? Ubuntu 16.04/CentOS 7.3 with latest updates using the official k8s repos with 1.6.4 and following the steps outlined here: https://kubernetes.io/docs/setup/independent/install-kubeadm/
Jun 13 09:57:21 tme-lnx1-centos kubelet: W0613 09:57:21.871413 10321 cni.go:157] Unable to update cni config: No networks found in /etc/cni/net.d
Jun 13 09:57:21 tme-lnx1-centos kubelet: E0613 09:57:21.871788 10321 kubelet.go:2067] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized

@drajen No, this affected _only v1.6.0_. It's expected that kubelet doesn't find a network since you haven't installed any. For example, just run

kubectl apply -f https://git.io/weave-kube-1.6

to install Weave Net and those problems will go away. You can choose to install Flannel, Calico, Canal or whatever CNI network if you'd like

@luxas I keep seeing references to this but how am I suppose to apply something to a cluster that is not running? I have nothing to connect to.

@drajen I think @luxas' point is that this is the wrong place to be asking about setup.
The various setup guides will get you past this point - the typical missing step in older guides, which luxas helpfully mentions, in that you need to apply a network configuration before everything will start working properly.

Yeah, it is might be non-obvious and we're sorry for that, but we can't have one single providers name there either.

Chatted with @drajen on Slack and the issue was cgroup related, the kubelet was unhealthy and wasn't able to create any Pods, hence the issue.

Thanks to @luxas for wrestling my particular problem to the ground: https://github.com/kubernetes/kubeadm/issues/302

Still hitting this in arch on 1.7, is there a quick fix anywhere?

Edit:

kubectl apply -f https://git.io/weave-kube-1.6

did the trick, looks like we just needed CNI running

At least for CentOS/RHEL, make sure you update /etc/systemd/system/kubelet.service.d/10-kubeadm.conf and add the flag --cgroup-driver="systemd"

If you reinstall again on the same machine, this is a full proper reset:
https://github.com/ReSearchITEng/kubeadm-playbook/blob/master/reset.yml
(this is required especially if you use flanneld)
If you want to do all in one, you may want to use the entire project: https://github.com/ReSearchITEng/kubeadm-playbook/

I hit this issue, and absolutely nothing I read above worked. So I tried again with a much more controlled setup, switching from Ubuntu to the latest CoreOS, going to an earlier version of k8s to start with, and in general being very anal about every last thing installed into each VM. I am NOT using kubeadm, but instead a combination of vagrant and ansible.

(why? because I had no idea what was going on in kubeadm and figured this way at least I'd have control and be able to bypass any overzealous preflight checks, not to mention feeling like I had more automation control in general, and also not having to worry about the warning to do-not-apply-alpha-software-in-production.)

When I tried this setup with an older (1.4.3) edition of k8s, this approach was golden. I then tried upgrading to 1.71. Once again, I am STILL hitting this same issue in spite of using no kubeadm at all.

I have confirmed that I am running calico in each of my nodes, including the Master and the three potential Workers. ALL of my nodes are reporting as NotReady, so I'm not really sure how I could apply weave (or anything else) to get it running.

This whole thing just seems chicken/egg ... can't allocate a pod because networking is failing, but need networking running to create a network at /etc/cni/net.d in order to be able to allocate pods. And again, all this was working a few hours ago with k8s 1.4.3. I'm very frustrated!

I'd appreciate any insights anybody could give.

Footnotes:

On master: journalctl -r -u kubelet gives me

Jul 24 00:48:16 rogue-kube-master-01 kubelet-wrapper[7647]: E0724 00:48:16.592274 7647 kubelet.go:2136] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is no
Jul 24 00:48:16 rogue-kube-master-01 kubelet-wrapper[7647]: W0724 00:48:16.590588 7647 cni.go:189] Unable to update cni config: No networks found in /etc/cni/net.d

docker ps | grep calico

(truncated for readability)
cde... quay.io/calico/leader-elector@sha256:... "/run.sh --election=c" 8 hours ago Up 8 hours
f72... calico/kube-policy-controller@sha256:... "/dist/controller" 8 hours ago Up 8 hours
c47... gcr.io/google_containers/pause-amd64:3.0 "/pause" 8 hours ago Up 8 hours

There is no /etc/cni/net.d

From my kubectl box:
kubectl get nodes
10.0.0.111 NotReady,SchedulingDisabled 8h v1.7.1+coreos.0
10.0.0.121 NotReady 8h v1.7.1+coreos.0
10.0.0.122 NotReady 8h v1.7.1+coreos.0
10.0.0.123 NotReady 8h v1.7.1+coreos.0

kubectl apply -f https://git.io/weave-kube-1.6

DID NOT fix anything and in fact only seems to create more problems.

bill@rogue:~/vagrant_rogue/rogue-cluster$ kubectl apply -f https://git.io/weave-kube-1.6
serviceaccount "weave-net" created
clusterrolebinding "weave-net" created
daemonset "weave-net" created
Error from server (Forbidden): clusterroles.rbac.authorization.k8s.io "weave-net" is forbidden: attempt to grant extra privileges: [PolicyRule{Resources:["pods"], APIGroups:[""], Verbs:["get"]} PolicyRule{Resources:["pods"], APIGroups:[""], Verbs:["list"]} PolicyRule{Resources:["pods"], APIGroups:[""], Verbs:["watch"]} PolicyRule{Resources:["namespaces"], APIGroups:[""], Verbs:["get"]} PolicyRule{Resources:["namespaces"], APIGroups:[""], Verbs:["list"]} PolicyRule{Resources:["namespaces"], APIGroups:[""], Verbs:["watch"]} PolicyRule{Resources:["nodes"], APIGroups:[""], Verbs:["get"]} PolicyRule{Resources:["nodes"], APIGroups:[""], Verbs:["list"]} PolicyRule{Resources:["nodes"], APIGroups:[""], Verbs:["watch"]} PolicyRule{Resources:["networkpolicies"], APIGroups:["extensions"], Verbs:["get"]} PolicyRule{Resources:["networkpolicies"], APIGroups:["extensions"], Verbs:["list"]} PolicyRule{Resources:["networkpolicies"], APIGroups:["extensions"], Verbs:["watch"]}] user=&{kube-admin [system:authenticated] map[]} ownerrules=[] ruleResolutionErrors=[]

bill@rogue:~/vagrant_rogue/rogue-cluster$ kubectl get pods --namespace=kube-system
NAME READY STATUS RESTARTS AGE
kube-apiserver-10.0.0.111 1/1 Running 1 8h
kube-controller-manager-10.0.0.111 1/1 Running 1 8h
kube-dns-v20-fcl01 0/3 Pending 0 8h
kube-proxy-10.0.0.111 1/1 Running 1 8h
kube-proxy-10.0.0.121 1/1 Running 1 8h
kube-proxy-10.0.0.122 1/1 Running 1 8h
kube-proxy-10.0.0.123 1/1 Running 1 8h
kube-scheduler-10.0.0.111 1/1 Running 1 8h
kubernetes-dashboard-v1.4.1-29zzk 0/1 Pending 0 8h
weave-net-2lplj 1/2 CrashLoopBackOff 3 3m
weave-net-2nbgd 1/2 CrashLoopBackOff 3 3m
weave-net-fdr1v 2/2 Running 0 3m
weave-net-jzv50 1/2 CrashLoopBackOff 3 3m

Deeper investigation of the weave errors indicate that they either (a) cannot connect to the apiserver, or else (b) in the case of the one marked "Running" it's complaining that it cannot connect to itself.

@billmilligan Having similar issues, dns stop working few minutes after container started

@Paxa @billmilligan If you want to get help, don't comment on this issue. Instead, open new ones in the kubeadm repo with sufficient details requested.

@luxas Respectfully, I have to question whether this is a new issue. Since I am getting the exact same result setting up k8s without kubeadm as everyone else is getting with kubeadm, this seems to eliminate kubeadm as the source of the problem. Perhaps this issue ought to be renamed accordingly?

@billmilligan respectfully, since the issue is about kubeadm, and your able to reproduce it without kubeadm, then it is the wrong place to file it? I think this thread solved the kubeadm issue. This is a new issue. I think it will get more attention as a new issue. As people on this thread think its already solved and are ignoring it.

I use kubeadm and was effected by this issue, and it has been solved since 1.6.1. I've deployed losts of k8s's since, so I really do think you have a separate issue.

@kfox1111 Thanks for the feedback. I'll file a new issue, but the number of people who still seem to be facing it elsewhere in 1.7.x still makes me wonder.

TL;DR;

The error message

runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized

is NOT necessarily bad.

That error message tells you that you have to plugin in a third party CNI spec implementation provider.

What is CNI and how does integrate with Kubernetes?

CNI stands for Container Network Interface and defines a specification that kubelet uses for creating a network for the cluster. See this page for more information how Kubernetes uses the CNI spec to create a network for the cluster.

Kubernetes doesn't care how the network is created as long as it satisfies the CNI spec.

kubelet is in charge of connecting new Pods to the network (can be an overlay network for instance).
kubelet reads a configuration directory (often /etc/cni/net.d) for CNI networks to use.
When a new Pod is created, the kubelet reads files in the configuration directory, exec's out to the CNI binary specified in the config file (the binary is often in /opt/cni/bin). The binary that will be executed belongs to and is installed by a third-party (like Weave, Flannel, Calico, etc.).

kubeadm is a generic tool to spin up Kubernetes clusters and does not know what networking solution you want and doesn't favor anyone specific. After kubeadm init is run, there is no such CNI binary nor configuration. This means the kubeadm init IS NOT ENOUGH to get a fully working cluster up and running.

This means, that after kubeadm init, the kubelet logs will say

runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized

this is very much expected. If this wasn't the case, we would have favored a specific network provider.

So how do I "fix" this error?
The next step in the kubeadm getting started guide is "Installing a Pod network".
This means, kubectl apply a manifest from your preferred CNI network provider.

The DaemonSet will copy out the CNI binaries needed to /opt/cni/bin and the needed configuration to /etc/cni/net.d/. Also it will run the actual daemon that sets up the network between the Nodes (by writing iptables rules for instance).

After the CNI provider is installed, the kubelet will notice that "oh I have some information how to set up the network", and will use the 3rd-party configuration and binaries.

And when the network is set up by the 3rd-party provider (by kubelet invoking it), the Node will mark itself Ready.

How is this issue related to kubeadm?

Late in the v1.6 cycle, a PR was merged that changed the way kubelet reported the Ready/NotReady status. In earlier releases, kubelet had always reported Ready status, regardless of whether the CNI network was set up or not. This was actually kind of wrong, and changed to respect the CNI network status. That is, NotReady when CNI was uninitialized and Ready when initialized.

kubeadm in v1.6.0 waited wrongly for the master node to be in the Ready state before proceeding with the rest of the kubeadm init tasks. When the kubelet behavior changed to report NotReady when CNI was uninitialized, kubeadm would wait forever for the Node to get Ready.

THAT WAIT MISCONCEPTION ON THE KUBEADM SIDE IS WHAT THIS ISSUE IS ABOUT

However, we quickly fixed the regression in v1.6.1 and released it some days after v1.6.0.

Please read the retrospective for more information about this, and why v1.6.0 could be released with this flaw.

So, what do you do if you think you see this issue in kubeadm v1.6.1+?

Well, I really think you don't. This issue is about when kubeadm init is deadlocking.
No users or maintainers have seen that in v1.6.1+.

What you WILL see though is

runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized

after every kubeadm init in all versions above v1.6, but that IS NOT BAD

Anyway, please open a new issue if you see something unexpected with kubeadm

Please do not comment more on this issue. Instead open a new one.

@billmilligan So you only have to kubectl apply a CNI provider's manifest to get your cluster up and running I think

I'm pretty much summarizing what has been said above, but hopefully in a more clear and detailed way.
If you have questions about how CNI work, please refer to the normal support channels like StackOverflow, an issue or Slack.

(Lastly, sorry for that much bold text, but I felt like it was needed to get people's attention.)