Kubernetes: kube-proxy ipvs模式几个小时后挂起，需要手动重启

/ sig网络

同样的错误。 从1.11.1升级。 一切都变好了，除了kube-proxy变得不稳定之外，尽管它们的端点还可以，但有时Pod不能访问svc。而且与名称解析相同。 所以我认为这里可能存在重要的错误。

/ area ipvs

@gjcarneiro您解决了这个问题吗？

好吧，我将群集切换回默认（iptables）模式，此后似乎非常稳定。

@ kubernetes / sig网络错误

@berlinsaint ：重申提及以触发通知：
@ kubernetes / sig-network-bugs

同样的问题，1.12.2和1.13.0-beta1。 @ m1093782566 @狮威

@gjcarneiro您的意思是在内核4.15中，它有同样的问题吗？

@annProg

same problem ,both 1.12.2 and 1.13.0-beta1. @m1093782566 @Lion-Wei

你邮箱多少啊啊，，认识下。。少年

@annProg

same problem ,both 1.12.2 and 1.13.0-beta1. @m1093782566 @Lion-Wei

你邮箱多少啊啊，，认识下。。少年

点我头像就看到了

我在集中式日志记录系统中也屡屡看到这种情况：

  | Time | sys_name | log_lvl | tag | log_msg
  | November 27th 2018, 17:56:23.501 | k8lab2bs | 6 | kube.kube-system.kube-proxy | lw: remote out of the list: 10.12.12.1:443/TCP/10.12.100.133:6443
  | November 27th 2018, 17:56:23.500 | k8lab2bs | 6 | kube.kube-system.kube-proxy | Deleting rs: 10.12.12.1:443/TCP/10.12.100.133:6443
  | November 27th 2018, 17:56:23.497 | k8lab2bs | 6 | kube.kube-system.kube-proxy | Trying to delete rs: 10.12.12.1:443/TCP/10.12.100.133:6443

我正在运行v1.12.3（全新安装，而不是升级）。

大家好。 感谢您的报告，我将进行研究，并尝试弄清发生了什么，我将在这里报告任何进展。

除了所发生的事情（很容易发现）之外，我对IPVS充满信心，除非有适当的代码来检测并重置损坏的netlink套接字。 因为从日志和跟踪信息中我可以猜测的是，netlink套接字进入了损坏状态，并且从/向其读取/写入的任务将永远挂起。 我认为kube-proxy需要具有检测此问题的代码，如果发生这种情况，请从头开始创建一个新的netlink套接字，并可能重新启动处理该问题的任务。 但是，不，我不是自愿参加，对不起，我什至都不知道Go。

因为从日志和跟踪信息中我可以猜测的是，netlink套接字进入了损坏状态，并且从/向其读取/写入的任务将永远挂起。

几天前，我们升级了go netlink版本，以支持正常终止。 netlink库之前运行非常稳定。 我不确定是否是原因。

1.12.3具有相同的错误。
过滤INFO日志后，
执行kubectl -n kube-system logs kube-proxy-29zh8|egrep ^E; kubectl -n kube-system logs kube-proxy-2pqdn|egrep ^E; kubectl -n kube-system logs kube-proxy-4xw8q|egrep ^E; kubectl -n kube-system logs kube-proxy-6j4bc|egrep ^E; kubectl -n kube-system logs kube-proxy-brbjb|egrep ^E; kubectl -n kube-system logs kube-proxy-r6cg2|egrep ^E; kubectl -n kube-system logs kube-proxy-rpl6t|egrep ^E;
并且所有代理节点日志是

bash E1129 01:01:10.862180 1 proxier.go:430] Failed to execute iptables-restore for nat: exit status 1 (iptables-restore: line 7 failed E1129 21:09:11.422553 1 proxier.go:1485] Failed to get IPVS service, error: Expected only one service obtained=0 E1129 21:09:11.422699 1 proxier.go:1116] Failed to sync endpoint for service: 172.17.0.1:31800/TCP, err: Expected only one service obtained=0 E1129 01:01:11.618125 1 proxier.go:430] Failed to execute iptables-restore for nat: exit status 1 (iptables-restore: line 7 failed E1129 01:01:11.905626 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address E1129 01:01:41.771596 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address E1129 01:02:11.923872 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address E1129 01:02:42.060658 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address E1129 01:03:12.326762 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address E1129 01:03:42.718417 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address E1129 01:04:12.859933 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address E1129 01:04:42.963527 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address E1129 01:05:13.078486 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address E1129 01:05:43.185141 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address E1129 01:06:13.303090 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address E1129 01:06:43.408582 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address E1129 01:07:13.550358 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address E1129 01:07:43.686235 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address E1129 01:08:13.786326 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address E1129 01:08:43.981764 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address E1129 01:09:14.188053 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address E1129 01:09:44.316316 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address E1129 01:10:14.440466 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address E1129 01:10:44.550092 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address E1129 01:11:14.658190 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address E1129 01:11:44.765396 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address E1129 01:12:14.975724 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address E1129 01:12:45.085632 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address E1129 01:13:15.317679 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address E1129 01:13:45.432469 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address E1129 01:14:15.689681 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address E1129 01:14:45.813565 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address E1129 01:15:33.434931 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address E1129 01:16:03.578017 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address E1129 01:16:33.709744 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address E1129 01:17:03.823325 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address E1129 01:17:33.935191 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address E1129 01:18:04.053183 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address E1129 01:18:34.248994 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address E1129 01:19:04.375946 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address E1129 01:19:34.552779 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address E1129 01:19:36.596182 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address E1129 01:19:47.433846 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address E1129 01:20:17.534206 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address E1129 01:20:47.682350 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address E1129 01:21:17.825175 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address E1129 01:21:47.942548 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address E1129 01:22:18.055820 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address E1129 01:22:48.149494 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address E1129 01:23:18.564985 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address E1129 01:23:48.678462 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address E1129 01:24:18.836217 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address E1129 01:24:48.991344 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address E1129 01:25:19.091032 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address E1129 01:25:49.187178 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address E1129 01:26:19.301770 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address E1129 01:26:49.425812 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address E1129 01:27:19.606711 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address E1129 01:27:49.721224 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address E1129 01:28:20.109117 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address E1129 01:28:50.211574 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address E1129 01:29:20.405007 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address E1129 01:29:50.507766 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address E1129 01:30:20.605471 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address E1129 01:30:50.702798 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address E1129 01:31:20.807400 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address E1129 01:31:50.903049 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address E1129 01:32:21.005618 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address E1129 01:32:51.106378 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address E1129 01:33:21.224447 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address E1129 01:33:51.354916 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address E1129 01:34:21.467898 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address E1129 01:34:51.583216 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address E1129 01:35:21.691099 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address E1129 01:35:51.797151 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address E1129 01:36:21.921057 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address E1129 01:36:52.083821 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address E1129 01:37:02.445737 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address E1129 01:37:07.427811 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address E1129 01:37:37.546362 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address E1129 01:38:07.651769 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address E1129 01:38:37.797858 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address E1129 01:39:08.064112 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address E1129 01:39:38.181089 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address E1129 01:40:08.282892 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address E1129 01:40:38.390167 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address E1129 01:41:08.508549 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address E1129 01:41:38.617921 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address E1129 01:42:08.728384 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address E1129 01:42:38.844074 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address E1129 01:43:08.973517 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address E1129 01:43:39.092095 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address E1129 01:44:09.202329 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address E1129 01:44:39.363721 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address E1129 01:45:09.476240 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address E1129 01:45:39.764944 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address E1129 01:46:09.894587 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address E1129 01:46:40.013388 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address E1129 01:47:10.127894 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address E1129 01:47:40.236119 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address E1129 01:48:10.397111 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address E1129 01:48:40.600084 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address E1129 01:49:10.877034 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address E1129 01:49:41.023593 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address E1129 01:50:11.149894 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address E1129 01:50:41.262320 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address E1129 01:51:11.410407 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address E1129 01:51:41.547057 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address E1129 01:52:11.609974 1 proxier.go:1544] Failed to add destination: 172.31.200.9:443, error: file exists E1129 01:52:11.610088 1 graceful_termination.go:89] Try delete rs "10.96.0.1:443/TCP/172.31.10.38:6443" err: invalid argument E1129 01:52:11.616690 1 proxier.go:1485] Failed to get IPVS service, error: Expected only one service obtained=0 E1129 01:52:11.616767 1 proxier.go:1116] Failed to sync endpoint for service: 172.18.0.1:31645/TCP, err: Expected only one service obtained=0 E1129 01:01:12.181112 1 proxier.go:430] Failed to execute iptables-restore for nat: exit status 1 (iptables-restore: line 7 failed E1129 01:01:22.278474 1 proxier.go:430] Failed to execute iptables-restore for nat: exit status 1 (iptables-restore: line 7 failed E1129 01:01:21.459378 1 proxier.go:430] Failed to execute iptables-restore for nat: exit status 1 (iptables-restore: line 7 failed E1129 01:01:17.442316 1 proxier.go:430] Failed to execute iptables-restore for nat: exit status 1 (iptables-restore: line 7 failed E1129 20:10:17.955872 1 proxier.go:1455] Failed to add IPVS service "dev-fffrf/mysql-testportal:mysql": file exists E1129 20:10:17.955863 1 graceful_termination.go:89] Try delete rs "10.100.217.68:15004/TCP/192.100.143.225:15004" err: Failed to delete rs "10.100.217.68:15004/TCP/192.100.143.225:15004", can't find the real server E1129 20:10:17.955892 1 proxier.go:1119] Failed to sync service: 172.17.0.1:31645/TCP, err: file exists E1129 20:10:17.956412 1 proxier.go:1485] Failed to get IPVS service, error: invalid argument E1129 20:10:17.956417 1 graceful_termination.go:89] Try delete rs "10.96.0.1:443/TCP/172.31.10.38:6443" err: Failed to delete rs "10.96.0.1:443/TCP/172.31.10.38:6443", can't find the real server E1129 20:10:17.956426 1 proxier.go:1116] Failed to sync endpoint for service: 172.31.200.67:31645/TCP, err: invalid argument E1129 20:10:17.956448 1 graceful_termination.go:89] Try delete rs "10.101.1.98:5672/TCP/192.100.29.180:5672" err: invalid argument E1129 20:10:17.956547 1 graceful_termination.go:183] Try flush graceful termination list err E1129 01:01:17.484679 1 proxier.go:430] Failed to execute iptables-restore for nat: exit status 1 (iptables-restore: line 7 failed E1129 02:52:17.498135 1 graceful_termination.go:89] Try delete rs "10.96.0.1:443/TCP/172.31.10.38:6443" err: invalid argument E1129 02:52:17.498396 1 graceful_termination.go:183] Try flush graceful termination list err E1129 06:28:17.607394 1 graceful_termination.go:89] Try delete rs "10.96.0.1:443/TCP/172.31.10.38:6443" err: device or resource busy E1129 06:28:17.607513 1 graceful_termination.go:183] Try flush graceful termination list err E1129 10:02:17.718458 1 proxier.go:1544] Failed to add destination: 192.100.183.139:3306, error: file exists E1129 10:02:17.718606 1 proxier.go:1544] Failed to add destination: 192.100.183.152:8080, error: invalid argument

顺便说一句。 从我的日志上方看，kube-proxy似乎在iptables-restore上仍然有很多问题。 我所有的节点都是UBUNTU 16.04，它具有iptable的1.6.0版本。
@ m1093782566 @狮威

从1.12.1升级时，我们会看到这种情况。 因此，它必须位于1.12.1和1.12.2的增量中。

问题与正常终止有关，我认为我们已经修复了v1.13.1中的错误。

@ m1093782566会将修补程序

从1.11.2升级到1.11.5时保持不变。

@ m1093782566您能否提供有关此修复程序的更多信息？ 我有兴趣查看提交。

这太疯狂了，我们今天早些时候真的被咬了。

从1.12.1升级时，我们会看到这种情况。 因此，它必须位于1.12.1和1.12.2的增量中。

从1.11.2升级到1.11.5时保持不变。

臭虫被挑出来了吗？

看起来像PR可以解决此问题： https :

真的@ Quentin-M

是的，这可能来自delete函数中的修复（与PR的标题没有真正的关系）。

@ m1093782566我可以为1.12创建樱桃选择。 我们要向后移植完整的PR还是仅删除功能中的修复程序？ （以避免更改1.12中的UDP流的行为）

臭虫被挑出来了吗？

@ Quentin-M似乎是由https://github.com/kubernetes/kubernetes/pull/66012引入了该错误，该错误已被选为1.11.5和1.12.2版本：
https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.11.md#changelog -since-v1114
https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.12.md#other -notable-changes-1

我认为所有＃71515都应挑剔。 总体上，仅以＃66012引入和终止了优美的终止，因此行为更改已经发生。

k8s.gcr.io/kube-proxy:v1.13.1-beta.0在我们的集群上仍然过时。 其中有60％在不到24小时内被卡住。 现在尝试1.12.1。

@ Quentin-M：您可以检查kube-proxy上的日志并确认这些消息类型消失了吗：

Nov 15 07:49:52 hex-48b-pm kube-proxy[5389]: I1115 07:49:52.393664    5389 graceful_termination.go:160] Trying to delete rs: 10.128.33.237:6379/TCP/10.125.6.34:6379
Nov 15 07:49:52 hex-48b-pm kube-proxy[5389]: E1115 07:49:52.393690    5389 graceful_termination.go:89] Try delete rs "10.128.33.237:6379/TCP/10.125.6.34:6379" err: Failed to delete rs "10.128.33.237:6379/TCP/10.125.6.34:6379", can't find the real server
Nov 15 07:49:52 hex-48b-pm kube-proxy[5389]: I1115 07:49:52.393697    5389 graceful_termination.go:93] lw: remote out of the list: 10.128.33.237:6379/TCP/10.125.6.34:6379

Nov 15 07:51:22 hex-48b-pm kube-proxy[5389]: E1115 07:51:22.420950    5389 proxier.go:1455] Failed to add IPVS service "monitoring/extradata-inserter:": file exists

两者都与PR中固定的删除功能有关。 如果这些都消失了并且您仍然看到问题，则此修复程序还不够

另外，如果一切都与1.12.1 ，则很有可能是由于正常终止而引起的问题（＃66012）

我想我有一个主意：我想知道如果我们尝试添加仍处于正常终止状态的真实服务器会发生什么情况。 在这种情况下，我们应该将权重增加到1并将其从宽容终止列表中删除。

我将尝试尽快测试这种情况（但这会很棘手，因为下周我要在Kubecon上）

@ Lion-Wei您怎么看？

好的，我刚刚进行了测试，并且效果很好。 对此负责的代码在这里： https ：

在那种情况下，RS从gracefulDelete列表和IPVS RS中删除。 我不知道将权重设置为1是否会更好

我将根据日志的这一部分尝试其他操作，这似乎指向尝试同步已删除服务的端点

Nov 15 07:51:22 hex-48b-pm kube-proxy[5389]: E1115 07:51:22.420734    5389 proxier.go:1496] Failed to list IPVS destinations, error: invalid argument
Nov 15 07:51:22 hex-48b-pm kube-proxy[5389]: E1115 07:51:22.420763    5389 proxier.go:809] Failed to sync endpoint for service: 10.128.33.156:8125/TCP, err: invalid argument
Nov 15 07:51:22 hex-48b-pm kube-proxy[5389]: E1115 07:51:22.420889    5389 proxier.go:1485] Failed to get IPVS service, error: Expected only one service obtained=0
Nov 15 07:51:22 hex-48b-pm kube-proxy[5389]: E1115 07:51:22.420903    5389 proxier.go:809] Failed to sync endpoint for service: 10.128.35.150:6379/TCP, err: Expected only one service obtained=0
Nov 15 07:51:22 hex-48b-pm kube-proxy[5389]: E1115 07:51:22.420950    5389 proxier.go:1455] Failed to add IPVS service "monitoring/extradata-inserter:": file exists
Nov 15 07:51:22 hex-48b-pm kube-proxy[5389]: E1115 07:51:22.420961    5389 proxier.go:812] Failed to sync service: 10.128.34.54:80/TCP, err: file exists
Nov 15 07:51:22 hex-48b-pm kube-proxy[5389]: E1115 07:51:22.421886    5389 proxier.go:1544] Failed to add destination: 10.125.27.66:80, error: file exists

在快速检查之后，确定删除问题的提交不在v1.13.1-beta.0或11月7日削减的v1.14.0-alpha.0中
我认为它将在下一个1.14 Alpha版本中，它将基于主版本

我也将为1.13创建樱桃精选

从1.12.0升级到1.13.0时遇到问题，似乎与此相关，但不一样。

我们无法通过恢复到iptables来解决它。 请帮我。

我已经将其发布在stackoverflow上

在1.11.5上也是如此。 临时从ipvs转换为iptables。

与1.11.5相同。 由于存在漏洞，因此无法回滚至1.11.4。 不得不切换到iptables。 此修复程序是否将在1.11.x中发布？

您可以非常具体地回滚kube-proxy，而不会受到漏洞的影响。

@ Quentin-M是的，这个想法已经打动了我，但是另一方面，如果我做到了，我将不会优雅地终止，而iptables一直都存在。 不确定我们的客户是否希望更改。

@ Quentin-M是否表示没有针对1.11.x的修复程序来解决此错误？

@emptywee我为1.11创建了一个摘樱桃： https :
希望它将很快合并，并且将在1.11.6

它已在1.12（将在1.12.4 ）和1.13（将在1.13.1 ）中合并。

樱桃签也刚刚在1.11中合并
我们只需要等待3个分支中的下一个补丁发布（但是如果您想对其进行测试，那么在我可以轻松创建自定义的kube-proxy构建之前（或者更容易地，使用该补丁的hyperkube docker映像）

非常感谢@lbernail ！ 一个公开可用的docker映像将很高兴！ 我很想测试一下！

该修复程序已经精选到v1.11，v1.12和v1.13。

您能否对其进行测试，以便我们解决此问题。

@ m1093782566我正在测试。 顺便说一句，社区何时会为此发布小版本？

感谢@berlinsaint ，请让我测试结果。

社区何时会为此发布小版本？

带有修复程序的新小版本将在不到2周的时间内发布。

@emptywee ：在这里： lbernail/hyperkube:v1.12.4-beta.1
让我们知道它如何为您服务！

@lbernail可以在v1.11.5上与其他所有功能一起使用吗？ 还是应该首先将kubernetes集群升级到v1.12.4？ 无论如何，非常感谢！

@emptywee是的，它应该可以正常工作（我已经在1.10集群上运行kube-proxy 1.11几个月了）

我也可以在今天晚些时候进行1.11构建

@lbernail是的，拥有来自同一发行版的图像会很好。

@emptywee您在这里： lbernail/hyperkube:v1.11.6-beta.1

@ m1093782566 @lbernail经过一夜的困扰，ipvs sync错误日志不再出现了，而
它至少在一夜前发生一次。 但是仍然需要更多的时间来验证它，非常感谢，这让我很烦恼了很长时间。

我将从明天（1.11）开始测试，我也会给它几天时间。

@lbernail我尝试过您lbernail/hyperkube:v1.11.6-beta.1 （谢谢！）
当我们停止看到can't find the real server日志消息时，我们立即认为事情已解决。 但是，一天半之后，我们发现3/6群集节点上的过时ipvs条目存在问题。
这是损坏的群集节点上服务的ipvs条目。
在重启kube-proxy之前：

-A -t 10.230.65.211:9494 -s rr
-a -t 10.230.65.211:9494 -r 10.230.158.10:9494 -m -w 1
-a -t 10.230.65.211:9494 -r 10.230.158.57:9494 -m -w 1
-a -t 10.230.65.211:9494 -r 10.230.213.245:9494 -m -w 1
-a -t 10.230.65.211:9494 -r 10.230.216.170:9494 -m -w 1
-a -t 10.230.65.211:9494 -r 10.230.216.180:9494 -m -w 0
-a -t 10.230.65.211:9494 -r 10.230.222.151:9494 -m -w 1
-a -t 10.230.65.211:9494 -r 10.230.222.158:9494 -m -w 0
-a -t 10.230.65.211:9494 -r 10.230.222.160:9494 -m -w 1
-a -t 10.230.65.211:9494 -r 10.230.226.82:9494 -m -w 1
-a -t 10.230.65.211:9494 -r 10.230.226.85:9494 -m -w 1
-a -t 10.230.65.211:9494 -r 10.230.226.117:9494 -m -w 0
-a -t 10.230.65.211:9494 -r 10.230.242.229:9494 -m -w 1

重新启动kube-proxy之后：

-A -t 10.230.65.211:9494 -s rr
-a -t 10.230.65.211:9494 -r 10.230.158.36:9494 -m -w 1
-a -t 10.230.65.211:9494 -r 10.230.213.236:9494 -m -w 1
-a -t 10.230.65.211:9494 -r 10.230.216.129:9494 -m -w 1
-a -t 10.230.65.211:9494 -r 10.230.216.163:9494 -m -w 1
-a -t 10.230.65.211:9494 -r 10.230.226.84:9494 -m -w 1
-a -t 10.230.65.211:9494 -r 10.230.226.124:9494 -m -w 1

@lbernail @bjornryden这就是我在1.11.5上切换到IPVS时进入该线程的原因。 第二天，我发现服务不可访问，因此我认为kube-proxy已挂起，因此不会更新服务条目。
只是为了让您知道，v1.11.5的kube-proxy存在类似的问题（ipvs表中的IP错误）。

@bjornryden删除错误的好消息（这是补丁所解决的问题）

后端未正确更新的事实很奇怪，因为正常终止不会影响新的终结点（例如，如果仍然有一些过时的终结点，我会感到惊讶）

您能否共享与服务10.230.65.211:9494相关的kube-proxy日志？

另外，此服务/部署是否有任何特殊行为？ （例如，广告连播是否经常更新或在HPA之后更新？）

所以，它已经由IPVS模式@lbernail在我与1.11.6-beta.1图像实验室集群中运行2天。
到目前为止，它还没有挂起并且一直在更新ipvs条目。

但是有一个奇怪的发现：

lvdkbm501 ~ # ipvsadm -L -n -t 10.158.9.61:443
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  10.158.9.61:443 rr
  -> 10.158.128.28:8443           Masq    1      0          0         
  -> 10.158.130.24:8443           Masq    1      0          0         
lvdkbm501 ~ # ipvsadm -L -n -t 10.158.9.61:443
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  10.158.9.61:443 rr
  -> 10.158.128.28:8443           Masq    1      0          0         
lvdkbm501 ~ # ipvsadm -L -n -t 10.158.9.61:443
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  10.158.9.61:443 rr
  -> 10.158.128.28:8443           Masq    1      0          0         
lvdkbm501 ~ # ipvsadm -L -n -t 10.158.9.61:443
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  10.158.9.61:443 rr
  -> 10.158.128.28:8443           Masq    1      0          0         
  -> 10.158.128.28:8443           Masq    1      0          0         
lvdkbm501 ~ # ipvsadm -L -n -t 10.158.9.61:443
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  10.158.9.61:443 rr
  -> 10.158.128.28:8443           Masq    1      0          0         
  -> 10.158.129.28:8443           Masq    1      0          0         

在删除该服务的一个pod之前，对ipvsadm的首次调用显示了初始状态。 几乎没有下一个节目表明某个条目已消失，因为该窗格已被删除。 但是，我不知道它是否应该如此，但是它创建了第二个条目，该条目的IP地址与剩余的Pod相同。 然后将其更新为新的Pod的正确IP，以替换已删除的Pod。

除此之外，还可以。 但是实验室集群并不像其他集群那么忙。 @bjornryden的评论让我有点担心在繁忙的集群中尝试该图像，这可能会影响我们的客户。

@emptywee我假设删除的pod是10.158.130.24 。 我不明白这一点：

TCP  10.158.9.61:443 rr
  -> 10.158.128.28:8443           Masq    1      0          0         
  -> 10.158.128.28:8443           Masq    1      0          0 

两次拥有相同的真实服务器没有意义。 你能复制吗？

删除Pod的正常行为应为：

• 如果没有与此Pod的连接，请删除RS
• 如果有连接，请将权重设置为0（已建立的连接会继续，没有新的连接）
• 每分钟检查所有权重为0（这是正常终止列表的RS）的RS，并删除不再具有连接的RS

@lbernail正确。 删除的Pod具有该IP地址。 是的，这对我来说也很奇怪。 如果有机会，我会尝试重现。 如果我能够重现相同的行为，您是否希望我增加日志详细程度？ 如果是，到什么水平？

@emptywee ：非常感谢， --v=5应该给我们很多信息
最主要的是，我什至不认为RS可以在IPVS的VS中出现两次。 您正在运行哪个内核？

@lbernail会将其设置为--v = 5并尝试重现，可能是在周末的某个时间。

我在CoreOS上稳定：

4.14是最近的版本，所以如果有IPVS问题，我会感到惊讶
我将尝试通过修补版本强调群集，以查看是否可以从@bjornryden重现该问题。

@lbernail不幸的是，我无法重现仅见过一次的ipvs条目复制行为。 尝试了多次，使用了不同的服务。 但是，我还没有看到任何权重为0的条目，但这可以通过以下事实来解释：我删除的Pod会很快退出。 另外，kube-proxy运行5天以上后仍具有响应能力。 我可以在这里更忙的集群中再试一次。

@emptywee这是“好消息”（即使我希望我们能够复制并理解它，也许这是ipvsadm中的显示故障？）

如果它们是一些ActiveConn或InActiveConn，则权重仅设置为0。 要查看它，只需在删除关联的Pod之前打开与服务的长期连接（例如使用telnet）

5天稳定是个好消息！ 让我们知道当/如果您尝试繁忙的群集时的情况

在此过程中，我有一些小改进：

• https://github.com/kubernetes/kubernetes/pull/71834 ：使用sessionAffinity使服务正常终止（已经在master中，为1.13、1.12和1.11创建了樱桃精选）
• https://github.com/kubernetes/kubernetes/pull/71895和https://github.com/kubernetes/kubernetes/pull/72106 ：删除服务时支持正常终止（第一个已经在master中，我是等待第二个分支在所有分支中都摘樱桃）

此外， 1.11.6 ， 1.12.4和1.13.1已发布，并在删除功能中包含此修复程序

我在周末/写了一篇关于这个问题的简短

@lbernail当然可以！ 您的hyperkube映像是否与1.11.6版本相同？ 我仅在更繁忙的集群中安装了用于kube-proxy的映像，然后再次切换到ipvs进行测试。 您要我使用官方的吗？

@emptywee我的映像和1.11.6基本相同：我在正式发行的几天前从release-1.11分支的头构建了它，因此区别最多是几次提交，没有涉及IPVS。 因此，我认为不值得再次更新集群

坏消息:(刚让我的kube-proxy无法更新ipvs后端。切换回iptables并收集日志。让kube-proxy运行时带有--v=5标志。

@lbernail因此，不确定是否需要多少日志，但是我们基本上遇到了一些服务的问题，这些服务的ipvs后端IP没有更新。 但是，日志告诉我们它正在正确更新端点：

$ cat kube-proxy-issue.log | egrep -e 'c-qa4/ycsdtleuk|10.148.191.69|10.148.192.14|10.148.183.70|10.148.184.67'
I1217 19:12:52.849863       1 graceful_termination.go:160] Trying to delete rs: 10.148.6.202:80/TCP/10.148.192.14:8080
I1217 19:12:52.849948       1 graceful_termination.go:173] Deleting rs: 10.148.6.202:80/TCP/10.148.192.14:8080
I1217 19:12:52.850000       1 graceful_termination.go:160] Trying to delete rs: 10.148.6.202:80/TCP/10.148.191.69:8080
I1217 19:12:52.850069       1 graceful_termination.go:173] Deleting rs: 10.148.6.202:80/TCP/10.148.191.69:8080
I1217 19:12:53.425438       1 endpoints.go:234] Setting endpoints for "c-qa4/ycsdtleuk:http" to [10.148.191.69:8080 10.148.192.14:8080]
I1217 19:13:00.501203       1 ipset.go:140] Successfully delete legacy ip set entry: 10.148.191.69,tcp:8080,10.148.191.69 from ip set: KUBE-LOOP-BACK
I1217 19:13:00.543691       1 ipset.go:140] Successfully delete legacy ip set entry: 10.148.192.14,tcp:8080,10.148.192.14 from ip set: KUBE-LOOP-BACK
I1217 19:13:06.916777       1 service.go:309] Adding new service port "c-qa4/ycsdtleuk:http" at 10.148.6.202:80/TCP
I1217 19:13:08.345142       1 proxier.go:1465] Adding new service "c-qa4/ycsdtleuk:http" 10.148.6.202:80/TCP
I1217 19:13:13.939763       1 ipset.go:148] Successfully add entry: 10.148.191.69,tcp:8080,10.148.191.69 to ip set: KUBE-LOOP-BACK
I1217 19:13:14.002663       1 ipset.go:148] Successfully add entry: 10.148.192.14,tcp:8080,10.148.192.14 to ip set: KUBE-LOOP-BACK
I1217 19:27:53.485483       1 endpoints.go:234] Setting endpoints for "c-qa4/ycsdtleuk:http" to [10.148.191.69:8080 10.148.192.14:8080]
I1217 19:27:53.485526       1 endpoints.go:234] Setting endpoints for "c-qa4/ycsdtleuk:http" to [10.148.191.69:8080 10.148.192.14:8080]
I1217 19:42:53.388638       1 endpoints.go:234] Setting endpoints for "c-qa4/ycsdtleuk:http" to [10.148.191.69:8080 10.148.192.14:8080]
I1217 19:42:53.388673       1 endpoints.go:234] Setting endpoints for "c-qa4/ycsdtleuk:http" to [10.148.191.69:8080 10.148.192.14:8080]
I1217 19:57:53.643829       1 endpoints.go:234] Setting endpoints for "c-qa4/ycsdtleuk:http" to [10.148.191.69:8080 10.148.192.14:8080]
I1217 19:57:53.644032       1 endpoints.go:234] Setting endpoints for "c-qa4/ycsdtleuk:http" to [10.148.191.69:8080 10.148.192.14:8080]
I1217 20:12:53.541080       1 endpoints.go:234] Setting endpoints for "c-qa4/ycsdtleuk:http" to [10.148.191.69:8080 10.148.192.14:8080]
I1217 20:12:53.541129       1 endpoints.go:234] Setting endpoints for "c-qa4/ycsdtleuk:http" to [10.148.191.69:8080 10.148.192.14:8080]
I1217 20:27:53.368129       1 endpoints.go:234] Setting endpoints for "c-qa4/ycsdtleuk:http" to [10.148.191.69:8080 10.148.192.14:8080]
I1217 20:27:53.368306       1 endpoints.go:234] Setting endpoints for "c-qa4/ycsdtleuk:http" to [10.148.191.69:8080 10.148.192.14:8080]
I1217 20:42:53.539517       1 endpoints.go:234] Setting endpoints for "c-qa4/ycsdtleuk:http" to [10.148.191.69:8080 10.148.192.14:8080]
I1217 20:42:53.539542       1 endpoints.go:234] Setting endpoints for "c-qa4/ycsdtleuk:http" to [10.148.191.69:8080 10.148.192.14:8080]
I1217 20:57:53.624320       1 endpoints.go:234] Setting endpoints for "c-qa4/ycsdtleuk:http" to [10.148.191.69:8080 10.148.192.14:8080]
I1217 20:57:53.624371       1 endpoints.go:234] Setting endpoints for "c-qa4/ycsdtleuk:http" to [10.148.191.69:8080 10.148.192.14:8080]
I1217 21:12:53.630561       1 endpoints.go:234] Setting endpoints for "c-qa4/ycsdtleuk:http" to [10.148.191.69:8080 10.148.192.14:8080]
I1217 21:12:53.630585       1 endpoints.go:234] Setting endpoints for "c-qa4/ycsdtleuk:http" to [10.148.191.69:8080 10.148.192.14:8080]
I1217 21:26:20.897397       1 endpoints.go:234] Setting endpoints for "c-qa4/ycsdtleuk:http" to [10.148.191.69:8080 10.148.192.14:8080]
I1217 21:26:20.897463       1 endpoints.go:234] Setting endpoints for "c-qa4/ycsdtleuk:http" to [10.148.191.69:8080 10.148.192.14:8080]
I1217 21:26:46.630013       1 endpoints.go:234] Setting endpoints for "c-qa4/ycsdtleuk:http" to [10.148.191.69:8080 10.148.192.14:8080]
I1217 21:26:46.630084       1 endpoints.go:234] Setting endpoints for "c-qa4/ycsdtleuk:http" to [10.148.191.69:8080 10.148.192.14:8080]
I1217 21:26:57.995544       1 endpoints.go:234] Setting endpoints for "c-qa4/ycsdtleuk:http" to [10.148.191.69:8080 10.148.192.14:8080]
I1217 21:26:57.995657       1 endpoints.go:234] Setting endpoints for "c-qa4/ycsdtleuk:http" to [10.148.183.68:8080 10.148.191.69:8080 10.148.192.14:8080]
I1217 21:27:24.296606       1 endpoints.go:234] Setting endpoints for "c-qa4/ycsdtleuk:http" to [10.148.183.68:8080 10.148.187.41:8080 10.148.191.69:8080 10.148.192.14:8080]
I1217 21:27:53.678172       1 endpoints.go:234] Setting endpoints for "c-qa4/ycsdtleuk:http" to [10.148.183.68:8080 10.148.187.41:8080 10.148.191.69:8080 10.148.192.14:8080]
I1217 21:28:21.517538       1 endpoints.go:234] Setting endpoints for "c-qa4/ycsdtleuk:http" to [10.148.183.68:8080 10.148.187.41:8080 10.148.191.69:8080]
I1217 21:28:21.595632       1 endpoints.go:234] Setting endpoints for "c-qa4/ycsdtleuk:http" to [10.148.183.68:8080 10.148.187.41:8080]
I1217 21:42:53.491595       1 endpoints.go:234] Setting endpoints for "c-qa4/ycsdtleuk:http" to [10.148.183.68:8080 10.148.187.41:8080]
I1217 21:57:53.492214       1 endpoints.go:234] Setting endpoints for "c-qa4/ycsdtleuk:http" to [10.148.183.68:8080 10.148.187.41:8080]
I1217 22:05:45.760620       1 endpoints.go:234] Setting endpoints for "c-qa4/ycsdtleuk:http" to [10.148.183.68:8080]
I1217 22:06:24.890429       1 endpoints.go:234] Setting endpoints for "c-qa4/ycsdtleuk:http" to [10.148.183.68:8080]
I1217 22:07:15.463144       1 endpoints.go:234] Setting endpoints for "c-qa4/ycsdtleuk:http" to [10.148.134.231:8080 10.148.183.68:8080]
I1217 22:07:43.201415       1 endpoints.go:234] Setting endpoints for "c-qa4/ycsdtleuk:http" to [10.148.134.231:8080]
I1217 22:08:18.268005       1 endpoints.go:234] Setting endpoints for "c-qa4/ycsdtleuk:http" to [10.148.134.231:8080]
I1217 22:08:58.774066       1 endpoints.go:234] Setting endpoints for "c-qa4/ycsdtleuk:http" to [10.148.134.231:8080 10.148.192.70:8080]
I1217 22:12:53.528515       1 endpoints.go:234] Setting endpoints for "c-qa4/ycsdtleuk:http" to [10.148.134.231:8080 10.148.192.70:8080]
I1217 22:27:53.522197       1 endpoints.go:234] Setting endpoints for "c-qa4/ycsdtleuk:http" to [10.148.134.231:8080 10.148.192.70:8080]
I1217 22:29:20.396794       1 endpoints.go:234] Setting endpoints for "c-qa4/ycsdtleuk:http" to [10.148.134.231:8080 10.148.192.70:8080]
I1217 22:29:43.994895       1 endpoints.go:234] Setting endpoints for "c-qa4/ycsdtleuk:http" to [10.148.134.231:8080 10.148.192.70:8080]
I1217 22:29:54.382253       1 endpoints.go:234] Setting endpoints for "c-qa4/ycsdtleuk:http" to [10.148.134.231:8080 10.148.183.70:8080 10.148.192.70:8080]
I1217 22:30:17.895084       1 endpoints.go:234] Setting endpoints for "c-qa4/ycsdtleuk:http" to [10.148.134.231:8080 10.148.183.70:8080 10.148.184.67:8080 10.148.192.70:8080]
I1217 22:31:45.791871       1 endpoints.go:234] Setting endpoints for "c-qa4/ycsdtleuk:http" to [10.148.134.231:8080 10.148.183.70:8080 10.148.184.67:8080]
I1217 22:31:45.855011       1 endpoints.go:234] Setting endpoints for "c-qa4/ycsdtleuk:http" to [10.148.183.70:8080 10.148.184.67:8080]
I1217 22:42:53.652587       1 endpoints.go:234] Setting endpoints for "c-qa4/ycsdtleuk:http" to [10.148.183.70:8080 10.148.184.67:8080]
I1217 22:57:53.509372       1 endpoints.go:234] Setting endpoints for "c-qa4/ycsdtleuk:http" to [10.148.183.70:8080 10.148.184.67:8080]

但是当我用ipvsadm检查它时，它仍然显示前两个：10.148.191.69:8080 10.148.192.14:8080

kubectl -n c-qa4 describe svc ycsdtleuk表示端点为10.148.183.70:8080 10.148.184.67:8080 ，而在其他框中，相同服务的ipvs条目已正确更新。

$ kubectl -n c-qa4 describe svc ycsdtleuk
Name:              ycsdtleuk
Namespace:         c-qa4
Labels:            country=uk
                   env=qa
                   run=ycsdtleuk
                   stack=c
                   track=4
Annotations:       run: ycsdtleuk
Selector:          load-balancer-ycsdtleuk=true
Type:              ClusterIP
IP:                10.148.6.202
Port:              http  80/TCP
TargetPort:        8080/TCP
Endpoints:         10.148.183.70:8080,10.148.184.67:8080
Session Affinity:  None
Events:            <none>

我基本上是同时在两个盒子上查看以下内容：

(incorrect ipvs config)
rnqkbm401 ~ # ipvsadm -L -t 10.148.6.202:80
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  rnqkbm401:http rr
  -> 10.148.191.69:http-alt       Masq    1      0          0
  -> 10.148.192.14:http-alt       Masq    1      0          0

(correct ipvs config)
rnqkbw401 ~ # ipvsadm -L -t 10.148.6.202:80
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  rnqkbw401:http rr
  -> 10.148.183.70:http-alt       Masq    1      0          0
  -> 10.148.184.67:http-alt       Masq    1      0          0

没有明显的错误消息或任何尖叫声指示任何地方的问题迹象。 如果您愿意，我可以将完整的日志文件（未压缩的18M）直接发送给您。

长话短说：侦听服务更改的部分工作正常，但与ipvs模块交互的部分感觉不好。 我什至没有看到它一旦添加两个IP地址就试图删除它们。 某些地方卡住了。

@emptywee谢谢您的详细报告。 这些日志没什么奇怪的。

令人惊讶的是， syncProxyRules至少每30秒运行一次（默认值），并且应该使IPVS真实服务器与端点保持一致。 它调用syncEndpoints触发AddRealServer和ipvsHandle.NewDestination

所有这一切都发生在删除旧端点之前，因此不应受到正常终止的影响

您能否将完整的日志发送给我，以便我能确切地看到21:26和21:29什么？

另外两个问题：

• 一个kube-proxy受到了影响，对吧？
• 一项服务受到了影响（其他服务是否继续正常更新？）

下次，尝试将SIGABRT发送到该进程，以便它打印任务状态。

当我遇到问题时，它总是在libnetwork/ipvs/netlink.go内部执行某项操作，这使我怀疑netlink系统调用由于某种原因而卡住了。

对不起，我很抱歉，但是周末之后我们不得不专注于恢复到iptables进行设置。

回顾请求的服务日志@lbernail ，我们有很多这样的日志：

I1214   11:48:44.889081       1   graceful_termination.go:160] Trying to delete rs:   172.17.0.1:31663/TCP/10.230.213.204:9300
I1214 11:48:44.889099       1 graceful_termination.go:173]   Deleting rs: 172.17.0.1:31663/TCP/10.230.213.204:9300
I1214 11:48:44.889113       1 graceful_termination.go:160] Trying   to delete rs: 172.17.0.1:31663/TCP/10.230.226.83:9300
I1214 11:48:44.889130       1 graceful_termination.go:173]   Deleting rs: 172.17.0.1:31663/TCP/10.230.226.83:9300
...

我们还看到了这些：

I1214 12:15:44.920364       1 graceful_termination.go:160] Trying to delete rs: 10.230.65.211:9494/TCP/10.230.226.124:9494
I1214 12:15:44.920489       1 graceful_termination.go:173] Deleting rs: 10.230.65.211:9494/TCP/10.230.226.124:9494
I1214 12:15:44.920512       1 graceful_termination.go:93] lw: remote out of the list: 10.230.65.211:9494/TCP/10.230.226.124:9494

正如其他人所报告的那样，这仅在某些节点上发生。 当我发现问题时，我们已经为该特定服务破坏了3个群集节点。

@lbernail绝对。 这是请求的日志片段，我也包括了启动片段： https :

让我知道您是否需要整个文件。 如果是，那么您希望我将其发送到或上传到的位置。

要回答您的问题：

1. 该集群有60多个节点，我检查了其他几个节点，ipvs条目是正确的。 虽然不确定所有条目。 但是在主节点上，通过我们将所有流量路由到服务，至少有两个服务没有正确更新。 可能还有更多，但是一旦我们开始收到客户的投诉，我们立即得知无法访问两项服务。 我收集了日志并重新启动了kube-proxy以最大程度地减少影响。 作为故障排除的一部分，我比较了该主节点和一个工作节点上ipvs条目的状态，并意识到这是不同的。 其他节点可能有其他条目可能没有正确更新，但由于流量没有通过它们，因此我们没有注意到任何影响。

2. 不，至少有两项服务受到了影响。 可能更多，但我没有检查。

日志中有两个有趣的事情：

• I1217 21:26:57.995657 1 endpoints.go:234] Setting endpoints for "c-qa4/ycsdtleuk:http" to [10.148.183.68:8080 10.148.191.69:8080 10.148.192.14:8080]从未由IPVS处理，但没有错误，因为您保留了两个旧的RS（我们无法确定，添加RS不会生成日志）
• I1217 21:28:21.517538 1 endpoints.go:234] Setting endpoints for "c-qa4/ycsdtleuk:http" to [10.148.183.68:8080 10.148.187.41:8080 10.148.191.69:8080]不会触发10.148.192.14:8080的删除调用（我们应该在那里保存日志）

@emptywee是IPVS在正常终止之前为您工作的，还是您正在使用iptables？

@ m1093782566 / @ Lion-Wei你知道这里会发生什么吗？

@lbernail我们一直等到升级到1.11才能切换到ipvs（我相信这是GA发布该功能的时间），我们直接升级到1.11.5，这是我们首次尝试使用ipvs。

@emptywee ：我将添加更多信息级别的日志以尝试找出问题所在。 您可以运行几个小时的详细版本吗？ （我完全理解您是否愿意，但我还无法在我这一边进行繁殖）

自5月以来我们一直在使用IPVS，在1.12.1和1.11.1中没有问题。

升级到1.12.3之后，我们还看到这种情况发生在多个节点上，但到目前为止还不是全部。 受影响的节点在升级后几个小时就停止更新IPVS，但是我无法辨别出协调一致的方式。

需要注意的一件事是，那时kube-proxy也完全停止了日志记录，因此看起来好像完全卡住了。

日志文件显示卡住了，并在SIGQUIT之后进行了goroutine转储（略作编辑）

@matthiasr的日志文件再次显示goroutine卡住了从netlink套接字读取的内容。 跟我一样。

您可能会说存在一些逻辑错误，导致与netlink套接字的交互过多。 但是，如果与netlink的通信不稳定，则ipvs最终会卡住。 如果逻辑错误得到修复，则可能需要更长的时间才能达到此状态，但这并不意味着如果有足够的时间就不可能发生。

这就是为什么我建议在netlink交互中设置某种超时的原因。 如果操作超时，请关闭netlink套接字并自动打开一个新的套接字，作为恢复机制。

@matthiasr好的，这是个好消息（我们也已经使用IPVS超过6个月了，没有任何问题，但是尚未在我们最繁忙的集群上使用正常终止来更新到版本）

看起来我们可能在某个地方陷入僵局

@lbernail虽然将整个群集切换到ipvs有点麻烦，但是如果这将有助于解决此问题，那么我很失望。 告诉我要使用的图像名称，我会尽力帮助您。

我们已在1.11.6上恢复到IPVS，以查看是否可以重现问题。 日志设置为5，并在接下来的几天在集群中运行某些活动。 我明天不上班，但是我将在线弹出查看是否有应增加日志记录的图像。

@matthiasr ： 1.12.3肯定有问题（请参阅本主题的其余部分）。 它的一部分在1.12.4 。 您将在日志中得到更少的错误。 所有这些都应该消失： Failed to add destination: 10.zz.yy.xx, error: file exists

我们仍然有一个问题，即kube-proxy似乎挂起了，不再更新真实服务器。
我调查了一下：

• 从@emptywee看来，问题开始后，我们从proxier.go，ipset.go或graceful_termination.go都看不到任何日志，这似乎表明我们被卡在某个地方。 @emptywee您可以检查21:30之后是否有这些文件的日志吗？
• 在go例程转储中，netlink套接字上有两个syscall接收。 所以我们在那里可能有问题。 此外，为了支持正常终止，还对libnetwork / ipvs进行了升级，因此行为可能已更改。 但是，libnetwork代码似乎在netlink套接字上设置了send / rcv超时（如果卡住了，我们应该看到它的分钟数）： https :

有趣的是，只有新版本的libnetwork才具有netlink超时：

libnetwork已在所有发行分支（1.11、1.12和1.13）中升级，以允许正常终止。 我们从： ba46b928444931e6865d8618dc03622cac79aa6f到a9cd636e37898226332c439363e2ed0ea185ae92

在旧版本中，代码为： https :

这两个提交之间的libnetwork中IPVS代码的唯一区别是：

• 超时的介绍
• 将ActiveConnections和InactiveConnections到Destination （正常终止所需）

我想知道这是否可能相关。

在对该问题进行了更多思考之后，我认为这是由于我们现在有两个使用同一netlink套接字的goroutine（因为gracefulTerminationManager在其自己的goroutine中运行并与代理共享utilipvs.Interface）。

我们可能会在pkg/util/ipvs/ipvs_linux.go所有调用中添加一个互斥锁，以确保单个goroutine在任何给定时间都在使用netlink套接字。

@ m1093782566您怎么看？

@lbernail不，在21:30之后的任何地方都没有这些文件中的条目。
这是各种文件的最后一个条目：

来自graceful_termination.go的最新条目

$ cat kube-proxy-issue.log | grep 'graceful_termination.go' | tail
I1217 21:18:20.757262       1 graceful_termination.go:66] Adding rs 10.148.11.105:8080/TCP/10.148.174.35:8081 to graceful delete rsList
I1217 21:18:52.267340       1 graceful_termination.go:160] Trying to delete rs: 10.148.0.145:80/TCP/10.148.188.28:8080
I1217 21:18:52.267456       1 graceful_termination.go:173] Deleting rs: 10.148.0.145:80/TCP/10.148.188.28:8080
I1217 21:18:52.267638       1 graceful_termination.go:160] Trying to delete rs: 10.148.0.145:80/TCP/10.148.145.194:8080
I1217 21:18:52.267755       1 graceful_termination.go:173] Deleting rs: 10.148.0.145:80/TCP/10.148.145.194:8080
I1217 21:18:52.405770       1 graceful_termination.go:160] Trying to delete rs: 10.148.0.145:80/TCP/10.148.188.28:8081
I1217 21:18:52.405906       1 graceful_termination.go:173] Deleting rs: 10.148.0.145:80/TCP/10.148.188.28:8081
I1217 21:18:52.406052       1 graceful_termination.go:160] Trying to delete rs: 10.148.0.145:80/TCP/10.148.145.194:8081
I1217 21:18:52.406156       1 graceful_termination.go:173] Deleting rs: 10.148.0.145:80/TCP/10.148.145.194:8081
I1217 21:18:53.240664       1 graceful_termination.go:160] Trying to delete rs: 10.148.11.105:8080/TCP/10.148.174.35:8081

来自proxier.go的最新条目：

$ cat kube-proxy-issue.log | grep 'proxier.go' | tail
I1217 21:18:52.570282       1 proxier.go:1023] Port "nodePort for kube-system/kube-dns:dns-tcp" (:30054/tcp) was open before and is still needed
I1217 21:18:52.663609       1 proxier.go:1023] Port "nodePort for ingress-nginx/ingress-nginx:http" (:31513/tcp) was open before and is still needed
I1217 21:18:53.241144       1 proxier.go:1465] Adding new service "c-qa4/yardchange-ws:http" 10.148.1.136:80/TCP
E1217 21:18:53.241277       1 proxier.go:1467] Failed to add IPVS service "c-qa4/yardchange-ws:http": file exists
E1217 21:18:53.241350       1 proxier.go:821] Failed to sync service: 10.148.1.136:80/TCP, err: file exists
I1217 21:18:53.241624       1 proxier.go:1465] Adding new service "c-qa4/receiving-ws-v2-uk:http-admin" 10.148.13.141:8080/TCP
E1217 21:18:53.241723       1 proxier.go:1467] Failed to add IPVS service "c-qa4/receiving-ws-v2-uk:http-admin": file exists
E1217 21:18:53.241761       1 proxier.go:821] Failed to sync service: 10.148.13.141:8080/TCP, err: file exists
E1217 21:18:53.242506       1 proxier.go:1496] Failed to get IPVS service, error: Expected only one service obtained=0
E1217 21:18:53.242558       1 proxier.go:818] Failed to sync endpoint for service: 10.148.11.84:8080/TCP, err: Expected only one service obtained=0

来自ipset.go的最新条目：

$ cat kube-proxy-issue.log | grep 'ipset.go' | tail
I1217 21:17:14.768694       1 ipset.go:148] Successfully add entry: 10.148.134.229,tcp:8080,10.148.134.229 to ip set: KUBE-LOOP-BACK
I1217 21:17:14.774040       1 ipset.go:148] Successfully add entry: 10.148.134.229,tcp:8081,10.148.134.229 to ip set: KUBE-LOOP-BACK
I1217 21:17:26.643709       1 ipset.go:148] Successfully add entry: 10.148.172.59,tcp:8080,10.148.172.59 to ip set: KUBE-LOOP-BACK
I1217 21:17:26.648504       1 ipset.go:148] Successfully add entry: 10.148.172.59,tcp:8081,10.148.172.59 to ip set: KUBE-LOOP-BACK
I1217 21:17:31.049667       1 ipset.go:148] Successfully add entry: 10.148.136.14,tcp:8080,10.148.136.14 to ip set: KUBE-LOOP-BACK
I1217 21:17:31.053901       1 ipset.go:148] Successfully add entry: 10.148.136.14,tcp:8081,10.148.136.14 to ip set: KUBE-LOOP-BACK
I1217 21:18:18.920550       1 ipset.go:140] Successfully delete legacy ip set entry: 10.148.181.11,tcp:8080,10.148.181.11 from ip set: KUBE-LOOP-BACK
I1217 21:18:18.924752       1 ipset.go:140] Successfully delete legacy ip set entry: 10.148.181.11,tcp:8081,10.148.181.11 from ip set: KUBE-LOOP-BACK
I1217 21:18:21.232725       1 ipset.go:140] Successfully delete legacy ip set entry: 10.148.174.35,tcp:8080,10.148.174.35 from ip set: KUBE-LOOP-BACK
I1217 21:18:21.237619       1 ipset.go:140] Successfully delete legacy ip set entry: 10.148.174.35,tcp:8081,10.148.174.35 from ip set: KUBE-LOOP-BACK

他们都在21:18左右结束伐木...这有什么线索吗？

@emptywee是的，它证实了以下假设：我们在共享netlink套接字的两个goroutine之间存在死锁。 他们都被困在等待接收消息的过程中，并继续重试而没有成功（由于3s接收超时）。

我将尝试验证该假设，并会尽快开始进行修复。

我将1.12.4放在测试集群上，然后开始重新部署一个应用程序，该应用程序在1.5小时内重现了该问题，因此我认为我将能够相当可靠地测试潜在的修复程序。

再次在netlink Receive上悬挂了2个goroutine（1和107）。

我可能对该错误进行了修复，可在此处找到代码： https :
这个想法是用Mutex保护所有的netlink调用以避免死锁。 我不确定这是否是最佳解决方案，但可以解决此问题（如果可行，请确认我们已确定该错误）。

如果您想要/可以测试它，我用修复程序创建了两个图像：

• lbernail / hyperkube：v1.11.7-beta.1
• lbernail / hyperkube：v1.12.5-beta.1

如果您进行测试，请让我们知道它是否有效，请分享有趣的日志并查看goroutine，以查看是否有些东西滞留在netlink调用中。

（我做了一些测试，但仅限于一个小型集群）

我可能会在12月24日星期一尝试一下！

今天就去试试。 将报告结果（如果有）。

这是一个错误。 我已经切换到iptables模式。

我还升级到了Beta版本，并很快又开始了负载测试。 我们通常会很快（不到一天）休息，并且持续地……手指交叉。

@ li-sen绝对是一个错误。 我们正在做这件事

@emptywee和@bjornryden非常感谢您的测试！

@lbernail，我的荣幸！ 因此，距离我使用1.11.7-beta.1图像发布kube-proxy大约两天了。 它还没有挂在我们身上。 我可以看到proxier.go和其他goroutine仍在运行并打印到日志中。 我会给它更多的时间，因为最近几天确实很慢而且很安静，没有很多部署在进行中。 无论哪种方式，我都觉得@lbernail添加的互斥锁会

我将在本周末更新此问题/线程。

@lbernail

您能提出解决此问题的公关吗？

/重新打开
（在一切正常的情况下，密切注意来自测试的反馈，并在创建樱桃点之前）

@lbernail ：除非您创建了一个问题/ PR，否则您无法重新打开它。

@ m1093782566您可以在我们完全确定已解决问题之前重新打开该问题吗？

我有Kube-proxy ipvs版本1.12.3的相同问题。
真正的服务器是错误的，并且Kube-proxy无法刷新i​​pvs规则。

这是最近的日志，它卡在这里，不再打印新日志。

27 23:31:24.113927       1 graceful_termination.go:93] lw: remote out of the list: 10.96.0.1:443/TCP/10.6.40.32:6443
E1227 23:31:24.113935       1 graceful_termination.go:183] Try flush graceful termination list err
I1227 23:32:24.114182       1 graceful_termination.go:160] Trying to delete rs: 10.96.0.1:443/TCP/10.6.40.32:6443
E1227 23:32:24.114316       1 graceful_termination.go:89] Try delete rs "10.96.0.1:443/TCP/10.6.40.32:6443" err: Failed to delete rs "10.96.0.1:443/TCP/10.6.40.32:6443", can't find the real server
I1227 23:32:24.114332       1 graceful_termination.go:93] lw: remote out of the list: 10.96.0.1:443/TCP/10.6.40.32:6443
I1227 23:32:24.114348       1 graceful_termination.go:160] Trying to delete rs: 10.96.0.1:443/TCP/10.6.40.33:6443
I1227 23:32:24.114397       1 graceful_termination.go:160] Trying to delete rs: 10.6.40.25:32444/TCP/10.244.20.41:32444
E1227 23:32:24.114446       1 graceful_termination.go:183] Try flush graceful termination list err
I1227 23:33:24.114853       1 graceful_termination.go:160] Trying to delete rs: 10.96.0.1:443/TCP/10.6.40.32:6443
E1227 23:33:24.115043       1 graceful_termination.go:89] Try delete rs "10.96.0.1:443/TCP/10.6.40.32:6443" err: Failed to delete rs "10.96.0.1:443/TCP/10.6.40.32:6443", can't find the real server
I1227 23:33:24.115106       1 graceful_termination.go:93] lw: remote out of the list: 10.96.0.1:443/TCP/10.6.40.32:6443
I1227 23:33:24.115128       1 graceful_termination.go:160] Trying to delete rs: 10.96.0.1:443/TCP/10.6.40.33:6443

@VincentFF ：存在两个问题：

• 删除失败：已在1.12.4修复
• 由于死锁（这是您要描述的问题），kube-proxy卡住了：希望从昨天起可以通过PR进行修复。 我们正在等待当前正在测试补丁的@emptywee和@bjornryden的反馈。 一旦我们确认该问题不再存在，我将为1.13、1.12和1.11创建Cherrypicks（如果该补丁有效，则可能会在下周初合并到所有分支中，并将成为该补丁的一部分）下一个补丁发布）

我还重新运行了我的实验https://github.com/kubernetes/kubernetes/issues/71071#issuecomment -449402484，修补程序已反向移植到1.12.4。 会回报！

当kube-proxy在IPVS模式下运行时，我们没有任何访问服务的问题，并且对服务进行了很多部署和更改。 到目前为止，我想说该补丁对我们来说效果很好。

@lbernail修复它很出色！ 非常感激！

好吧，让我们等到星期一，以便我们也能收到@matthiasr和@bjornryden的反馈（我想避免赶时间，确保这次我们有一个干净的解决方案）

@lbernail好消息，感谢您的回复。

使用https://github.com/DataDog/kubernetes/commit/9121f0122d805f55f85f42d26c294898fe29a92f在1.12.4上12小时以上没有任何问题–不用了，我在1.5小时内就锁定了，所以看起来非常好！

@lbernail请问我另一个问题。 如果下一个补丁是1.12.5版本？
如果我只是将kube-proxy升级到1.12.5，其他单元（apiserver，kubelet，控制器管理器，调度程序...）保持1.12.3，可以吗？ 还是我必须升级所有装置？

@VincentFF希望它将达到1.12.5是。 您可以将kube-proxy 1.12.5与运行1.12.3的其他单元一起运行，没问题

更新：24小时后，仍然没有问题，我现在停止实验并认为这是完全成功。 期待1.12.5！

好的，这是个好消息。 我将为所有分支创建“自动选择”，以便尽快获得正式版本中的修复。
非常感谢您参与帮助解决问题和测试！

樱桃小菜：
1.13 ： https :
1.12 ： https :
1.11 ： https :

我可以加入快乐的测试人员合唱团。 4天无故障。

非常感谢所有参与人员，尤其是@lbernail。 新年快乐！

我构建它并部署到测试环境，超过24小时没有问题。 新年快乐。

您好，我的环境是
Ubuntu 16.04.4 LTS
Kubernetes版本（使用Kubectl版本）：v1.13.0
云提供商或硬件配置：裸机，64位linux
内核4.4.0-75通用

相同的错误：

I0101 22：20：09.742898 267576 graceful_termination.go：160]试图删除rs：10.200.198.70：9090 / TCP / 172.200.101.32：9090
E0101 22：20：09.742975 267576 proxier.go：1519]无法获得IPVS服务，错误：预期仅获得一项服务= 0
E0101 22：20：09.743036 267576 proxier.go：843]无法同步服务端点：10.200.36.192:80/TCP，错误：期望仅获得一项服务= 0

我需要重启kube-proxy修复它

@daigong v1.13.1已修复此问题。 就我而言，一周以上没有任何问题。

@daigong ：完整修补程序实际上将在1.13.2 （这取决于＃72426合并的速度）

@lbernail @FrostyLeaf
谢谢，我使用＃72426测试此错误

现在没事了

PR已合并到所有发行分支中，因此它将存在于：

• 1.11.7
• 1.12.5
• 1.13.2

@annProg

same problem ,both 1.12.2 and 1.13.0-beta1. @m1093782566 @Lion-Wei

你邮箱多少啊啊，，认识下。。少年

一样一样

有机会在v1.11.x中进行新构建吗？

在过去的几天中，我几乎每小时都在刷新发行页面，希望能出现1.12.5，以便我可以在多个群集上修补此错误。 :-)

@verwilst哈哈，同样，迫不及待想要发布v1.11.7以升级当前的产品集群。 v1.12.5也可以在较低的环境中开始升级。

在过去的几天中，我几乎每小时都在刷新发行页面，希望能出现1.12.5，以便我可以在多个群集上修补此错误。 :-)

它在这里:-P

在过去的几天中，我几乎每小时都在刷新发行页面，希望能出现1.12.5，以便我可以在多个群集上修补此错误。 :-)

它在这里:-P

为什么在1.11中不存在？ :(

此错误已在1.11.7中修复吗？ 我没有看到任何发行说明。

是的，它是https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.11.md#v1117 ：

Fix race condition introduced by graceful termination which can lead to a deadlock in kube-proxy (#72361, @lbernail)

是的，我忘了在PR的发行说明中添加“ IPVS”，对此感到抱歉
让我们知道现在是否正常！ （以防万一，我们将问题悬而未决）

自升级至1.12.5以来，我们没有任何问题

好消息！

@ m1093782566，@thockin我认为，我们可以关闭这个问题（在约2个月没有新的报告）

作为参考，错误会以优雅的终止方式影响初始发行版：

• 1.11.5-1.11.6
• 1.12.2-1.12.4
• 1.13.0-1.13.1

在以下情况下存在修复：

• 1.11.7
• 1.12.5
• 1.13.2

/关

@ m1093782566 ：关闭此问题。

我们可以重新营业吗？ 我们在1.12.7上看到了这个问题。

是的，我们也看到了1.12.9中的问题

您确定是同一问题吗？
您可以发布kube-proxy实例的日志吗？

抱歉，这可能是一个错误警报...我们看到的是日志，并且不是相同的错误

I0709 11:59:48.640888       1 graceful_termination.go:160] Trying to delete rs: 192.168.240.51:80/TCP/172.30.71.181:5050
I0709 11:59:48.640304       1 graceful_termination.go:93] lw: remote out of the list: 192.168.240.51:80/TCP/172.30.106.125:5050

由于某些原因，当群集处于负载状态时，它无法路由到某些群集服务IP。 我们仍然不确定这是否是IPVS问题。

如果apiserver超载，端点传播可能会花费一些时间？

这些消息是普通的信息级别消息：
https://github.com/kubernetes/kubernetes/blob/master/pkg/proxy/ipvs/graceful_termination.go#L94

https://github.com/kubernetes/kubernetes/blob/master/pkg/proxy/ipvs/graceful_termination.go#L161

我可以在1.14.4中确认相同的日志。 该日志何时发布？

I0712 01:34:49.925012       1 graceful_termination.go:161] Trying to delete rs: 11.3.0.1:443/TCP/10.0.22.42:443
I0712 01:34:49.925057       1 graceful_termination.go:175] Deleting rs: 11.3.0.1:443/TCP/10.0.22.42:443
I0712 01:35:11.907742       1 graceful_termination.go:161] Trying to delete rs: 11.3.0.10:53/TCP/11.2.5.13:53
I0712 01:35:11.907874       1 graceful_termination.go:175] Deleting rs: 11.3.0.10:53/TCP/11.2.5.13:53
I0712 01:35:11.908006       1 graceful_termination.go:161] Trying to delete rs: 11.3.0.10:53/UDP/11.2.5.13:53
I0712 01:35:11.908059       1 graceful_termination.go:175] Deleting rs: 11.3.0.10:53/UDP/11.2.5.13:53
I0712 01:35:21.992487       1 graceful_termination.go:161] Trying to delete rs: 11.3.0.10:53/TCP/11.2.4.11:53
I0712 01:35:21.992594       1 graceful_termination.go:175] Deleting rs: 11.3.0.10:53/TCP/11.2.4.11:53
I0712 01:35:21.992784       1 graceful_termination.go:161] Trying to delete rs: 11.3.0.10:53/UDP/11.2.4.11:53
I0712 01:35:21.992858       1 graceful_termination.go:175] Deleting rs: 11.3.0.10:53/UDP/11.2.4.11:53
I0712 01:36:29.411592       1 graceful_termination.go:161] Trying to delete rs: 11.3.0.199:44134/TCP/11.2.0.5:44134
I0712 01:36:29.411660       1 graceful_termination.go:175] Deleting rs: 11.3.0.199:44134/TCP/11.2.0.5:44134
I0712 01:36:29.534158       1 graceful_termination.go:161] Trying to delete rs: 11.3.0.17:8085/TCP/11.2.0.3:8085
I0712 01:36:29.534394       1 graceful_termination.go:175] Deleting rs: 11.3.0.17:8085/TCP/11.2.0.3:8085
I0712 01:36:50.745469       1 graceful_termination.go:161] Trying to delete rs: 11.3.0.1:443/TCP/10.0.22.162:443
I0712 01:36:50.745508       1 graceful_termination.go:172] Not deleting, RS 11.3.0.1:443/TCP/10.0.22.162:443: 0 ActiveConn, 12 InactiveConn
I0712 01:37:37.475724       1 graceful_termination.go:161] Trying to delete rs: 11.3.0.1:443/TCP/10.0.22.162:443
I0712 01:37:37.475841       1 graceful_termination.go:172] Not deleting, RS 11.3.0.1:443/TCP/10.0.22.162:443: 0 ActiveConn, 4 InactiveConn
I0712 01:38:37.475956       1 graceful_termination.go:161] Trying to delete rs: 11.3.0.1:443/TCP/10.0.22.162:443
I0712 01:38:37.476201       1 graceful_termination.go:172] Not deleting, RS 11.3.0.1:443/TCP/10.0.22.162:443: 0 ActiveConn, 4 InactiveConn
I0712 01:39:37.476324       1 graceful_termination.go:161] Trying to delete rs: 11.3.0.1:443/TCP/10.0.22.162:443
I0712 01:39:37.476585       1 graceful_termination.go:175] Deleting rs: 11.3.0.1:443/TCP/10.0.22.162:443
I0712 01:39:37.476628       1 graceful_termination.go:94] lw: remote out of the list: 11.3.0.1:443/TCP/10.0.22.162:443
I0712 01:40:36.883345       1 graceful_termination.go:161] Trying to delete rs: 11.3.0.245:5473/TCP/10.0.22.25:5473
I0712 01:40:36.883566       1 graceful_termination.go:175] Deleting rs: 11.3.0.245:5473/TCP/10.0.22.25:5473
I0712 01:40:36.886079       1 graceful_termination.go:161] Trying to delete rs: 11.3.0.17:8085/TCP/11.2.1.3:8085
I0712 01:40:36.886310       1 graceful_termination.go:175] Deleting rs: 11.3.0.17:8085/TCP/11.2.1.3:8085
I0712 01:40:46.927450       1 graceful_termination.go:161] Trying to delete rs: 11.3.0.245:5473/TCP/10.0.22.191:5473
I0712 01:40:46.927525       1 graceful_termination.go:175] Deleting rs: 11.3.0.245:5473/TCP/10.0.22.191:5473
I0712 01:40:56.968884       1 graceful_termination.go:161] Trying to delete rs: 11.3.0.245:5473/TCP/10.0.22.75:5473
I0712 01:40:56.968936       1 graceful_termination.go:175] Deleting rs: 11.3.0.245:5473/TCP/10.0.22.75:5473
I0712 01:41:16.747362       1 graceful_termination.go:161] Trying to delete rs: 11.3.0.1:443/TCP/10.0.23.45:443

这些只是信息日志：

• Trying to delete rs ：已删除的后端Pod，尝试将其删除。 如果已有连接，请使其正常终止，否则删除id
• Deleting rs ：后端没有连接，将其从IPVS中删除

@andrewsykim也许我们现在可以降低这些日志的级别，以便例如仅在v = 5以上显示？

在https://github.com/kubernetes/kubernetes/pull/78395中，我们将日志级别从v = 0提升到v = 2，我认为v = 5有意义。 https://github.com/kubernetes/kubernetes/pull/80100进行更新