It's not /etc/environment, it's not the current bash session that kubeadm is running in, it's not docker or kubelet environment. I verified this by setting no_proxy
to a different value in all these instances. And for some reason after a kubeadm init
it still continues to set another value for no_proxy
. Restart, daemon-reload, restarting the services all doesn't change that fact.
Honestly it's really annoying that it only prints the line "the ip address fo.oo.ba.rr has a proxy set to blubb" instead of saying where it takes the value from. And why doesn't it simply read the value from /etc/environment, which is the one true source of truth when it comes to proxy setting, or the current bash session in which I call kubeadm
which is the easiest place to make changes to?
What I expect would be something like this:
http_proxy
. (or https_proxy
if secure communication is configured)HTTP_PROXY
and warns if it is different.http_proxy
in /etc/environment. It warns if it is different.kubeadm reset
internally.no_proxy
settings (add the end it may get cut of). <-- Also it would be so much better to use a hostname if possible, since no_proxy
is actually meant for names, not IPs.I seriously can't express how many working hours it would save people in enterprise networks.
@erikbgithub Thanks a lot for this issue!
Up front I must say that I'm no proxy expert as I haven't experimented in such environments much.
So I can't comment on the exact statements above really, but I'd be very glad if you wanted to contribute to kubeadm to make the behavior behind a proxy better.
To answer your question, here is the relevant go code:
https://github.com/kubernetes/kubernetes/blob/master/cmd/kubeadm/app/phases/controlplane/manifests.go#L432
func getProxyEnvVars() []v1.EnvVar {
envs := []v1.EnvVar{}
for _, env := range os.Environ() {
pos := strings.Index(env, "=")
if pos == -1 {
// malformed environment variable, skip it.
continue
}
name := env[:pos]
value := env[pos+1:]
if strings.HasSuffix(strings.ToLower(name), "_proxy") && value != "" {
envVar := v1.EnvVar{Name: name, Value: value}
envs = append(envs, envVar)
}
}
return envs
}
https://github.com/kubernetes/kubernetes/blob/master/cmd/kubeadm/app/preflight/checks.go#L291
// HTTPProxyCheck checks if https connection to specific host is going
// to be done directly or over proxy. If proxy detected, it will return warning.
type HTTPProxyCheck struct {
Proto string
Host string
Port int
}
func (hst HTTPProxyCheck) Check() (warnings, errors []error) {
url := fmt.Sprintf("%s://%s:%d", hst.Proto, hst.Host, hst.Port)
req, err := http.NewRequest("GET", url, nil)
if err != nil {
return nil, []error{err}
}
proxy, err := http.DefaultTransport.(*http.Transport).Proxy(req)
if err != nil {
return nil, []error{err}
}
if proxy != nil {
return []error{fmt.Errorf("Connection to %q uses proxy %q. If that is not intended, adjust your proxy settings", url, proxy)}, nil
}
return nil, nil
}
I seriously can't express how many working hours it would save people in enterprise networks.
Couldn't agree more
cc @kad @timothysc
@luxas Thanks I'll work through that when I get a round tuit. Before I can supply patches I need to learn some go though, so I would appreciate if others can churn in for now. ;-)
First sub-question I'll look into is what go actually gets via os.Environ()
.
@erikbgithub Let me know if you need some help with creating patches and I'll help
@erikbgithub as original author of that check, I'll be happy to answer any questions.
First few answers:
$ env | grep -i _proxy= | sort
. E.g. inside our company firewall I have something like this:!shell
$ env | grep -i _proxy= | sort
ALL_PROXY=http://proxy-ir.example.com:911
FTP_PROXY=http://proxy-ir.example.com:911
HTTPS_PROXY=http://proxy-ir.example.com:911
HTTP_PROXY=http://proxy-ir.example.com:911
NO_PROXY=.example.com
all_proxy=http://proxy-ir.example.com:911
ftp_proxy=http://proxy-ir.example.com:911
http_proxy=http://proxy-ir.example.com:911
https_proxy=http://proxy-ir.example.com:911
no_proxy=.example.com
$
NO_PROXY=10.0.0.0/8, 192.168.0.0/16
will not have any effect and will still produce warning in pre-flight check.I "fixed" this problem by including all my cluster node IPs in NO_PROXY and using the same NO_PROXY on all the minions when joining the cluster.
$ export NO_PROXY='ip,ip,ip,ip,.example.com'
[master]$ kubeadm init
[minion]$ kubeadm join --token={token} a.b.c.d:6443
To be honest, I'm not sure if it's all the IP addresses being enumerated or the .example.com that fixed the problem.
if PR kubernetes/kubernetes#52788 will be merged, it will be possible to specify in NO_PROXY IP ranges for your nodes. it will simplify things a lot.
A little bit weired. if i look into the code "checks.go".
it always return error message if there are value in proxy.
if proxy != nil {
return []error{fmt.Errorf("Connection to %q uses proxy %q. If that is not intended, adjust your proxy settings", url, proxy)}, nil
}
return nil, nil
In enterprise...there are necessarily three proxy options. (http_proxy, https_proxy, no_proxy)
http_* is mandatory option to pull images for connection to internet.
if there is no_proxy option be set...then it should return error message.
" pl, set option(no_proxy) not to be routed to the proxy for internal connection"
I want to ask if kubeadm join supports http_proxy ?
I manage to get kubeadm init to work with http_proxy and no_proxy but it seems kubeadm join produce errors such as
kubelet.go:2105] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
remote_runtime.go:92] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = failed pulling image "gcr.io/google_containers/pause-amd64:3.0": Get https://gcr.io/v1/_ping: read tcp <my-ip>:58742->74.125.68.82:443: read: connection reset by peer
and also
/etc/environment is empty instead of filled with configuration like in the master.
which let me to believe maybe http_proxy and no_proxy is not yet supported for kubeadm join.
Running into this problem once more. It still uses the proxy incorrectly and I seem to be unable to modify proxy and no_proxy settings.
see #687, #182, or stack overflow: https://stackoverflow.com/questions/45580788/how-to-install-kubernetes-cluster-behind-proxy-with-kubeadm
From my experience, kubeadm use the proxy defined in /etc/environment
From my experience, kubeadm use the proxy defined in /etc/environment
Yup - in my case it is also /etc/environment
Most helpful comment
I "fixed" this problem by including all my cluster node IPs in NO_PROXY and using the same NO_PROXY on all the minions when joining the cluster.
To be honest, I'm not sure if it's all the IP addresses being enumerated or the .example.com that fixed the problem.