Moby: docker daemon unable to access registry - Client.Timeout exceeded while awaiting headers

Exactly the same issue here.

+1 on 1.11.2 under Ubuntu 14.04.4 LTS

Hi, I think we have hit the #15949 issue

Lowering my server's MTU to 1500 has worked around the problem.

Cheers

I already hat MTU size of 1500 and still not working.

Apparently, enabling MTU probing in the kernel resolved the issue. To enable it just run:

 sudo sysctl -w net.ipv4.tcp_mtu_probing=1

To make this permanently, add the following line to /etc/sysctl.conf:

net.ipv4.tcp_mtu_probing=1

https://thesimplecomputer.info/pages/adventures-in-linux-tcp-tuning-page2

Glad you found a solution.

I'm also getting this issue:

Error getting v2 registry: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Attempting next endpoint for pull after error: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Not continuing with pull after error: Network timed out while trying to connect to https://index.docker.io/v1/repositories/armhf/busybox/images. You may want to check your internet connection or if you are behind a proxy.

docker info:

Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 0
Server Version: 1.12.1
Storage Driver: overlay
 Backing Filesystem: extfs
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: overlay null bridge host
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options: seccomp
Kernel Version: 4.4.19-1-ARCH
Operating System: Arch Linux ARM
OSType: linux
Architecture: armv7l
CPUs: 4
Total Memory: 922.2 MiB
Name: pi3arch
ID: 7MTA:UZIW:SVAS:H3U7:YBHE:FWXV:KWOS:4QAQ:HDLG:EWIV:DOAP:TWFR
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Insecure Registries:
 127.0.0.0/8

I've set the mtu to 1500 and net.ipv4.tcp_mtu_probing=1 with no success :(
Any ideas?

Same issue here as reported by @luxas.

Same problem here, mtu already be 1500 and net.ipv4.tcp_mtu_probing=1 with no success.
@luxas @MathiasRenner

I found out that the problem might be in /etc/resolv.conf. I had:

nameserver 10.0.0.10
nameserver 8.8.8.8
nameserver 8.8.4.4

but moving the non-working (yet) 10.0.0.10 nameserver to the bottom fixed it for me I think

nameserver 8.8.8.8
nameserver 8.8.4.4
nameserver 10.0.0.10

Thank you luxas
It works great!!!

@luxas I worked for my linux system, but not getting what to do on windows. Please help if possible.

facing the same issue here changed the resolv.conf , modified mtu size .. no use. docker is still able to contact the registry.

docker run hello-world
Unable to find image 'hello-world:latest' locally
docker: Error response from daemon: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers).
See 'docker run --help'.
Am seeing the above error

@diptiburli I had the similar issue, what I did to resolve that:

1. update docker and docker-machine to the latest version, 1.13.0
2. brew link --overwrite docker and docker-machine
3. docker-machine restart {VM}

Although I still have error with curl https://registry-1.docker.io/v2/, but the docker pull works for me:

$ curl https://registry-1.docker.io/v2/
{"errors":[{"code":"UNAUTHORIZED","message":"authentication required","detail":null}]}

Let me know if this works for you.

For Windows 10 Home, Putting 8.8.8.8 as the first dns resolves it.

@aasaanhai also worked for me.
Simply set the default DNS to 8.8.8.8 on the "vEthernet (DockerNAT)" network adapter.

thanks but where simply set the default DNS to 8.8.8.8 ?
on debian for exemple... thanks again

@strainel Should be on your network connections area, you're running linux so it should be similar to my ubuntu, use the GUI to go into your network settings, select the connection you're on, and then put the google dns settings in there. I don't have practical experience with debian but a google search should do the trick.

thanks @JohnVonNeumann i solved my issue.
the message "Error response from daemon: Get https://registry-1.docker.io/v2/: net/http: request canceled (Client.Timeout exceeded while awaiting headers)" was caused by network connection with a proxy. with https://docs.docker.com/engine/admin/systemd/ (§ http proxy), it is ok.
i am surprised docker use systemd on debian... and not environment variables.
thanks again

+1 in RHEL 7

[root@minion2 yum.repos.d]# docker info
Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 0
Server Version: 17.03.0-ce
Storage Driver: overlay
 Backing Filesystem: xfs
 Supports d_type: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 977c511eda0925a723debdc94d09459af49d082a
runc version: a01dafd48bc1c7cc12bdb01206f9fea7dd6feb70
init version: 949e6fa
Security Options:
 seccomp
  Profile: default
Kernel Version: 3.10.0-514.el7.x86_64
Operating System: Red Hat Enterprise Linux Server 7.3 (Maipo)
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 2.781 GiB
Name: minion2
ID: Q6KC:VGM5:DE7P:I2TW:H2SQ:A3CI:GFRM:IPLZ:MKA5:SHJI:6CLP:4XSU
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false
[root@minion2 yum.repos.d]#

Setting the DNS server to fixed (8.8.8.8) from automatic in docker settings solved the problem for me.

Hello,

I am still facing same issue. The server is behind the corporate proxy but woks for all other applications.

export http_proxy=http://user:password@proxyIP:8080/
export https_proxy=http://user:password@proxyIP:8080/
docker login
Login with your Docker ID to push and pull images from Docker Hub. If you don't have a Docker ID, head over to https://hub.docker.com to create one.
Username: xxxxxxx
Password:
Error response from daemon: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

I got it solved by adding the proxy details in the docker config file /etc/default/docker.

Same problem faced running docker-machine on windows. Restarting docker-machine worked for me suggested by chenrui333. My docker version is 1.13.1

docker-machine restart

docker login as well as docker pull is working now.

Same problem here, mtu already be 1500 and net.ipv4.tcp_mtu_probing=1 with no success.dns=8.8.8.8 also no use

I am getting the same issue...
I have tried change the mtu (1500,1400,1300) and nothing happens. There is not a proxy issue neither.
Can someone help ?
Thanks

Unable to find image 'mysql:latest' locally docker: Error response from daemon: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers). See 'docker run --help'.

docker version
Client:
Version: 17.03.1-ce
API version: 1.27
Go version: go1.7.5
Git commit: c6d412e
Built: Tue Mar 28 00:40:02 2017
OS/Arch: darwin/amd64

Server:
Version: 17.03.1-ce
API version: 1.27 (minimum version 1.12)
Go version: go1.7.5
Git commit: c6d412e
Built: Fri Mar 24 00:00:50 2017
OS/Arch: linux/amd64
Experimental: false

I've experienced the above described problem this morning on my Ubuntu 16.04 machine (4.4.0-75-generic) as well.

$ docker version
Client:
 Version:      17.04.0-ce
 API version:  1.28
 Go version:   go1.7.5
 Git commit:   4845c56
 Built:        Mon Apr  3 18:07:42 2017
 OS/Arch:      linux/amd64

Server:
 Version:      17.04.0-ce
 API version:  1.28 (minimum version 1.12)
 Go version:   go1.7.5
 Git commit:   4845c56
 Built:        Mon Apr  3 18:07:42 2017
 OS/Arch:      linux/amd64
 Experimental: false

After login with sudo rights (sudo docker login -u username -p password), which apparently worked, I've deleted the docker config (sudo rm ~/.docker/config.json) and restarted the docker service (sudo /etc/init.d/docker restart). For some magical reason I was now able to login again without super user rights.

Hope that helps.

root@xxxx:~# cat /etc/issue
Ubuntu 16.04 LTS \n \l
root@xxxx:~# docker version
Client:
Version: 17.03.1-ce
API version: 1.27
Go version: go1.7.5
Git commit: c6d412e
Built: Mon Mar 27 17:14:09 2017
OS/Arch: linux/amd64

Server:
Version: 17.03.1-ce
API version: 1.27 (minimum version 1.12)
Go version: go1.7.5
Git commit: c6d412e
Built: Mon Mar 27 17:14:09 2017
OS/Arch: linux/amd64
Experimental: false
root@xxxx:~# docker run hello-world
Unable to find image 'hello-world:latest' locally
docker: Error response from daemon: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers).
See 'docker run --help'.

In my case, setting the environment variable http_proxy directly in shell and then run systemctl restart docker doesn't work at all, but we need to follow the instructions related to proxy server settings in https://docs.docker.com/engine/admin/systemd/, then it works!

Please refer to this: https://docs.docker.com/engine/admin/systemd/#httphttps-proxy
I have resolved on Ubuntu 16.04 LTS

I'm getting this on Mac OS 10.12.5. I tried re-installing to the latest version of docker, and still no luck. Here's the error I'm getting:
Pulling wordpress (wordpress:latest)...
ERROR: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

Same issue here on Mac 10.12.4

Romans-Mac-Pro.local ➜  experimenting-armhf  docker build -t ruby-armhf-test .
Sending build context to Docker daemon  2.048kB
Step 1/2 : FROM arm32v7/ruby:2.4.1
Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Romans-Mac-Pro.local ➜  experimenting-armhf  wget 'https://registry-1.docker.io/v2/'
--2017-07-24 18:15:51--  https://registry-1.docker.io/v2/
Resolving registry-1.docker.io... 54.86.192.189, 52.72.188.69, 34.225.140.215
Connecting to registry-1.docker.io|54.86.192.189|:443... connected.
HTTP request sent, awaiting response... 401 Unauthorized

Username/Password Authentication Failed.
Romans-Mac-Pro.local ➜  experimenting-armhf  

Any ideas?

I was getting this same error in Windows 10 Home using Docker Toolbox. Changing my DNS to Google's 8.8.8.8 (secondary to 8.8.4.4) worked for me (per @aasaanhai above). Detailed instructions for how to do so here (see section II).

@alldev0825, thanks! It works for me.

As kirstin said setting the DNS on Docker fixes the issue

DNS to 8.8.8.8 on the "vEthernet (DockerNAT)" network adapter.

You can do this from Powershell with

Get-DnsClientServerAddress | Where-Object { $_.InterfaceAlias -like "*(DockerNAT)" } | ForEach-Object { Set-DnsClientServerAddress -InterfaceIndex $_.InterfaceIndex -ServerAddresses @("8.8.8.8", "8.8.4.4") }

Here is a test setup with a Debian 9 Stretch; NOT behind a proxy.
From time to time the error is not there without system changes, so a pull occasionally succeeds or not.
Maybe this is due to slow network and timeout could be set higher?

The solution from @brunolm worked for me! Thanks!

adding name server 8.8.8.8 does it, wonder why docker don't care at all still this day? They want you to get the enterprise version or use their cloud?

I had a similar issue trying to login to my own docker repo.
Here is how i've managed to resolve it:
https://stackoverflow.com/questions/46822391/docker-toolbox-tutorial-client-timeout-exceeded-while-awaiting-headers/48064026#48064026

[ubuntu16.04, docker 18.01.0]
In my case, I could solve by setting proxy like this. This way is different one from above.

$ sudo mkdir -p /etc/systemd/system/docker.service.d
$ sudo vi /etc/systemd/system/docker.service.d/http_proxy.conf
[Service]
Environment="HTTP_PROXY=http://:/"
$ sudo vi /etc/systemd/system/docker.service.d/https_proxy.conf
[Service]
Environment="HTTPS_PROXY=http://:/"
$ sudo systemctl daemon-reload
$ sudo systemctl restart docker

I added below configuration on /etc/resolv.conf as suggested above and it works

nameserver 8.8.8.8
nameserver 8.8.4.4
nameserver 10.0.0.10
​

I faced the same problem on a Windows 10 machine. To solve the problem I accessed the Docker client Settings screen, tab Network, and changed the DNS Server to fixed IP 8.8.8.8. After restarting docker everything start working properly.

I'm on a Windows 10 machine as well, not behind proxy and setting DNS to 8.8.8.8 isn't working for me. No idea what is going on.

Changed the DNS Server to fixed IP 8.8.8.8 and it worked on windows 10

I tried this "solution" and it doesn't work.

8.8.8.8 didn't work for me, but 8.8.4.4 did work.
looks like whoever got this error- try different dns's and hopefully one of them will work.

Windows 10: in Docker Client Settings, Network Tab. DNS Server set to Fixed and restart and then set back to Automatic and restart - worked for me.

thx

Hi Everyone,
Request every one to give any solution to me.
I have implemented everything that is written in this thread, but still the i am getting this error
"Client.Timeout exceeded while awaiting headers"
I am using Ubuntu 17.10.

Regards,
Aj

Hi, in my case setting a fixed DNSmade things worse, the main course was the companys proxy (as always :) ). Try to change the proxy to a working one or working without one.

Yes i also think its due to company proxy. Though i have mentioned the
proxy in/etc/environment file and/etc/apt/apt.conf file
Let me try few more proxy combinations.

On Jun 13, 2018 6:30 PM, "supernicky" notifications@github.com wrote:

Hi, in my case setting a fixed DNSmade things worse, the main course was
the companys proxy (as always :) ). Try to change the proxy to a working
one or working without one.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/moby/moby/issues/22635#issuecomment-396927877, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AGz0zDK7Ty98wix4mtRdr1qtHHjbFW4dks5t8QzWgaJpZM4IbXBr
.

@alldev0825 thanks. For old docker 1.13, can use /etc/sysconfig/docker to set proxy ,but for 17.06 docker-ce, it does not work,need setup proxy in the systemd dir as the link you supplied.

Thanks AllDev and Baul,

Let me try this, I will update you.

Thanks Buddy

On Mon, Jul 2, 2018 at 4:32 PM, baul notifications@github.com wrote:

@alldev0825 https://github.com/alldev0825 thanks. For old docker 1.13,
can use /etc/sysconfig/docker to set proxy ,but for 17.06 docker-ce, it
does not work,need setup proxy in the systemd dir as the link you supplied.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/moby/moby/issues/22635#issuecomment-401768028, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AGz0zEiSo7eyIYO7opx2bMhAGv1ULj5Fks5uCf3UgaJpZM4IbXBr
.

I already hat MTU size of 1500 and still not working.

Apparently, enabling MTU probing in the kernel resolved the issue. To enable it just run:

 sudo sysctl -w net.ipv4.tcp_mtu_probing=1

To make this permanently, add the following line to /etc/sysctl.conf:

net.ipv4.tcp_mtu_probing=1

https://thesimplecomputer.info/pages/adventures-in-linux-tcp-tuning-page2

Thank you.. It fixed the issue 💯

I added below configuration on /etc/resolv.conf as suggested above and it works

nameserver 8.8.8.8
nameserver 8.8.4.4
nameserver 10.0.0.10
​

this one worked for me! thank you so much :)

docker login to private registry in our environment is failing intermittently.
We are having same issue with private registry.
Configuring Google's DNS would work for docker.io but for internal registries how can we extend the login timeout value, esp with slow AD Authorizations.

We have configured our docker NGINX with proxy_ignore_client_abort on; to ignore client timeouts to verify if there is a server kick outs. But the client looks hard coded with 15s timeout irrespective of the server behaviour. Any way we can change/extend this timeout?

one thing to point out: using docker login from Docker: check /etc/resolv.conf which is used by daemon to resolve the registry. Not the /etc/hosts.

Same Issue.
With docker Server Version: 17.03.1-ce on CentOS Linux release 7.4.1708 (Core).
Got mtu as 1500, net.ipv4.tcp_mtu_probing=1 and modified /etc/resolv.conf file, no success.

Please anyone light me. Thanks a lot.

On my Win10, changing the DNS does _only_ work because the Docker daemon restarts to apply the change. In other words: restarting the daemon helps, _no matter what_ DNS server is configured.

what if i dont have a dns when the registry is local in the lan. then manually adding reg server to /etc/hosts file worked for me

I just simply changed the nameserver /etc/resolv.conf
nameserver 192.168.1.1
nameserver 8.8.8.8
nameserver 8.8.4.4
Thank god, It worked for me. Thank you!

I found out that the problem might be in /etc/resolv.conf. I had:

nameserver 10.0.0.10
nameserver 8.8.8.8
nameserver 8.8.4.4

but moving the non-working (yet) 10.0.0.10 nameserver to the bottom fixed it for me I think

nameserver 8.8.8.8
nameserver 8.8.4.4
nameserver 10.0.0.10

This worked for me thanks

I found out that the problem might be in /etc/resolv.conf. I had:

nameserver 192.168.0.1

after editing to

nameserver 8.8.8.8
nameserver 8.8.4.4
nameserver 10.0.0.10
This worked for me.

for me below steps worked

Create a systemd drop-in directory for the docker service:

sudo mkdir -p /etc/systemd/system/docker.service.d
Create a file named /etc/systemd/system/docker.service.d/http-proxy.conf that adds the HTTP_PROXY environment variable:

[Service]
Environment="HTTP_PROXY=http://proxy.example.com:80"
If you are behind an HTTPS proxy server, set the HTTPS_PROXY environment variable:

[Service]
Environment="HTTPS_PROXY=https://proxy.example.com:443"
Multiple environment variables can be set; to set both a non-HTTPS and a HTTPs proxy;

[Service]
Environment="HTTP_PROXY=http://proxy.example.com:80"
Environment="HTTPS_PROXY=https://proxy.example.com:443"

then do

sudo systemctl stop docker
sudo systemctl daemon-reload
sudo systemctl start docker