Moby: Docker container not reachable, bad TCP checksums

Created on 8 Oct 2015  ·  3Comments  ·  Source: moby/moby

Hi,

We are unable to reach the ports n the container directly or mapped.

INFO

Docker version 1.5.0, build a8a31ef

Debian 7 backported kernel 3.16.0-0.bpo.4-amd64 #1 SMP Debian 3.16.7-ckt9-3~deb8u1~bpo70+1 (2015-04-27) x86_64 GNU/Linux

Containers: 14
Images: 29
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 57
Execution Driver: native-0.2
Kernel Version: 3.16.0-0.bpo.4-amd64
Operating System: Debian GNU/Linux 7 (wheezy)
CPUs: 4
Total Memory: 7.815 GiB
Name: xxxxx
ID: PRMV:P7KM:5U6J:PYGZ:I3LQ:QWSB:ANFJ:VA23:URM4:JQOG:7TLM:4PTI
WARNING: No memory limit support
WARNING: No swap limit support

This is our additional settings:

docker0   Link encap:Ethernet  HWaddr 56:84:7a:fe:97:99  
          inet addr:172.17.42.1  Bcast:0.0.0.0  Mask:255.255.0.0
          inet6 addr: fe80::5484:7aff:fefe:9799/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:26765 errors:0 dropped:0 overruns:0 frame:0
          TX packets:30835 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:2173785 (2.0 MiB)  TX bytes:63950837 (60.9 MiB)

 vethc356415 Link encap:Ethernet  HWaddr 9e:e7:da:08:87:c6  
          inet6 addr: fe80::9ce7:daff:fe08:87c6/64 Scope:Link
          UP BROADCAST RUNNING  MTU:1500  Metric:1
          RX packets:200 errors:0 dropped:0 overruns:0 frame:0
          TX packets:147 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:12828 (12.5 KiB)  TX bytes:9398 (9.1 KiB)

Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         xxxxxxxxxxxx   0.0.0.0         UG    0      0        0 eth0
xxxxxxxxxx   0.0.0.0         255.255.255.240 U     0      0        0 eth0
172.17.0.0      0.0.0.0         255.255.0.0     U     0      0        0 docker0

 iptables -nL DOCKER
 Chain DOCKER (1 references)
 target     prot opt source               destination         
 ACCEPT     tcp  --  0.0.0.0/0            172.17.0.20          tcp dpt:22
 ACCEPT     tcp  --  0.0.0.0/0            172.17.0.20          tcp dpt:80

The running container with the interface vethc356415 is a Discourse container. These are its host and network settings:

 "HostConfig": {
        "Binds": [
            "/var/discourse/shared/standalone:/shared",
            "/var/discourse/shared/standalone/log/var-log:/var/log"
        ],
        "CapAdd": null,
        "CapDrop": null,
        "ContainerIDFile": "",
        "Devices": [],
        "Dns": null,
        "DnsSearch": null,
        "ExtraHosts": null,
        "IpcMode": "",
        "Links": null,
        "LxcConf": [],
        "NetworkMode": "bridge",
        "PidMode": "",
        "PortBindings": {
            "22/tcp": [
                {
                    "HostIp": "",
                    "HostPort": "2222"
                }
            ],
            "80/tcp": [
                {
                    "HostIp": "",
                    "HostPort": "10001"
                }
            ]
        },
        "Privileged": false,
        "PublishAllPorts": false,
        "ReadonlyRootfs": false,
        "RestartPolicy": {
            "MaximumRetryCount": 0,
            "Name": "always"
        },
        "SecurityOpt": null,
        "VolumesFrom": null
    },

 "NetworkSettings": {
        "Bridge": "docker0",
        "Gateway": "172.17.42.1",
        "GlobalIPv6Address": "",
        "GlobalIPv6PrefixLen": 0,
        "IPAddress": "172.17.0.20",
        "IPPrefixLen": 16,
        "IPv6Gateway": "",
        "LinkLocalIPv6Address": "fe80::42:acff:fe11:14",
        "LinkLocalIPv6PrefixLen": 64,
        "MacAddress": "02:42:ac:11:00:14",
        "PortMapping": null,
        "Ports": {
            "22/tcp": [
                {
                    "HostIp": "0.0.0.0",
                    "HostPort": "2222"
                }
            ],
            "80/tcp": [
                {
                    "HostIp": "0.0.0.0",
                    "HostPort": "10001"
                }
           ]
        }
    },

From inside of the container we are able to ping websites:

 ping google.com
 PING google.com (62.168.125.50) 56(84) bytes of data.
 64 bytes from 62.168.125.50: icmp_seq=1 ttl=59 time=0.486 ms
 64 bytes from 62.168.125.50: icmp_seq=2 ttl=59 time=0.520 ms

We are also able to get the index page of Discourse with wget (eg. doing a wget localhost from inside of container).

The problem is when we try to reach the container from outside (eg. from the Docker host):

$ wget 172.17.0.20

or 

$ wget localhost:10001

does not work and hangs. When we tried sniffing the traffic on docker0 interface we found out that there is a TCP checksum problem:

# tcpdump -i docker0 -vvv
tcpdump: listening on docker0, link-type EN10MB (Ethernet), capture size 65535 bytes
11:19:11.098052 IP (tos 0x0, ttl 64, id 30762, offset 0, flags [DF], proto TCP (6), length 60)
    172.17.42.1.51633 > 172.17.0.20.http: Flags [S], cksum 0xcc74 (correct), seq 2431285397, win 29200, options [mss 1460,sackOK,TS val 3003252927 ecr 0,nop,wscale 7], length 0
11:19:11.098127 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    172.17.0.20.http > 172.17.42.1.51633: Flags [S.], cksum 0x8266 (incorrect -> 0x26fb), seq 449304527, ack 2431285398, win 28960, options [mss 1460,sackOK,TS val 3003252927 ecr 3003252927,nop,wscale 7], length 0
11:19:12.095536 IP (tos 0x0, ttl 64, id 30763, offset 0, flags [DF], proto TCP (6), length 60)
    172.17.42.1.51633 > 172.17.0.20.http: Flags [S], cksum 0xcb7a (correct), seq 2431285397, win 29200, options [mss 1460,sackOK,TS val 3003253177 ecr 0,nop,wscale 7], length 0
11:19:12.095613 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    172.17.0.20.http > 172.17.42.1.51633: Flags [S.], cksum 0x8266 (incorrect -> 0x2601), seq 449304527, ack 2431285398, win 28960, options [mss 1460,sackOK,TS val 3003253177 ecr 3003252927,nop,wscale 7], length 0
11:19:13.303543 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    172.17.0.20.http > 172.17.42.1.51633: Flags [S.], cksum 0x8266 (incorrect -> 0x24d3), seq 449304527, ack 2431285398, win 28960, options [mss 1460,sackOK,TS val 3003253479 ecr 3003252927,nop,wscale 7], length 0
11:19:14.099543 IP (tos 0x0, ttl 64, id 30764, offset 0, flags [DF], proto TCP (6), length 60)
    172.17.42.1.51633 > 172.17.0.20.http: Flags [S], cksum 0xc985 (correct), seq 2431285397, win 29200, options [mss 1460,sackOK,TS val 3003253678 ecr 0,nop,wscale 7], length 0
11:19:14.099625 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    172.17.0.20.http > 172.17.42.1.51633: Flags [S.], cksum 0x8266 (incorrect -> 0x240c), seq 449304527, ack 2431285398, win 28960, options [mss 1460,sackOK,TS val 3003253678 ecr 3003252927,nop,wscale 7], length 0
11:19:15.903555 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    172.17.0.20.http > 172.17.42.1.51607: Flags [S.], cksum 0x8266 (incorrect -> 0x0eeb), seq 1342518319, ack 2198677968, win 28960, options [mss 1460,sackOK,TS val 3003254129 ecr 3003244349,nop,wscale 7], length 0

Seems like the docker container sends packets that are dropped because of bad checksums (?). We tried to disable any kind of offloading on the vethc356415:

ethtool -K vethc356415 gso off lro off tso off 
ethtool --offload vethc356415 tx off rx off

with no results. Right now we are stuck. Is this an issue or a configuration problem ?

picture gndgnd

Most helpful comment

Can you please provide some info on the firewall settings that needed to be changed to resolve this

picture surild on 29 Oct 2015
👍9

All 3 comments

Hi!

Please read this important information about creating issues.

If you are reporting a new issue, make sure that we do not have any duplicates already open. You can ensure this by searching the issue list for this repository. If there is a duplicate, please close your issue and add a comment to the existing issue instead.

If you suspect your issue is a bug, please edit your issue description to include the BUG REPORT INFORMATION shown below. If you fail to provide this information within 7 days, we cannot debug your issue and will close it. We will, however, reopen it if you later provide the information.

This is an automated, informational response.

Thank you.

For more information about reporting issues, see https://github.com/docker/docker/blob/master/CONTRIBUTING.md#reporting-other-issues

BUG REPORT INFORMATION

Use the commands below to provide key information from your environment:

docker version:
docker info:
uname -a:

Provide additional environment details (AWS, VirtualBox, physical, etc.):

List the steps to reproduce the issue:
1.
2.
3.

Describe the results you received:

Describe the results you expected:

Provide additional info you think is important:

----------END REPORT ---------

ENEEDMOREINFO

GordonTheTurtle picture GordonTheTurtle on 8 Oct 2015

The issue persisted after upgrade to docker 1.8.2. Issue resolved by changing the host firewall settings.

gndgnd picture gndgnd on 9 Oct 2015

Can you please provide some info on the firewall settings that needed to be changed to resolve this

surild picture surild on 29 Oct 2015
👍9
Was this page helpful?
0 / 5 - 0 ratings

Related issues

nisanharamati picture nisanharamati  ·  3Comments
cefn picture cefn  ·  3Comments
kunyan picture kunyan  ·  3Comments
cmingxu picture cmingxu  ·  3Comments
moxiegirl picture moxiegirl  ·  3Comments