Flannel: Could you share results of route and traceroute when flannel is used with docker?

Created on 28 Feb 2016  ·  5Comments  ·  Source: coreos/flannel

My current environment is centos7/docker1.9.1/flannel0.5.5(backend is vxlan)/kubernetes1.1.7.
I has tested almost every examples of kubernetes.io and they have worked.
but redis sentinel example failed.
( reason : redis-cli>info => slave ip is not slave redis-server container ip but flannel.1 ip of slave redis-server container host. )

...
Kubernetes imposes the following fundamental requirements on any networking implementation
(barring any intentional network segmentation policies):

all containers can communicate with all other containers without NAT
all nodes can communicate with all containers (and vice-versa) without NAT
the IP that a container sees itself as is the same IP that others see it as
...

First of all, i would like to check whether routing table and traceroute result is right or not.
Could anyone share your routing table results and traceroute results?
( and could anyone tell me what this flannel log means exactly? Ignoring not a miss
and could anyone tell me where i can find the definitions of flannel log? )

Thank you in advance.

Below is my settings and results.

Basic Information

Node1

192.168.0.202
flannel.1 : 172.16.4.0
docker0 : 172.16.4.1

Node2

192.168.0.97
flannel.1 : 172.16.53.0
docker0 : 172.16.53.1

Routing Table Result( route command )

Node1

Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.0.1     0.0.0.0         UG    100    0        0 em1
172.16.0.0      0.0.0.0         255.255.0.0     U     0      0        0 flannel.1
172.16.4.0      0.0.0.0         255.255.255.0   U     0      0        0 docker0
192.168.0.0     0.0.0.0         255.255.255.0   U     100    0        0 em1
192.168.122.0   0.0.0.0         255.255.255.0   U     0      0        0 virbr0

Node2

Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.0.1     0.0.0.0         UG    100    0        0 enp3s0
172.16.0.0      0.0.0.0         255.255.0.0     U     0      0        0 flannel.1
172.16.53.0     0.0.0.0         255.255.255.0   U     0      0        0 docker0
192.168.0.0     0.0.0.0         255.255.255.0   U     100    0        0 enp3s0
192.168.122.0   0.0.0.0         255.255.255.0   U     0      0        0 virbr0

traceroute result from containers : from Node1 container 172.16.4.2 to Node2 container 172.16.53.3

traceroute to 172.16.53.3 (172.16.53.3), 30 hops max, 60 byte packets
 1  gateway (172.16.4.1)  0.119 ms  0.022 ms  0.020 ms
 2  172.16.53.0 (172.16.53.0)  2.094 ms  2.049 ms  2.017 ms
 3  172.16.53.3 (172.16.53.3)  1.987 ms  1.957 ms  1.925 ms

flannel setting Node1

systemctl status flanneld

flanneld.service - Flanneld overlay address etcd agent
   Loaded: loaded (/usr/lib/systemd/system/flanneld.service; enabled; vendor preset: disabled)
   Active: active (running) since Sat 2016-02-27 15:56:02 EST; 3min 32s ago
  Process: 600 ExecStartPost=/usr/libexec/flannel/mk-docker-opts.sh -k DOCKER_NETWORK_OPTIONS -d /run/flannel/docker (code=exited, status=0/SUCCESS)
  Process: 541 ExecStartPre=/opt/kubernetes/cluster/centos/node/bin/remove-docker0.sh (code=exited, status=0/SUCCESS)
 Main PID: 561 (flanneld)
   Memory: 40.0K
   CGroup: /system.slice/flanneld.service
           └─561 /opt/flannel-0.5.5/flanneld --ip-masq -etcd-endpoints=http://127.0.0.1:2379 -etcd-prefix=/docker.intranet/network
Feb 27 15:56:02 mars flanneld[561]: I0227 15:56:02.654159 00561 device.go:159] calling GetL2List() dev.link.Index: 8
Feb 27 15:56:02 mars flanneld[561]: I0227 15:56:02.654336 00561 vxlan.go:280] fdb already populated with: 192.168.0.97 d6:00:92:5d:f1:cc
Feb 27 15:56:02 mars flanneld[561]: I0227 15:56:02.654389 00561 device.go:176] calling NeighDel: 192.168.0.97, d6:00:92:5d:f1:cc
Feb 27 15:56:02 mars systemd[1]: Started Flanneld overlay address etcd agent.
Feb 27 15:57:44 mars flanneld[561]: I0227 15:57:44.882885 00561 vxlan.go:232] Subnet added: 172.16.53.0/24
Feb 27 15:57:44 mars flanneld[561]: I0227 15:57:44.884067 00561 device.go:164] calling NeighAdd: 192.168.0.97, d6:00:92:5d:f1:cc
Feb 27 15:58:35 mars flanneld[561]: I0227 15:58:35.795639 00561 vxlan.go:345] L3 miss: 172.16.53.3
Feb 27 15:58:35 mars flanneld[561]: I0227 15:58:35.795722 00561 device.go:187] calling NeighSet: 172.16.53.3, d6:00:92:5d:f1:cc
Feb 27 15:58:35 mars flanneld[561]: I0227 15:58:35.796089 00561 vxlan.go:356] AddL3 succeeded
Feb 27 15:58:52 mars flanneld[561]: I0227 15:58:52.367089 00561 vxlan.go:340] Ignoring not a miss: d6:00:92:5d:f1:cc, 172.16.53.3

cat /usr/lib/systemd/system/flanneld.service

[Unit]
Description=Flanneld overlay address etcd agent
After=network.target
After=network-online.target
Wants=network-online.target
After=etcd.service
Before=docker.service
[Service]
Type=notify
EnvironmentFile=/etc/sysconfig/flanneld
EnvironmentFile=-/etc/sysconfig/docker-network
ExecStartPre=/opt/kubernetes/cluster/centos/node/bin/remove-docker0.sh
ExecStart=/opt/flannel-0.5.5/flanneld --ip-masq -etcd-endpoints=${FLANNEL_ETCD} -etcd-prefix=${FLANNEL_ETCD_KEY} $FLANNEL_OPTIONS
ExecStartPost=/usr/libexec/flannel/mk-docker-opts.sh -k DOCKER_NETWORK_OPTIONS -d /run/flannel/docker
Restart=on-failure
[Install]
WantedBy=multi-user.target
RequiredBy=docker.service

cat /etc/sysconfig/flanneld

FLANNEL_ETCD="http://127.0.0.1:2379"
FLANNEL_ETCD_KEY="/docker.intranet/network"

docker setting from Node1

systemctl status docker

 docker.service - Docker Application Container Engine
   Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; vendor preset: disabled)
  Drop-In: /usr/lib/systemd/system/docker.service.d
           └─flannel.conf
   Active: active (running) since Sat 2016-02-27 15:56:04 EST; 5min ago
     Docs: http://docs.docker.com
 Main PID: 864 (docker)
   Memory: 216.0K
   CGroup: /system.slice/docker.service
           └─864 /usr/bin/docker daemon --bip=172.16.4.1/24 --mtu=1450 -H tcp://127.0.0.1:4243 -H unix:///var/run/docker.sock -s devicemapper --selinux-enabled=false
Feb 27 15:56:04 mars docker[864]: time="2016-02-27T15:56:04.302811588-05:00" level=info msg="Loading containers: start."
Feb 27 15:56:04 mars docker[864]: .................................................................................................................................................................................
Feb 27 15:56:04 mars docker[864]: time="2016-02-27T15:56:04.435453869-05:00" level=info msg="Loading containers: done."
Feb 27 15:56:04 mars docker[864]: time="2016-02-27T15:56:04.435476739-05:00" level=info msg="Daemon has completed initialization"
Feb 27 15:56:04 mars docker[864]: time="2016-02-27T15:56:04.435503959-05:00" level=info msg="Docker daemon" commit=a34a1d5 execdriver=native-0.2 graphdriver=devicemapper version=1.9.1
Feb 27 15:56:04 mars systemd[1]: Started Docker Application Container Engine.
Feb 27 15:56:11 mars docker[864]: time="2016-02-27T15:56:11.613112287-05:00" level=info msg="POST /v1.21/containers/create?name=redis"
Feb 27 15:56:11 mars docker[864]: time="2016-02-27T15:56:11.803031083-05:00" level=info msg="POST /v1.21/containers/86801e7123c5617df6cf292f107cfb47903376f8b2edaf2a7381410d8deef82b/attach?stder...out=1&stream=1"
Feb 27 15:56:11 mars docker[864]: time="2016-02-27T15:56:11.804056489-05:00" level=info msg="POST /v1.21/containers/86801e7123c5617df6cf292f107cfb47903376f8b2edaf2a7381410d8deef82b/start"
Feb 27 15:56:12 mars docker[864]: time="2016-02-27T15:56:12.039108222-05:00" level=info msg="POST /v1.21/containers/86801e7123c5617df6cf292f107cfb47903376f8b2edaf2a7381410d8deef82b/resize?h=59&w=211"
Hint: Some lines were ellipsized, use -l to show in full.

cat /usr/lib/systemd/system/docker.service

[Unit]
Description=Docker Application Container Engine
Documentation=http://docs.docker.com
After=network.target flanneld.service
Requires=flanneld.service
[Service]
Type=notify
EnvironmentFile=-/run/flannel/docker
EnvironmentFile=-/opt/kubernetes/cfg/docker
WorkingDirectory=/opt/kubernetes/server/kubernetes/server/bin
ExecStart=/usr/bin/docker daemon $DOCKER_OPT_BIP $DOCKER_OPT_MTU $DOCKER_OPTS
LimitNOFILE=1048576
LimitNPROC=1048576
[Install]
WantedBy=multi-user.target

cat /run/flannel/docker

DOCKER_OPT_BIP="--bip=172.16.4.1/24"
DOCKER_OPT_IPMASQ="--ip-masq=false"
DOCKER_OPT_MTU="--mtu=1450"
DOCKER_NETWORK_OPTIONS=" --bip=172.16.4.1/24 --ip-masq=false --mtu=1450 "

flannel log( journalctl -u flanneld )

Feb 27 15:56:02 mars systemd[1]: Starting Flanneld overlay address etcd agent...
Feb 27 15:56:02 mars flanneld[561]: I0227 15:56:02.626893 00561 main.go:275] Installing signal handlers
Feb 27 15:56:02 mars flanneld[561]: I0227 15:56:02.627020 00561 main.go:130] Determining IP address of default interface
Feb 27 15:56:02 mars flanneld[561]: I0227 15:56:02.628028 00561 main.go:188] Using 192.168.0.202 as external interface
Feb 27 15:56:02 mars flanneld[561]: I0227 15:56:02.628058 00561 main.go:189] Using 192.168.0.202 as external endpoint
Feb 27 15:56:02 mars flanneld[561]: I0227 15:56:02.630035 00561 etcd.go:204] Picking subnet in range 172.16.1.0 ... 172.16.255.0
Feb 27 15:56:02 mars flanneld[561]: I0227 15:56:02.630662 00561 etcd.go:84] Subnet lease acquired: 172.16.4.0/24
Feb 27 15:56:02 mars flanneld[561]: I0227 15:56:02.636348 00561 ipmasq.go:50] Adding iptables rule: FLANNEL -d 172.16.0.0/16 -j ACCEPT
Feb 27 15:56:02 mars flanneld[561]: I0227 15:56:02.641073 00561 ipmasq.go:50] Adding iptables rule: FLANNEL ! -d 224.0.0.0/4 -j MASQUERADE
Feb 27 15:56:02 mars flanneld[561]: I0227 15:56:02.645597 00561 ipmasq.go:50] Adding iptables rule: POSTROUTING -s 172.16.0.0/16 -j FLANNEL
Feb 27 15:56:02 mars flanneld[561]: I0227 15:56:02.648876 00561 ipmasq.go:50] Adding iptables rule: POSTROUTING ! -s 172.16.0.0/16 -d 172.16.0.0/16 -j MASQUERADE
Feb 27 15:56:02 mars flanneld[561]: I0227 15:56:02.652895 00561 vxlan.go:153] Watching for L3 misses
Feb 27 15:56:02 mars flanneld[561]: I0227 15:56:02.652930 00561 vxlan.go:159] Watching for new subnet leases
Feb 27 15:56:02 mars flanneld[561]: I0227 15:56:02.654135 00561 vxlan.go:273] Handling initial subnet events
Feb 27 15:56:02 mars flanneld[561]: I0227 15:56:02.654159 00561 device.go:159] calling GetL2List() dev.link.Index: 8
Feb 27 15:56:02 mars flanneld[561]: I0227 15:56:02.654336 00561 vxlan.go:280] fdb already populated with: 192.168.0.97 d6:00:92:5d:f1:cc
Feb 27 15:56:02 mars flanneld[561]: I0227 15:56:02.654389 00561 device.go:176] calling NeighDel: 192.168.0.97, d6:00:92:5d:f1:cc
Feb 27 15:56:02 mars systemd[1]: Started Flanneld overlay address etcd agent.
Feb 27 15:57:44 mars flanneld[561]: I0227 15:57:44.882885 00561 vxlan.go:232] Subnet added: 172.16.53.0/24
Feb 27 15:57:44 mars flanneld[561]: I0227 15:57:44.884067 00561 device.go:164] calling NeighAdd: 192.168.0.97, d6:00:92:5d:f1:cc
Feb 27 15:58:35 mars flanneld[561]: I0227 15:58:35.795639 00561 vxlan.go:345] L3 miss: 172.16.53.3
Feb 27 15:58:35 mars flanneld[561]: I0227 15:58:35.795722 00561 device.go:187] calling NeighSet: 172.16.53.3, d6:00:92:5d:f1:cc
Feb 27 15:58:35 mars flanneld[561]: I0227 15:58:35.796089 00561 vxlan.go:356] AddL3 succeeded
Feb 27 15:58:52 mars flanneld[561]: I0227 15:58:52.367089 00561 vxlan.go:340] Ignoring not a miss: d6:00:92:5d:f1:cc, 172.16.53.3
kinbug kinsupport revieweneeds more information
Source
picture euclid-geometry

Most helpful comment

I solved my problem.
I think there is a problem of documentation about flannel and kubernetes.
I would like to discuss that later.

picture euclid-geometry on 29 Feb 2016
👍3

All 5 comments

Additional Information about TCP connection between docker containers of different host

Container A from Node1( 192.168.0.202 )

ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1450
        inet 172.16.30.2  netmask 255.255.255.0  broadcast 0.0.0.0
        inet6 fe80::42:acff:fe10:1e02  prefixlen 64  scopeid 0x20<link>
        ether 02:42:ac:10:1e:02  txqueuelen 0  (Ethernet)
        RX packets 6  bytes 508 (508.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 6  bytes 508 (508.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

Container B from Node2( 192.168.0.97 )

ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1450
        inet 172.16.83.3  netmask 255.255.255.0  broadcast 0.0.0.0
        inet6 fe80::42:acff:fe10:5303  prefixlen 64  scopeid 0x20<link>
        ether 02:42:ac:10:53:03  txqueuelen 0  (Ethernet)
        RX packets 6  bytes 508 (508.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 6  bytes 508 (508.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

TCP connection information at Container A

( Container A is redis master, Container B is redis slave )
ss -tanp

| State | Recv-Q | Send-Q | Local Address:Port | Peer Address:Port | Process |
| --- | --- | --- | --- | --- | --- |
| LISTEN | 0 | 128 | :6379 | _:_ | users:(("redis-server",pid=15,fd=5)) |
| ESTAB | 0 | 0 | 172.16.30.2:6379 | 172.16.83.0:53216 | users:(("redis-server",pid=15,fd=6)) |
| LISTEN | 0 | 128 | :::6379 | ::: | users:(("redis-server",pid=15,fd=4)) |

My problem is 172.16.83.0:53216.
I expected 172.16.83.3:53216.

Could anyone tell me why ip address is 172.16.83.0:53216?

euclid-geometry picture euclid-geometry on 29 Feb 2016

I solved my problem.
I think there is a problem of documentation about flannel and kubernetes.
I would like to discuss that later.

euclid-geometry picture euclid-geometry on 29 Feb 2016
👍3

Could you also share your solution?

oussama1983-zz picture oussama1983-zz on 29 Apr 2016
👍1

@euclid-geometry could you share your solution? we suffer with the same problem, thank you.

ganting picture ganting on 12 Jun 2016

@euclid-geometry could you share your solution? thank you.

libn-net picture libn-net on 22 Jun 2017
Was this page helpful?
0 / 5 - 0 ratings

Related issues

lvthillo picture lvthillo  ·  6Comments
SyCode7 picture SyCode7  ·  4Comments
Dieken picture Dieken  ·  6Comments
TLmaK0 picture TLmaK0  ·  3Comments
smorrisfv picture smorrisfv  ·  7Comments