Moby: docker swarm mode: ports on 127.0.0.1 are exposed to 0.0.0.0

Yes this should output an error; services (by default) "publish" using the "ingress" network, and do not support specifying an IP-address, as it's not possible to predict which _node_ they end up on (thus not known what IP-addresses are available - although 127.0.0.1 could be possible). This issue is tracking that feature https://github.com/docker/docker/issues/26696 (and this "epic" tracks other options not (yet) supported by services https://github.com/docker/docker/issues/25303)

The bug here is that docker should produce an error, instead of silently ignoring the option; reproducible using this minimal docker-compose file;

version: "3.2"
services:
  mongodb:
    image: nginx:alpine
    ports:
      - "127.0.0.1:27017:80"

ping @dnephin @vdemeester

@fer2d2 On swarm mode, if you publish something (ports for stack deploy), it is published on the ingress network, and thus it is public. There is a few ways to get around, but putting kind/bug on that because we should at least warn people about that when doing a stack deploy with ports that have this notation (i.e. host:port:port).

To work around this, there is a few ways:

• first, you should publish mongo ports only if you want it to be public, otherwise, it is available through the name discovery bundle in docker (another container/service on the same network will be able to reach it through mongo dns name).
• If you want to publish it in the host and not in ingress (so not swarm public, just on the host it is running, same way as without swarm mode), you need to use ports expanded syntax.

    ports:
      - mode: host
        target: 80
        published: 9005

It will do the same as docker run -p 80:9005 … so it will bind it to 0.0.0.0, but on limited to the host.

But as @thaJeztah said, "The bug here is that docker should produce an error, instead of silently ignoring the option" 👼

/cc @mavenugo @aboch to see if there would be a way to actually being able to bind it to a specific ip ? (really tricky to achieve because node's ip will be different so..)

@vdemeester Could I specify localhost as the host target using this notation?

    ports:
      - mode: host
        target: 127.0.0.1:80
        published: 9005

As it is an extended format for ports configuration, it should work properly.

Thanks in advance

It seems that both target and published are enforced as integer type in the long syntax

I think this is not the desired behaviour if you are connecting to some services via SSH tunnels. For example, if you want to have your MySQL or MongoDB server on 127.0.0.1 and connect via SSH Tunnel, with Docker Swarm you must expose the database port on 0.0.0.0 or create a custom database container with SSH running inside (and both options are very insecure).

There are many database clients using SSH tunnels, like SQL Workbench or Robomongo that can't be used due to this limitation (specific interface binding).

We have the same problem in our company as @fer2d2, trying to connect Mongobooster with a docker swarm via ssh tunnel. The only solution we found was opening 27017 port and protecting the database with user and password.

Any news?

+1

+1

Another use case for allowing ip_address:port pair for long form port mapping is for anycast addresses or any other addresses that may be associated to loopback. These would be similar to a 127.0.0.1 address in that they are only visible on the loopback network. A service restricted to nodes with this property may wish to expose a port only on an anycast address in order to avoid port collisions while avoiding iptables rules for port translation.

Can it possibly be an option when you specify:

placement:
        constraints:
          - node.id ==

Cheers

+1

+1

+1

for myself I solved this problem so:

iptables -I DOCKER-USER -i eth0 -j DROP
iptables -I DOCKER-USER -m state --state RELATED,ESTABLISHED -j ACCEPT

The docker does not touch these rules. just adds your own
-A DOCKER-USER -j RETURN
As a result, although the port listens on 0.0.0.0 but is not accessible from the external interface eth0

This behaviour is in violation of “secure by default”, and putting a note in the docs will not be good enough. For now it should cause an error.

It is also related to the mode: ingress/host (these two issues seem to be confused in the discussion). There is nothing about ingress mode that should stop the service being bound to local addresses on all nodes, but not to external addresses. Therefore 127.x.x.x should be allowed. (in non-swarm mode (using docker run) I bind to 127.0.0.2:80 and 127.0.0.3:80 etc. To test several servers locally in development. ).

The other issue is that ingress mode is the default. This is unexpected, and also leads to a security problem. I just tried to start a service on a node constrained to be on a private part of the network, with a port bound to 127.0.0.3:80. It was then also bound to the public interface of the public node. (That is silently ignoring the IP-address, and silently using ingress mode, and boom my data is public).

Usecases

• Use cases that have effected me (definitely real)
  
  
  
  • 1 Bind to port of specific node, because port is in use on other nodes. You can use host mode for this, but the default is a surprise.
  
  
  • 2 Bind to port of specific node, because other nodes have public interfaces. You can use host mode for this, but default is in violation of “secure by default”.
  
  
  • 3 Bind locally because, you don't want it visible, to other hosts, default is in violation of “secure by default”
  
  
  • 4 Bind to 127.0.0.3, because your development machine has a lot of stuff on it, and 127.0.0.1 is in use, by this port. And use /etc/hosts, so that each domain-name is sent to a different container. This works with docker run, but not with compose.
  
  
• Other usecases
  
  
  
  • Bind to specific interface e.g. 192.168.0.x, because this is an internal network. default is in violation of “secure by default”
  
  
  • Bind to specific specific node, but not constrain the service to run on this node. This is a similar use case to 1 or 2, but with out use of constraints. Traffic would be routed through the swarm.
  
  

So in summary

• Ignoring IP address and binding to 0.0.0.0, and default ingress mode, are both in violation of “secure by default”. Errors should be issued if IP-Address specified, and docs updated. Error should be issued if mode not specified (no default), and docs updated. (this fixes the mode problem, and stops surprise security problems.)
• Support can then be added for IP addresses in host mode.
• Support for IP addresses in ingress mode, that is limited to local addresses 127.x.x.x, could be added. (different local addresses e.g. 127.0.0.2 and 127.0.0.3 should be treated as different (just passed to OS)).

Allow bind to local address is useful for constrained nodes. Allow bind to particular address, would work for constrained node, or routed through the swarm to one of the addresses on one of the nodes (may be ingress mode only). This routing is done already

@richard-delorenzi Moby does not even accept a host IP currently. So outside of the feature request, this sounds like a client side issue...specifically how the compose yaml is translated in The Docker CLI.

The way ingress works is pretty well documented, but agree this is poor behavior on the CLI.

+1

+1

+1

I have a workaround of sorts that I use. I run standalone containers and connect them to a network named 'core', which is used by all of our back-end services (mongo, elasticsearch, influxdb etc) that are running inside a swarm.

I can't see how to do this in a compose file, so we're just running standalone containers like so:

docker run --name kibana --rm -d -v /var/lib/kibana:/usr/share/kibana/config -p 127.0.0.1:5601:5601 --network core docker.elastic.co/kibana/kibana:6.1.2

docker run --name chronograf --rm -d -v /var/lib/chronograf:/var/lib/chronograf -p 127.0.0.1:8888:8888 --network core chronograf:1.4 chronograf --influxdb-url=http://influxdb:8086

After starting these, docker ps shows the new containers as being bound to 127.0.0.1. Amen. I can then tunnel to the docker host from my local workstation for secure access, like so:

ssh -i my_ssh_key.pem [email protected]  -L 8888:localhost:8888  -L 5601:localhost:5601 -N

From my browser, I can then connect to http://localhost:8888 or http://localhost:5601

Works for me.

In the event that a UNIX socket can replace a 127.0.0.1 TCP/IP socket, a possible workaround that I have implemented for fluent-bit is available here

Maybe adding another option to mode could help. Something like local in addition to host and ingress.

Please remove the wording "Usable security: Moby provides secure defaults without compromising usability." on the moby readme file. This is definitely false advertisement, see comment of @richard-delorenzi.

Services do not publish ports by default, so won't be accessible unless you specify that they should publish a port. Binding to a specific IP-address is not supported currently; if your service should not be accessible, don't publish ports, and connect to the service using an internal (overlay) network.

Adding support for binding to an IP-address is discussed in https://github.com/moby/moby/issues/26696, but not trivial to implement (taking into account non-"localhost" IP-addresses)

A warning was added when deploying a stack;

docker stack deploy -c- test <<'EOF'
version: '3'
services:
  web:
    image: nginx:alpine
    ports:
      - "127.0.0.1:8080:80"
EOF

WARN[0000] ignoring IP-address (127.0.0.1:8080:80/tcp) service will listen on '0.0.0.0' 
Creating network test_default
Creating service test_web

And when attempting to deploy a service with an IP-address specified, it will fail to deploy with an error;

docker service create -p 127.0.0.1:9090:80 nginx:alpine
invalid argument "127.0.0.1:9090:80" for "-p, --publish" flag: hostip is not supported
See 'docker service create --help'.

@dalu if your system is exposed to the internet and you told Docker to expose a service on the cluster, I'm not sure why the expectation would be something else.

Definitely this compose format that's strattles dev and actual deployments has some gross compromises.

@cpuguy83

if your system is exposed to the internet and you told Docker to expose a service on the cluster, I'm not sure why the expectation would be something else.

Nope. Why should it be accessible public if someone bind it to non public ip like 127.0.0.1 or 10.0.0.0? Actually, that is the correct answer:

Binding to a specific IP-address is not supported currently

@dalu

But it should be accessible, only not publicly. And that's the whole deal here.
You're insecure by default and evading a fix with semantics.
The issue has been open for almost 2 years now without a proper resolution.

I'm transitioning from swarm to kubernetes because swarm isn't usable. I'm totally happy with this decision even this transition is very costly.

@Bessonv It litterally tells you "I'm ignoring this"

The problem is the compose format is designed for dev envs and has been pushed to support cluster deploys. "docker stack" should just error out, but then people want to be able to use one compose file to rule them all and so there's this mess.

@cpuguy83
I'm not sure that I'm comfortable with this description. At the end compose format is just description of desired state. To have some differences between one-machine (compose) and cluster (swarm) is totally OK. From my point of view, there is no point to support compose at all. Especially because activate swarm mode is so easy. But this requires fixing swarm.

The problem isn't in swarm at all and 100% in the compose format + the implementation in the docker cli.
Note that stacks are 100% a client side implementation currently.

We've found that inside the stack you don't have to explicitly expose any ports for internal services such as database, redis etc. Just omitting the ports configuration of internal service and referencing by name works fine.

Example db service inside of stack

services:
  db:
    image: postgres:11-alpine
  networks:
    - backend

... can be consumed by Django app service by default port like this:

DATABASES = {
    'default': env.db(default='postgres://user:pass@db:5432/catalog'),
}

So in this case when you explicitly expose only public services this looks like secure-by-default

The problem isn't in swarm at all and 100% in the compose format + the implementation in the docker cli.
Note that stacks are 100% a client side implementation currently.

Whatever: I stopped using stack (because of this problem), and no longer care. Blame the library, blame docker, blame my cat.

I have not seen this problem when using docker directly, or when using compose.

Looks like this approach can help (should be executed on every node in swarm):

1. leave swarm
2. remove network docker_gwbridge
3. recreate network docker_gwbridge with additional option com.docker.network.bridge.host_binding_ipv4=IP
4. join swarm back
   Works for ports published in mode "host". Without mode "host" ingress network is used with other driver and scope "swarm".

Horrible solution:

$ mv /usr/bin/docker-proxy /usr/bin/docker-proxy-original
$ cat << 'EOF' > /usr/bin/docker-proxy
#!/bin/sh
exec /usr/bin/docker-proxy-original `echo $* | sed s/0.0.0.0/127.0.0.1/g`
EOF
$ chmod 755 /usr/bin/docker-proxy
$ service docker restart

@jsmouret I can't even find docker-proxy on latest docker release. Is it some legacy ? Or the name is different?

Looks like it depends...

$ apt-file search docker-proxy
docker-ce: /usr/bin/docker-proxy
docker.io: /usr/sbin/docker-proxy

This behavior should be documented somehow in the documentation.
Currently, it just ignores the host from the short port mapping. And silently does not work.

Another weird thing is you cannot set the host in the long syntax schema.

This behavior should be documented somehow in the documentation.

I agree; I thought it was mentioned somewhere on that page, but can't find it; feel free to open an issue in the docs repository; https://github.com/docker/docker.github.io/issues

Currently, it just ignores the host from the short port mapping. And silently does not work.

what version of docker are you using? it should print a warning (when using docker stack deploy), or an _error_ (when using docker service create); see https://github.com/moby/moby/issues/32299#issuecomment-472793444

what version of docker are you using? it should print a warning (when using docker stack deploy), or an error (when using docker service create);

Ugh, looks like it's my fault. It really does it when I tried to deploy a stack from the console.
Previously I did it via portainer UI and it didn't show any errors or warnings.

Really disappointed that almost two years some of the Docker devs ignore one valid and very useful use-case when that functionality is must have: when you want to bind containarized relational database as swarm service to local interface in order to access it securily throught SSH tunnel. Currently that scenario is impossible to do.

A workable, clean solution is running an SSH server in a second container which is connected to the same docker network as your database. The SSH port can then be published on the host (to a port different from 22 of course), so you can forward through the SSH container to your database.

@nartamonov I don't see how this could be done securely from ingress unless the protocol itself is secure.
The way to access it securely would be through an encrypted data plane (--opt encrypted for overly networks) and spin up a container with whatever tools you need attached to that network.

This probably has other unrelated side effects but setting "iptables": false in /etc/docker/daemon.json does the trick as a workaround as well. A less drastical solution is adding just the custom rule like @helldweller suggested.

Either way I would love to see some more support for this after 3 years.

Looks like this approach can help (should be executed on every node in swarm):

1. leave swarm

2. remove network docker_gwbridge

3. recreate network docker_gwbridge with additional option com.docker.network.bridge.host_binding_ipv4=IP

4. join swarm back
   Works for ports published in mode "host". Without mode "host" ingress network is used with other driver and scope "swarm".

@ienovytskyi
If I'm not mistaken, this makes all published ports bind to a given default IP address? So to be clear, this is not a usable workaround if you only want to constraint the bound interface for some ports of some services.

I would like to report my workaround.

Use case:
Some services in swarm need to be listening on all interfaces, or at least on the public interface - this container in my example is a Reverse Proxy
In those swarm nodes there is also a database instance on every node, those use a swarm network defined as:

docker network create --scope swarm NETWORK_NAME --attachable -d overlay

Web services that need a database connections need to be joined to that NETWORK_NAME of course

For admin purposes, sometimes, it's necessary to connect directly to the database

Solution:
Only services that need to be exposed on all the networks (reverse proxies in my example) can have ports: ['SOMEPORT:ANOTHERPORT'] in their services definition

All the others services, needs to have a paired docker non-swarm container on the host.
That non-swarm container will bridge the port present on NETWORK_NAME/nodeXYZ:port to localhost

Example with mongodb:

docker run --rm -it --net=NETWORK_NAME -d --name expose-mongo -p 127.0.0.1:27017:47017 alpine/socat tcp-listen:47017,fork,reuseaddr tcp-connect:mongo01:27017

Downside: there should be a non-swarm container for every swarm node, so with many nodes is really boring unless adopting ansible/heavy scripting

My workaround for the "if you are connecting to some services via SSH tunnels" problem that @fer2d2 mentioned was to add an ssh service with a Dockerfile like:

FROM alpine

RUN apk add --no-cache openssh
RUN mkdir ~/.ssh
RUN ssh-keygen -A
RUN echo "root:root" | chpasswd
RUN echo 'PasswordAuthentication no' >> /etc/ssh/sshd_config
RUN echo 'Port 22' >> /etc/ssh/sshd_config
RUN echo -e " \
Match User root \n\
  AllowTcpForwarding yes\n\
  X11Forwarding no\n\
  AllowAgentForwarding no\n\
  ForceCommand /bin/false\n\
" >> /etc/ssh/sshd_config

EXPOSE 22
CMD /usr/sbin/sshd -D -e "$@"

Then in the docker-compose.yml:

...
  db:
    image: mysql:5.6
    environment:
      MYSQL_ROOT_PASSWORD: ${MYSQL_ROOT_PASSWORD:?err}
      MYSQL_ROOT_HOST: '%'
    volumes:
      - "./mysql:/var/lib/mysql"
    deploy:
      placement:
        constraints: [node.role == manager]

  sshd:
    image: maxisme/sshd:latest
    volumes:
      - "~/.ssh:/root/.ssh"
    ports:
      - "2223:22"
    deploy:
      placement:
        constraints: [node.role == manager]

which allows me to add my authorized_keys to the ~/.ssh folder and then ssh proxy jump through port 2223 to my database using the db hostname.

A workable, clean solution is running an SSH server in a second container which is connected to the same docker network as your database. The SSH port can then be published on the host (to a port different from 22 of course), so you can forward through the SSH container to your database.

valid

another example why this feature is important.
I have a server with plesk installed, plesk already has its configurations but i can add another config just to point to a docker swarm service. This plesk server is a swarm node.
I would like to use plesk to proxy_pass to a port. This port should be published because the container is in the overlay network but it needs an external port to communicate with the world.

So the proxypass should point to a local interface like 127.0.0.1: someport
and the container in the swarm should publish the port only to localhost.

In this way the container port is accessible only by the proxypass and not from the world directly

I like your workaround @maxisme, but how do you manage the authorized_keys ownership? On OS X it works for me (the mount belongs to root) but on a production Linux machine I'm getting:

Authentication refused: bad ownership or modes for file /root/.ssh/authorized_keys
Connection closed by authenticating user root 85.145.195.174 port 60535 [preauth]

The volume belongs to UID of the host user, which is not root and SSHD refuses to work with it. A workaround on top of your workaround 😬 is to use configs, like this:

services:
  sshd:
    image: [...]/sshd:${version}
    configs:
      # FIXME: It would be much better to use a bind volume for this, as it
      # would always be in sync with the host configuration. So revoking a key
      # in the host machine would automatically revoke it in the container. But
      # I can't figure out how to give the volume right ownership. It keeps UID
      # from the host which doesn't align with the container user.
      - source: authorized_keys
        target: /root/.ssh/authorized_keys
        mode: 0600

configs:
  authorized_keys:
    file: ~/.ssh/authorized_keys

I understand that due to the fact that you don't know what host a container will be deployed to you cannot tell the service to bind to a specific host ip address.

However often hosts have e.g. north and south bound interfaces. You might want the swarm ports to bind only to the northbound interfaces on all swarm hosts.

If the interface names of all the interfaces you want a service to bind to are the same (e.g. eth0), it might be an idea to offer an option to specify an interfacename to bind swarm ports to (in service ports section).

    nginx:
      image: nvbeta/swarm_nginx
      networks:
        - demonet1
      ports:
        - "eth0:8088:80"

When eth0 is not available on a swarm node the specified port won't be bound to any interface.

@tad-lispy You should be able to change uid and gid of the container user to be the same as the volume owner on the host.
The linuxserver image supports this by setting environment variables (see https://hub.docker.com/r/linuxserver/openssh-server, User / Group Identifiers),