Nomad: Interactive jobs for docker driver

Created on 28 Dec 2017 · 3Comments · Source: hashicorp/nomad

Hi all, hope this is the right place for this sort of thing. It doesn't seem that nomad's docker driver supports interactive jobs. Our use case is to launch operator containers (we're a rails shop, so it's particularly useful to be able to launch a container with rails console in pre-prod environments). We're essentially searching for a nomad equivalent of docker run --rm -it.

Is this something being considered for future releases? I can think of some pretty ugly workarounds, but we'd prefer to use something first-party if that were an option. :)

themdrivedocker typquestion

Source

byronwolfman

👍2

Most helpful comment

While we may someday add remote execution to Nomad directly, currently there's no way to attach to an interactive session on the remote client when executing nomad run some.job.

In the meantime you will have to use docker exec -it ${CONTAINER_NAME} /bin/sh on the client running the job. The container is named ${TASK_NAME}-${ALLOC_ID}, so you may be able to script it using bash+curl+jq:

# Example alloc
taskname="redis"
allocid="1f54361f-e371-efd0-6f54-803614c719f8"

nodeid="$(curl -s localhost:4646/v1/allocation/$allocid | jq -r .NodeID)"
nodeip=$(curl -s localhost:4646/v1/node/$nodeid | jq -r '.Attributes."unique.network.ip-address"')
ssh $nodeip docker exec -it "$taskname-$allocid" /bin/sh

Closing for now, but feel free to comment and reopen if that doesn't work for you or I got something wrong!

schmichael on 3 Jan 2018

👍2

All 3 comments

While we may someday add remote execution to Nomad directly, currently there's no way to attach to an interactive session on the remote client when executing nomad run some.job.

# Example alloc
taskname="redis"
allocid="1f54361f-e371-efd0-6f54-803614c719f8"

nodeid="$(curl -s localhost:4646/v1/allocation/$allocid | jq -r .NodeID)"
nodeip=$(curl -s localhost:4646/v1/node/$nodeid | jq -r '.Attributes."unique.network.ip-address"')
ssh $nodeip docker exec -it "$taskname-$allocid" /bin/sh

Closing for now, but feel free to comment and reopen if that doesn't work for you or I got something wrong!

schmichael on 3 Jan 2018

👍2

Thanks! That's a helpful enough workaround in the meantime. 👍

byronwolfman on 3 Jan 2018

As another alternative Weave Scope would be a good candidate. You would be able to exec into a container from a web ui, plus it shows majority of network connections between your container close to real time. It runs on every host in your cluster.

It can be run without Weave Net, but you would need to provide it locations of all other hosts (it might cluster if everyone connects to one host, but I have not checked that). Guys from WeaveWorks provide a bash script that would start it for you, but it can be run without the script as Nomad job or from Ansible for example:

---
- name: Start Weave.Scope
  docker_container:
    name: weavescope
    image: weaveworks/scope
    privileged: yes
    network_mode: host
    pid_mode: host
    entrypoint: "/home/weave/entrypoint.sh --probe.docker=true {{ cluster_with | join(' ') }}"
    published_ports:
      - 4040:4040
    restart_policy: always
    volumes:
      - /var/run/scope/plugins:/var/run/scope/plugins
      - /sys/kernel/debug:/sys/kernel/debug
      - /var/run/docker.sock:/var/run/docker.sock