这确实需要更多信息，并需要重现步骤

dockerd[101922]: time="2018-01-04T15:54:07.406980654Z" level=debug msg="Calling POST /v1.35/containers/4c2b5e7f466c/stop"

• 上面的消息仅显示调用已停止容器； 之后还有什么消息吗？
• 码头工人如何设置？ 您在运行docker-in-docker吗？
• 您是否已验证容器仍在运行？ 容器的docker inspect显示什么？ 输出中是否有PID ？ 而且该过程仍在运行吗？ （主机上的ps auxf ）
• 如果没有詹金斯，您能否重现问题？ 您能提供复制的确切步骤吗？

不再记录任何消息。

同时，在容器进入此状态后，我进行了更多测试：

• 我无法停止容器
• 我不能docker exec扑面而来。
• 我可以启动和停止其他容器

要退出此状态，我必须：

• 服务泊坞窗停止
• 杀死容器进程，（如果不是，则docker无法启动）
• 服务docker启动

我认为我曾经在詹金斯之外重现过这一点，但我认为这是另一个问题。

与Jenkins一样，更容易重现此代码，我将等待下一次执行docker inspect container。

设置（在CENTOS vm中运行）：

• 使用docker-compose up启动具有特定图像的容器
• 使用容器中启动的应用程序执行一些测试
• 使用docker-compose down停止容器

这些步骤是针对每个测试完成的，而我正在同时运行测试。

很抱歉没有提供更多信息，但这是我到目前为止可以收集到的信息。

我在_different docker version_上遇到了类似的问题_now_ 。 我无法停止创建的任何容器。

并记录所有容器。

```Jan 08 16:53:10 dockerd [7012]：time =“ 2018-01-08T16：53：10.983935134Z” level = debug msg =“正在调用POST /v1.34/containers/9cdc36c44340/stop”
1月8日16:53:10 dockerd [7012]：time =“ 2018-01-08T16：53：10.984024605Z” level = debug msg =“ Send kill signal 15 to container 9cdc36c44340cd23a5cbfb884c1fab4d47b173552dd992f392d4398603b46a94”
1月8日16:53:12 dockerd [7012]：time =“ 2018-01-08T16：53：12.985034572Z” level = info msg =“容器在向该进程发送信号15后无法停止，强制杀死”
1月8日16:53:12 dockerd [7012]：time =“ 2018-01-08T16：53：12.985087603Z” level = debug msg =“将终止信号9发送到容器9cdc36c44340cd23a5cbfb884c1fab4d47b173552dd992f392d4398603b46a94”
1月08日16:53:12 dockerd [7012]：time =“ 2018-01-08T16：53：12.986759908Z” level = debug msg =“ FIXME：得到了一个错误与任何预期类型都不匹配的API！找不到\ ngithub.com/docker/docker/vendor/github.com/containerd/containerd/errdefs.init \ n \ t / go / src / github.com / docker / docker / vendor / github.com / containerd / containerd /errdefs/errors.go:25\ngithub.com/docker/docker/vendor/github.com/containerd/containerd/content.init\n\t/go/src/github.com/docker/docker/vendor/github .com / containerd / containerd / content / helpers.go：141 \ ngithub.com/docker/docker/vendor/github.com/containerd/containerd.init \ n \ t / go / src / github.com / docker / docker /vendor/github.com/containerd/containerd/task_opts.go:78\ngithub.com/docker/docker/container.init\n\t/go/src/github.com/docker/docker/container/view.go ：496 \ ngithub.com/docker/docker/builder.init \ n \ t / go / src / github.com / docker / docker / builder / builder.go：108 \ ngithub.com/docker/docker/api/server /backend/build.init\n\t/go/src/github.com/docker/docker/api/server/backend/build/ta g.go：85 \ nmain.init \ n \ t / go / src / github.com / docker / docker / cmd / dockerd / service_unsupported.go：15 \ nruntime.main \ n \ t / usr / local / go / src / runtime / proc.go：173 \ nruntime.goexit \ n \ t / usr / local / go / src / runtime / asm_amd64.s：2197“ error_type =” errors.fundamental“ module = api1月8日16:53:12 dockerd [7012]：time =“ 2018-01-08T16：53：12.986856140Z” level = error msg =“ POST的处理程序/v1.34/containers/9cdc36c44340/stop返回的错误：无法停止容器：9cdc36c44340：无法终止容器9cdc36c44340cd23a5cbfb884c1fab4d47b173552dd992f392d4398603b46a94：进程9cdc36c44340cd23a5cbfb884c1fab4d47b173552dd992f392d4398603b46a94“未找到：1月08日16:53:12 dockerd [7012]：time =“ 2018-01-08T16：53：12.987051906Z” level = debug msg =“ FIXME：得到了一个错误与任何预期类型都不匹配的API！找不到\ ngithub.com/docker/docker/vendor/github.com/containerd/containerd/errdefs.init \ n \ t / go / src / github.com / docker / docker / vendor / github.com / containerd / containerd /errdefs/errors.go:25\ngithub.com/docker/docker/vendor/github.com/containerd/containerd/content.init\n\t/go/src/github.com/docker/docker/vendor/github .com / containerd / containerd / content / helpers.go：141 \ ngithub.com/docker/docker/vendor/github.com/containerd/containerd.init \ n \ t / go / src / github.com / docker / docker /vendor/github.com/containerd/containerd/task_opts.go:78\ngithub.com/docker/docker/container.init\n\t/go/src/github.com/docker/docker/container/view.go ：496 \ ngithub.com/docker/docker/builder.init \ n \ t / go / src / github.com / docker / docker / builder / builder.go：108 \ ngithub.com/docker/docker/api/server /backend/build.init\n\t/go/src/github.com/docker/docker/api/server/backend/build/ta

**docker-info :** 
```Containers: 6
 Running: 6
 Paused: 0
 Stopped: 0
Images: 61
Server Version: 17.11.0-ce
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 992280e8e265f491f7a624ab82f3e238be086e49
runc version: 0351df1c5a66838d0c392b4ac4cf9450de844e2d
init version: 949e6fa
Security Options:
 apparmor
 seccomp
  Profile: default
Kernel Version: 4.10.0-42-generic
Operating System: Ubuntu 16.04.3 LTS
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 31GiB
Name: Laptop-749
ID: WZVE:HR5Q:3GYH:WNS6:FJCQ:TGHD:UMU5:PPWM:7Z77:QSBV:G2SW:HI77
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): true
 File Descriptors: 26
 Goroutines: 48
 System Time: 2018-01-08T16:58:47.457072503Z
 EventsListeners: 0
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

WARNING: No swap limit support

码头工人版本：

客户
版本：17.11.0-ce
API版本：1.34
Go版本：go1.8.3
Git提交：1caf76c
建成：2017年11月20日星期一18:37:39
操作系统/ Arch：linux / amd64

服务器：
版本：17.11.0-ce
API版本：1.34（最低版本1.12）
Go版本：go1.8.3
Git提交：1caf76c
建成：2017年11月20日星期一18:36:09
操作系统/ Arch：linux / amd64
实验性：错误

**docker inspect** : 
```[
    {
        "Id": "9cdc36c44340cd23a5cbfb884c1fab4d47b173552dd992f392d4398603b46a94",
        "Created": "2018-01-08T16:32:30.716158282Z",
        "Path": "/opt/entrypoint.sh",
        "Args": [],
        "State": {
            "Status": "running",
            "Running": true,
            "Paused": false,
            "Restarting": false,
            "OOMKilled": false,
            "Dead": false,
            "Pid": 477,
            "ExitCode": 0,
            "Error": "",
            "StartedAt": "2018-01-08T16:32:31.370353796Z",
            "FinishedAt": "0001-01-01T00:00:00Z",
            "Health": {
                "Status": "healthy",
                "FailingStreak": 0,
                "Log": [
                    {
                        "Start": "2018-01-08T16:40:52.760255527Z",
                        "End": "2018-01-08T16:40:52.814916997Z",
                        "ExitCode": 0,
                        "Output": ""
                    },
                    {
                        "Start": "2018-01-08T16:41:12.821209911Z",
                        "End": "2018-01-08T16:41:12.872327217Z",
                        "ExitCode": 0,
                        "Output": ""
                    },
                    {
                        "Start": "2018-01-08T16:41:32.879017542Z",
                        "End": "2018-01-08T16:41:32.932394782Z",
                        "ExitCode": 0,
                        "Output": ""
                    },
                    {
                        "Start": "2018-01-08T16:41:52.938598813Z",
                        "End": "2018-01-08T16:41:52.993106466Z",
                        "ExitCode": 0,
                        "Output": ""
                    },
                    {
                        "Start": "2018-01-08T16:42:12.998820005Z",
                        "End": "2018-01-08T16:42:13.056301771Z",
                        "ExitCode": 0,
                        "Output": ""
                    }
                ]
            }
        },
        "Image": "sha256:71843cc0ac81d2a365553dd5b69f6643dab212fd8b45d498c6a92614352ed75f",
        "ResolvConfPath": "/var/lib/docker/containers/9cdc36c44340cd23a5cbfb884c1fab4d47b173552dd992f392d4398603b46a94/resolv.conf",
        "HostnamePath": "/var/lib/docker/containers/9cdc36c44340cd23a5cbfb884c1fab4d47b173552dd992f392d4398603b46a94/hostname",
        "HostsPath": "/var/lib/docker/containers/9cdc36c44340cd23a5cbfb884c1fab4d47b173552dd992f392d4398603b46a94/hosts",
        "LogPath": "/var/lib/docker/containers/9cdc36c44340cd23a5cbfb884c1fab4d47b173552dd992f392d4398603b46a94/9cdc36c44340cd23a5cbfb884c1fab4d47b173552dd992f392d4398603b46a94-json.log",
        "Name": "/kegfngsmzx_component_1",
        "RestartCount": 0,
        "Driver": "overlay2",
        "Platform": "linux",
        "MountLabel": "",
        "ProcessLabel": "",
        "AppArmorProfile": "docker-default",
        "ExecIDs": null,
        "HostConfig": {
            "Binds": [
                "/dev/null:/tmp/conf/4:rw",
                "/home/joao.suzana/gitprojects/superComponent/docker/configurations/default/component/common:/tmp/conf/1:rw",
                "/home/joao.suzana/gitprojects/superComponent/docker/configurations/default/component/basic:/tmp/conf/0:rw",
                "/home/joao.suzana/gitprojects/superComponent/docker/configurations/system-tests/component:/tmp/conf/3:rw",
                "/home/joao.suzana/gitprojects/superComponent/docker/configurations/custom/component:/tmp/conf/2:rw"
            ],
            "ContainerIDFile": "",
            "LogConfig": {
                "Type": "json-file",
                "Config": {}
            },
            "NetworkMode": "kegfngsmzx_default",
            "PortBindings": {
                "1099/tcp": [
                    {
                        "HostIp": "",
                        "HostPort": ""
                    }
                ],
                "7000/tcp": [
                    {
                        "HostIp": "",
                        "HostPort": ""
                    }
                ],
                "8080/tcp": [
                    {
                        "HostIp": "",
                        "HostPort": ""
                    }
                ]
            },
            "RestartPolicy": {
                "Name": "",
                "MaximumRetryCount": 0
            },
            "AutoRemove": false,
            "VolumeDriver": "",
            "VolumesFrom": [],
            "CapAdd": null,
            "CapDrop": null,
            "Dns": [],
            "DnsOptions": [],
            "DnsSearch": [],
            "ExtraHosts": null,
            "GroupAdd": null,
            "IpcMode": "shareable",
            "Cgroup": "",
            "Links": null,
            "OomScoreAdj": 0,
            "PidMode": "",
            "Privileged": false,
            "PublishAllPorts": false,
            "ReadonlyRootfs": false,
            "SecurityOpt": null,
            "UTSMode": "",
            "UsernsMode": "",
            "ShmSize": 67108864,
            "Runtime": "runc",
            "ConsoleSize": [
                0,
                0
            ],
            "Isolation": "",
            "CpuShares": 0,
            "Memory": 0,
            "NanoCpus": 0,
            "CgroupParent": "",
            "BlkioWeight": 0,
            "BlkioWeightDevice": null,
            "BlkioDeviceReadBps": null,
            "BlkioDeviceWriteBps": null,
            "BlkioDeviceReadIOps": null,
            "BlkioDeviceWriteIOps": null,
            "CpuPeriod": 0,
            "CpuQuota": 0,
            "CpuRealtimePeriod": 0,
            "CpuRealtimeRuntime": 0,
            "CpusetCpus": "",
            "CpusetMems": "",
            "Devices": null,
            "DeviceCgroupRules": null,
            "DiskQuota": 0,
            "KernelMemory": 0,
            "MemoryReservation": 0,
            "MemorySwap": 0,
            "MemorySwappiness": null,
            "OomKillDisable": false,
            "PidsLimit": 0,
            "Ulimits": null,
            "CpuCount": 0,
            "CpuPercent": 0,
            "IOMaximumIOps": 0,
            "IOMaximumBandwidth": 0
        },
        "GraphDriver": {
            "Data": {
                "LowerDir": "/var/lib/docker/overlay2/5fbfde1f36fe0da7cee8ed92b728f7b92ddd4a1b8d9aeafe44b7e8cde581aeb3-init/diff:/var/lib/docker/overlay2/a9c4a86986bf84eff4d3156580e986daed91c7a37d937c5e4f608cd90b78f50a/diff:/var/lib/docker/overlay2/566bb33f0a3140bdb3726e3581bc703557f729010d2fb5b76ba21ac04157e5eb/diff:/var/lib/docker/overlay2/92302187d5633c0e6f3577edf93e2f1fbc133ccfcd11c6ce4a2b0fd06eb33db4/diff:/var/lib/docker/overlay2/3ac16dcca78ec2202d9af5e2e1ca50053612b75247d685c66418516aa7a1f91e/diff:/var/lib/docker/overlay2/3c2bef86bfac98dace20fb5ad4461601d444797454a5561bb543e4478d3aed25/diff:/var/lib/docker/overlay2/82de5471b51e7a55f8d9ff61983b36e9302b2fc7f4ba3fcc6ce5bde9f426ac9b/diff:/var/lib/docker/overlay2/7103da23a70519f91ae53950b6da99797d75104815ff43a1662efc92a933dc45/diff:/var/lib/docker/overlay2/70d522784351b087ee139f429dd041e1966308365e222f9022ab33f1f6da5089/diff:/var/lib/docker/overlay2/05d68822eebc4564c7e4597ee7c3d2bece406703e2e042bdf2ec35061a178f3a/diff:/var/lib/docker/overlay2/cc4fbcefd6fc474463d00d55d708988fc68f6eca5534675992e157743cb04af7/diff:/var/lib/docker/overlay2/50a363caa96c54de6cf17bfa477e384694f0fdf15a81c27cb92b830c0a8782b1/diff:/var/lib/docker/overlay2/ee1dadb2c4a98b37896eeb4e97f0715d97485bd10ef2b70d3b279d7fb93a4b18/diff:/var/lib/docker/overlay2/a66b6a45869ab5484cc04259ee7e11d32526a1fa1c91748f71754b57a87b69d9/diff:/var/lib/docker/overlay2/58472f6337dd2f95a5bda690e630fc6ddf4f661b6e965cfa798c666cde72457a/diff:/var/lib/docker/overlay2/22657f15e2d1411269f3201e63705babaaa7a04275f6c91ca5df4dc167abd93f/diff:/var/lib/docker/overlay2/5483cd1fad2a005e68e2656c5fcee54b8844576743288c06e49f40f6a4381a63/diff:/var/lib/docker/overlay2/ba02a2666cd21a254805404d1757f8ed90e28089e4a924e15a524c1e09265d0a/diff:/var/lib/docker/overlay2/07359ba2f66ba314629b1a6df441a7b96470e5d55ec22b88a48cc7c93b34f515/diff:/var/lib/docker/overlay2/99ecef114a5db24e123e4f5d9a8a01c3a79fa6aaed1af1095669f374a689294d/diff:/var/lib/docker/overlay2/7cfa73084c807c05112368f9c60627622b807b5ad932ace14541994f95209329/diff:/var/lib/docker/overlay2/b8e4cd0ea2811b61210129cc97ef4d10489bcb61b3b1dbe64d5a7af65bc284e2/diff:/var/lib/docker/overlay2/5cb7c00c701b24ca232c773eff803b0ca26a4bb137a5960920f5f3e9c96cfe7b/diff:/var/lib/docker/overlay2/6e722e736fb0acf96c2bbd2b29cd10e79955fe4b5fd8bf862a17ffa241b68a1b/diff:/var/lib/docker/overlay2/160835aace0cb1e2f4b9360934188b99ca9a65c74ee8d100f613275024e9d811/diff:/var/lib/docker/overlay2/5c7ba1cf63c83cda117ef0eca2bfd65d9bd44669e0e80933e351620bce546354/diff:/var/lib/docker/overlay2/c58b587a8318b57dc1f39c2aa2df68fa86295280fc007650a16008d05685b356/diff",
                "MergedDir": "/var/lib/docker/overlay2/5fbfde1f36fe0da7cee8ed92b728f7b92ddd4a1b8d9aeafe44b7e8cde581aeb3/merged",
                "UpperDir": "/var/lib/docker/overlay2/5fbfde1f36fe0da7cee8ed92b728f7b92ddd4a1b8d9aeafe44b7e8cde581aeb3/diff",
                "WorkDir": "/var/lib/docker/overlay2/5fbfde1f36fe0da7cee8ed92b728f7b92ddd4a1b8d9aeafe44b7e8cde581aeb3/work"
            },
            "Name": "overlay2"
        },
        "Mounts": [
            {
                "Type": "bind",
                "Source": "/home/joao.suzana/gitprojects/superComponent/docker/configurations/default/component/common",
                "Destination": "/tmp/conf/1",
                "Mode": "rw",
                "RW": true,
                "Propagation": "rprivate"
            },
            {
                "Type": "bind",
                "Source": "/home/joao.suzana/gitprojects/superComponent/docker/configurations/custom/component",
                "Destination": "/tmp/conf/2",
                "Mode": "rw",
                "RW": true,
                "Propagation": "rprivate"
            },
            {
                "Type": "bind",
                "Source": "/home/joao.suzana/gitprojects/superComponent/docker/configurations/system-tests/component",
                "Destination": "/tmp/conf/3",
                "Mode": "rw",
                "RW": true,
                "Propagation": "rprivate"
            },
            {
                "Type": "bind",
                "Source": "/dev/null",
                "Destination": "/tmp/conf/4",
                "Mode": "rw",
                "RW": true,
                "Propagation": "rprivate"
            },
            {
                "Type": "bind",
                "Source": "/home/joao.suzana/gitprojects/superComponent/docker/configurations/default/component/basic",
                "Destination": "/tmp/conf/0",
                "Mode": "rw",
                "RW": true,
                "Propagation": "rprivate"
            }
        ],
        "Config": {
            "Hostname": "9cdc36c44340",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "ExposedPorts": {
                "1099/tcp": {},
                "7000/tcp": {},
                "8080/tcp": {}
            },
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "LC_ALL=en_US.UTF-8",
                "JDK_RPM=jdk-8u131-linux-x64.rpm",
                "JAVA_HOME=/usr/java/jdk1.8.0_131/",
                "COMPONENT_HOME=/opt/component"
            ],
            "Cmd": [
                "/opt/entrypoint.sh"
            ],
            "Healthcheck": {
                "Test": [
                    "CMD-SHELL",
                    "grep -q \"App Service is ready.\""
                ],
                "Interval": 20000000000,
                "Retries": 30
            },
            "ArgsEscaped": true,
            "Image": "docker.privateimage.com/private:latest",
            "Volumes": {
                "/tmp/conf/0": {},
                "/tmp/conf/1": {},
                "/tmp/conf/2": {},
                "/tmp/conf/3": {},
                "/tmp/conf/4": {}
            },
            "WorkingDir": "",
            "Entrypoint": null,
            "OnBuild": null,
            "Labels": {
                "build-date": "20171128",
                "com.docker.compose.config-hash": "51a3c3781142fce6292e53a5a42dd804a41e5c6e81b02b2dab14647d5f3fe774",
                "com.docker.compose.container-number": "1",
                "com.docker.compose.oneoff": "False",
                "com.docker.compose.project": "kegfngsmzx",
                "com.docker.compose.service": "private-component",
                "com.docker.compose.version": "1.17.1",
                "com.super.component": "Super",
                "license": "GPLv2",
                "name": "CentOS Base Image",
                "vendor": "CentOS"
            }
        },
        "NetworkSettings": {
            "Bridge": "",
            "SandboxID": "4a6a1b4492dce570a42cb735915c76fab4c0e92dd712bf81ae323df8eec1d0a3",
            "HairpinMode": false,
            "LinkLocalIPv6Address": "",
            "LinkLocalIPv6PrefixLen": 0,
            "Ports": {
                "1099/tcp": [
                    {
                        "HostIp": "0.0.0.0",
                        "HostPort": "32863"
                    }
                ],
                "7000/tcp": [
                    {
                        "HostIp": "0.0.0.0",
                        "HostPort": "32862"
                    }
                ],
                "8080/tcp": [
                    {
                        "HostIp": "0.0.0.0",
                        "HostPort": "32861"
                    }
                ]
            },
            "SandboxKey": "/var/run/docker/netns/4a6a1b4492dc",
            "SecondaryIPAddresses": null,
            "SecondaryIPv6Addresses": null,
            "EndpointID": "",
            "Gateway": "",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "IPAddress": "",
            "IPPrefixLen": 0,
            "IPv6Gateway": "",
            "MacAddress": "",
            "Networks": {
                "kegfngsmzx_default": {
                    "IPAMConfig": null,
                    "Links": null,
                    "Aliases": [
                        "component",
                        "9cdc36c44340"
                    ],
                    "NetworkID": "19e6624e9254883228576ad289770611fd066ed7fc1c847eb0dd25899b240d07",
                    "EndpointID": "850780c0914d118382913f0ff287433e88c01a56d3e42fa95ce890c737027b76",
                    "Gateway": "172.18.0.1",
                    "IPAddress": "172.18.0.7",
                    "IPPrefixLen": 16,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": "02:42:ac:12:00:07",
                    "DriverOpts": null
                }
            }
        }
    }
]

@Timunas您可以尝试更新到17.12吗？

最初的问题是17.12

关于原始问题，我再次进行了复制，但我无法通过docker检查它是否仅挂在所有命令上

我没有使用docker-compose也遇到同样的问题。 我正在使用docker swarm。 不过，同样的事情，我偶尔会得到一些容器，这些容器既无法使docker swarm集群也无法停止使用docker CLI。 这导致docker swarm最终收集了超出其无法缩减规模的副本。 有时，这些副本仍可以为请求提供服务并接收流量。 删除容器的唯一方法是在受影响的节点上重新启动docker。

我对docker swarm有同样的问题。 我删除了多个docker堆栈之一，但是只删除了堆栈中的某些容器，而有些容器却在附近徘徊。 在挂起的容器上执行docker inspect或docker rm的命令只是挂在命令行上，直到我按Ctrl-c为止。 需要重新启动才能删除容器。 仅在升级到17.12.0-ce后才在17.09中没有问题（在17.12.0-ce-rc4上也有问题）。

我在Azure VM上有问题： docker info

``容器：95
跑步：83
已暂停：0
已停止：12
图片：579
服务器版本：17.12.0-ce
存储驱动程序：overlay2
支持文件系统：extfs
支持d_type：true
本机覆盖差异：true
记录驱动程序：json-file
Cgroup驱动程序：cgroupfs
外挂程式：
数量：本地
网络：网桥主机macvlan空覆盖
日志：awslogs流利的gcplogs gelf记录日志的json文件日志条目splunk syslog
群：活跃
节点ID：hy0kx44q5m9jg0lc1n5ylxkw6
是经理：是
群集ID：ordhsz694y98k3r4604ksc937
管理人员：1
节点数：1
编排：
任务历史记录保留限制：2
筏：
快照间隔：10000
要保留的旧快照数量：0
心跳跳动：1
选举价格：3
调度员：
心跳周期：5秒
CA配置：
有效期：3个月
强制旋转：0
自动锁定管理器：false
正在进行根旋转：false
节点地址：10.0.0.10
经理地址：
10.0.0.10:2377
运行时：runc
默认运行时：runc
初始化二进制文件：docker-init
容器版本：89623f28b87a6004d4b785663257362d1658a729
runc版本：b2567b37d7b75eb4cf325b77297b140ea686ce8f
初始版本：949e6fa
安全选项：
装甲
seccomp
个人资料：默认
内核版本：4.4.0-104-通用
作业系统：Ubuntu 16.04.3 LTS
OSType：Linux
架构：x86_64
处理器：8
总内存：27.47GiB
名称：build-agent-vm001
ID：S7WY： RCKF：G3P7 ：TI3H：MJ2F：UXZ3：C5DS：YQG3：OPF4：V4RS：5EQ7：AWG4
Docker根目录：/ var / lib / docker
调试模式（客户端）：false
调试模式（服务器）：false
注册表： https ：
标签：
实验性：错误
不安全的注册表：
127.0.0.0/8
启用实时还原：false

警告：不支持交换限制

I also have the same issue on Docker for Mac (Edge: 17.12): `docker info`

```Containers: 110
 Running: 65
 Paused: 0
 Stopped: 45
Images: 607
Server Version: 17.12.0-ce
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host ipvlan macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: active
 NodeID: qfzh0tqkchl2m42uhju7k3ml4
 Is Manager: true
 ClusterID: q14zy6epqkpx0w112wusdtd3u
 Managers: 1
 Nodes: 1
 Orchestration:
  Task History Retention Limit: 2
 Raft:
  Snapshot Interval: 10000
  Number of Old Snapshots to Retain: 0
  Heartbeat Tick: 1
  Election Tick: 3
 Dispatcher:
  Heartbeat Period: 5 seconds
 CA Configuration:
  Expiry Duration: 3 months
  Force Rotate: 0
 Autolock Managers: false
 Root Rotation In Progress: false
 Node Address: 192.168.65.3
 Manager Addresses:
  192.168.65.3:2377
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 89623f28b87a6004d4b785663257362d1658a729
runc version: b2567b37d7b75eb4cf325b77297b140ea686ce8f
init version: 949e6fa
Security Options:
 seccomp
  Profile: default
Kernel Version: 4.9.60-linuxkit-aufs
Operating System: Docker for Mac
OSType: linux
Architecture: x86_64
CPUs: 6
Total Memory: 5.817GiB
Name: linuxkit-025000000001
ID: DSXX:YVTO:DLFW:MN3X:MTJC:3EGK:MUYT:6JMN:C2NC:TQMW:BE44:3P6H
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): true
 File Descriptors: 260
 Goroutines: 491
 System Time: 2018-01-09T00:13:09.053688513Z
 EventsListeners: 28
HTTP Proxy: docker.for.mac.http.internal:3128
HTTPS Proxy: docker.for.mac.http.internal:3128
Registry: https://index.docker.io/v1/
Labels:
Experimental: true
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

我们还在某些命令上遇到了无响应的docker-deamon：

目前我不能

docker rmi
docker system prune -f
docker exec
docker logs

这发生在所有运行17.12的多个引擎上。

似乎与https://github.com/moby/moby/issues/35408有关

我遇到同样的错误。 但是，这并不一致。 我还没有看到一种模式，但是它确实发生了。

我正在运行Docker for Mac版本17.12.0-ce-mac46（21698）。 我不在Docker中运行Docker。

容器由docker-compose up 。

是的，我可以看到容器仍在运行，但是stop或kill只是挂起而没有执行任何操作。

10:13:13 Alexei-Workstation /Users/alexei.chekulaev/Projects/SBD-MASTER
$ docker ps
CONTAINER ID        IMAGE                     COMMAND                  CREATED             STATUS                    PORTS                                                    NAMES
f0e36d3589d3        docksal/cli:1.3-php7      "/opt/startup.sh sup…"   44 hours ago        Up 28 minutes (healthy)   22/tcp, 9000/tcp                                         sbdmaster_cli_1
b93c84c9a3a3        docksal/ssh-agent:1.0     "/run.sh ssh-agent"      44 hours ago        Up 29 minutes                                                                      docksal-ssh-agent
91ce00eb35fa        docksal/dns:1.0           "/opt/entrypoint.sh …"   44 hours ago        Up 29 minutes             192.168.64.100:53->53/udp                                docksal-dns
ae867cca0f21        docksal/vhost-proxy:1.1   "docker-entrypoint.s…"   44 hours ago        Up 29 minutes             192.168.64.100:80->80/tcp, 192.168.64.100:443->443/tcp   docksal-vhost-proxy
10:13:17 Alexei-Workstation /Users/alexei.chekulaev/Projects/SBD-MASTER
$ docker stop f0e36d3589d3
^C
10:16:03 Alexei-Workstation /Users/alexei.chekulaev/Projects/SBD-MASTER
$ docker kill f0e36d3589d3
^C
10:30:51 Alexei-Workstation /Users/alexei.chekulaev/Projects/SBD-MASTER

（您可以看到在按Ctrl-C之前经过了几分钟）

在另一个终端中，我尝试启动另一个docker-compose项目，这是我第一次在输出中看到的内容：

$ docker-compose up
rm: can't remove '/.ssh/id_rsa.pub': Stale file handle
rm: can't remove '/.ssh/authorized_keys': Stale file handle
rm: can't remove '/.ssh/id_rsa2.pub': Stale file handle
rm: can't remove '/.ssh/known_hosts': Stale file handle
rm: can't remove '/.ssh/id_test': Stale file handle
rm: can't remove '/.ssh/id_test.pub': Stale file handle
rm: can't remove '/.ssh/id_rsa2': Stale file handle
rm: can't remove '/.ssh/id_dsa': Stale file handle
rm: can't remove '/.ssh/id_boot2docker': Stale file handle
rm: can't remove '/.ssh/id_sbd.pub': Stale file handle
rm: can't remove '/.ssh/id_sbd': Stale file handle
rm: can't remove '/.ssh/id_rsa': Stale file handle
rm: can't remove '/.ssh/id_boot2docker.pub': Stale file handle
rm: can't remove '/.ssh': Directory not empty
Starting services...
Creating network "demonodb_default" with the default driver
Creating demonodb_cli_1 ... done
Creating demonodb_cli_1 ... 
Creating demonodb_web_1 ... done

另一个项目开始正常，但出现上述有关旧文件名的错误。 另一个项目的后续停止和启动没有引发任何错误，并且运行正常。

这些文件位于命名卷上。 该卷在docker-compose中安装为ro ，所以我不确定为什么会有“无法删除”消息。

重新启动Docker守护程序可以暂时解决此问题。 我忘记做docker inspect并已经重新启动了守护进程，但是我认为inspect会像stop和kill那样挂起。

更新：要注意有问题的容器上有运行状况检查。 看起来这可能是罪魁祸首。

我遇到同样的问题。 可以在不同的环境下每次重现它：
Docker for Mac版本17.12.0-ce-mac46（更新后开始挂起）
或在Arch Linux（内核4.14.14-1-ARCH）上本地使用docker时，无法使用systemctl restart docker.service重新启动docker服务，它也会挂起。 docker info

Client:
 Version:       18.01.0-ce
 API version:   1.35
 Go version:    go1.9.2
 Git commit:    03596f51b1
 Built: Sun Jan 14 23:10:39 2018
 OS/Arch:       linux/amd64
 Experimental:  false
 Orchestrator:  swarm
Server:
 Engine:
  Version:      18.01.0-ce
  API version:  1.35 (minimum version 1.12)
  Go version:   go1.9.2
  Git commit:   03596f51b1
  Built:        Sun Jan 14 23:11:14 2018
  OS/Arch:      linux/amd64
  Experimental: false

journalctl节目

dockerd[26382]: time="2018-01-25T12:39:22.289082720+03:00" level=error msg="stream copy error: reading from a closed fifo"

在18.01上也看到了这一点。 挂在容器上检查。

Client:
 Version:   18.01.0-ce
 API version:   1.35
 Go version:    go1.9.2
 Git commit:    03596f51b1
 Built: Sun Jan 14 23:10:39 2018
 OS/Arch:   linux/amd64
 Experimental:  false
 Orchestrator:  swarm

Server:
 Engine:
  Version:  18.01.0-ce
  API version:  1.35 (minimum version 1.12)
  Go version:   go1.9.2
  Git commit:   03596f51b1
  Built:    Sun Jan 14 23:11:14 2018
  OS/Arch:  linux/amd64
  Experimental: false

compose.cli.command.get_client: Docker version: Platform={'Name': ''}, Components=[{'Name': 'Engine', 'Version': '18.01.0-ce', 'Details': {'ApiVersion': '1.35', 'Arch': 'amd64', 'BuildTime': '2018-01-14T23:11:14.000000000+00:00', 'Experimental': 'false', 'GitCommit': '03596f51b1', 'GoVersion': 'go1.9.2', 'KernelVersion': '4.14.15-1-ARCH', 'MinAPIVersion': '1.12', 'Os': 'linux'}}], Version=18.01.0-ce, ApiVersion=1.35, MinAPIVersion=1.12, GitCommit=03596f51b1, GoVersion=go1.9.2, Os=linux, Arch=amd64, KernelVersion=4.14.15-1-ARCH, BuildTime=2018-01-14T23:11:14.000000000+00:00
compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=False, filters={'label': ['com.docker.compose.project=discord']})
urllib3.connectionpool._make_request: http://localhost:None "GET /v1.24/containers/json?limit=-1&all=0&size=0&trunc_cmd=0&filters=%7B%22label%22%3A+%5B%22com.docker.compose.project%3Ddiscord%22%5D%7D HTTP/1.1" 200 1762
compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 1 items)
compose.cli.verbose_proxy.proxy_callable: docker inspect_container <- ('59760b63049318f7b0bef2605e63d0fd8b13f4e134a7aea435db9eb1bdf2b389')

由于此问题在17.12（macOS和显然还有Linux）上，我们已完全停止使用17.12，并回滚到17.09。

这是一个关键的持久性问题。

不幸的是，除了大量使用docker之外，我没有找到重新创建它的方法。

我在使用17.12的多台服务器中遇到相同的问题。 正如@rfay所说，它并未在17.09发生。

检查变更日志，17.12和17.09之间的主要区别在于，自17.11起，Docker基于容器化。 因此，正如证据似乎表明这是运行时中的问题一样，也许最好沿着这条道路进行调查。

是的，这里也一样。 我坚持使用17.09，并建议所有人使用docker-compose或蜂拥而至，直到问题解决。

如果您可以从正在运行的守护程序中获取stacktrace，则将非常有帮助。
您可以点击GET /debug/pprof/goroutine?debug=2

但是，我怀疑这是在runc中发现的最近的错误，这是处理容器I / O的竞赛……很显然，此错误已经存在了很长时间。
如果是这样，我们怀疑这是内核更改所揭露的，每个人最近都在升级他们的内核以获取Spectre / meltdown补丁程序。

相关的runc补丁在这里，如果您不想等待补丁的docker版本，可以尝试以下补丁： https :

@ cpuguy83

You can get this by hitting GET /debug/pprof/goroutine?debug=2

请提供命令。 我不了解如何“点击”相对网址，以及相对网址。 我在Mac上使用Docker。 我应该打什么？

@achekulaev
假设您有docker在/var/run/docker.sock（默认值）的unix套接字上侦听：

curl --unix-socket /var/run/docker.sock http:/./debug/pprof/goroutine?debug=2

或TCP套接字

curl http://<ip>:<port>/debug/pprof/goroutine?debug=2

以下文件是在使用Docker版本17.12.0-ce，构建c97c6d6的AWS Ubuntu 16.04实例中运行的命令的输出

Moby 35933.txt

@ ay0o谢谢！
现在系统上是否有阻塞的东西？
不幸的是，我没有看到任何进行中的停止/失败，只是看起来像一堆正在运行的容器。

我在运行macOS High Sierra 10.13.3，运行docker 18.01.0-ce-mac48，频道：edge ee2282129d的MacBook Pro上记录了日志。

docker_output.log

@ AlterEgo7谢谢！ 看起来docker在系统调用中被阻止写入磁盘，甚至至少在一个地方从磁盘读取。 似乎在docker4mac中为该Docker VM分配的磁盘出了点问题。

实际上，许多受I / O绑定的系统调用阻塞了大约1分钟。

@ cpuguy83 ，这是由于

我也经常遇到这个问题。 在17.12上，它似乎主要出现在具有绑定安装卷的容器上。 在我们的情况下，这些卷是主机上的nfs份额。

@ cpuguy83 @mborejdo如果有所作为，则在我的计算机上运行的约10个容器使用docker-sync统一卷。

确实有帮助。 也许docker-sync工具在17.12之后损坏了？
这肯定可以解释阻止写入的原因，因为nfs对断开的后端不是很友好。

在运行同一组容器而不运行任何卷或任何docker-sync容器时相同。 pprof日志已附加。

docker_output_no_volumes.log

@ cpuguy83 ，我也看到使用swarm的docker ps -a仍然显示退出状态下的大约10个容器-我可以这样做，因为那是唯一的问题，因为我添加了最后一步，在每个节点上调用docker system prune -f 。 不幸的是，大约有25％的时间，在执行stack rm命令后，我的容器仍在运行（在某些情况下，我已经等待了10分钟以上），所以docker system prune无法正常工作-当我看到此消息时就像其他人描述的那样，该节点上的大多数docker命令都挂起。 我必须在有问题的PID上执行kill -15才能获得控制权

如您所建议，我确实尝试过https://github.com/opencontainers/runc/pull/1698 ，但还是没有运气。 在我们说https://github.com/opencontainers/runc/pull/1698无法正常工作之前，您可以验证我是否对其进行了正确修补吗？ 我手动构建docker-ce / engine（分支17.12）位以更新runc进行提交：9f9c96235cc97674e935002fc3d78361b696a69e，然后用新建的二进制文件覆盖/ usr / bin / runc。 我在集群中的每个节点上执行了此操作，重新启动了docker并运行docker info | grep runc然后看到：

Default Runtime: runc
runc version: 9f9c96235cc97674e935002fc3d78361b696a69e (expected: b2567b37d7b75eb4cf325b77297b140ea686ce8f)

我应该尝试替换所有docker二进制文件吗？

我还附加了我的stacktrace（ curl --unix-socket /var/run/docker.sock http://localhost/debug/pprof/goroutine?debug=2 ）。
goroutine_debug.log

@foleymic您看到的问题似乎与runc问题类似。
替换dockerd不是必需的，而是替换docker-runc二进制文件。

请注意，出于测试目的，您可以将修补后的runc安装到自定义位置，并告诉docker将其用作默认运行时（或基于每个容器）。
为此，基本上只是使用上述提交构建runc，将其放置在/usr/local/bin/patched-runc ，然后使用--add-runtime myrunc=/usr/local/bin/patched-runc --default-runtime=myrunc启动docker（也可以将它们放置在/etc/docker/daemon.json ）。

@ cpuguy83-感谢您的快速回复。 我基本上只是更换runc，而是通过从17.12分支建立完整的泊坞窗-CE引擎，然后更改RUNC_COMMIT二进制文件，并提交给vendor.conf b2567b37d7b75eb4cf325b77297b140ea686ce8f。 无论如何，我认为您的描述听起来要好得多，所以让我尝试仅构建runc并替换它并重复测试。 再次感谢！

@foleymic太棒了。 也许docker-containerd-ctr pprof --debug-socket /run/docker/containerd/docker-containerd-debug.sock goroutines的输出也是谨慎的。

@ AlterEgo7您的堆栈跟踪非常错误...有趣...
有许多goroutine（绿色线程）处于“可运行”状态……这意味着它们正在等待真正的OS线程实际运行它们。

您可以从主机获取进程列表吗？ docker run --pid=host busybox ps aux应该做。

@ cpuguy83-我重建了独立的runc（检出了commit 9f9c96235cc97674e935002fc3d78361b696a69e ）并按照您的建议进行了部署（下面的daemon.json片段）并重现了该问题。 根据要求，我附加了docker-containerd-debug.sock goroutines的输出

感谢您的帮助，如果您需要我做或尝试其他任何事情，请告诉我。

docker-containerd-debug.log

cat /etc/docker/daemon.json
{
    "default-runtime":"patchedrunc",
    "runtimes": {
        "patchedrunc": {
             "path": "/usr/local/bin/patched-runc"
        }
    },
    ....

码头工人信息

docker info | grep runc
Runtimes: patchedrunc runc
Default Runtime: patchedrunc
WARNING: bridge-nf-call-ip6tables is disabled
runc version: 9f9c96235cc97674e935002fc3d78361b696a69e (expected: b2567b37d7b75eb4cf325b77297b140ea686ce8f)

我的开发团队中的多个人都遇到了MacOS 10.13和Docker For Mac 17.12挂起的docker命令。 尝试停止/杀死容器时发生在我身上，但在其他命令上也发生过，例如docker inspect

附件为curl --unix-socket /var/run/docker.sock http:/./debug/pprof/goroutine?debug=2的输出
docker-debug.log

我在等待已挂起超过30秒的docker-compose ps拿了它。 有趣的是docker ps此杭期间工作正常，但docker inspect任何返回的容器的docker ps挂起就像docker-compose ps一样。

我们不使用docker-sync，但确实通过Docker for Mac通过OSXFS文件系统使用卷挂载。
让我知道我还有什么可以做以帮助调查的。

@ cpuguy83抱歉，我知道您只是想要该日志，而不管此刻是否失败。

我目前正在使用的撰写有36个容器。 我试图通过简单地运行docker-compose up和docker-compose down来重现该问题。 第一次很棒，但是第二次，三个容器仍处于“升起”状态，其他所有容器仍处于“退出状态”。 这是日志的输出：

docker_debug.txt

这是docker-compose down报告的错误：

ERROR: An HTTP request took too long to complete. Retry with --verbose to obtain debug information. If you encounter this issue regularly because of slow network conditions, consider setting COMPOSE_HTTP_TIMEOUT to a higher value (current value: 60).

我注意到的一件事是，它似乎只是一个阻止其他容器的容器。 特别是在这种情况下，没有停止的3个容器是postgres，etcd和帮助配置etcd的助手。 但是，看起来是postgres阻止了其他人。 例如，我可以运行docker inspect etcd并且它可以工作，但是docker inspect postgres超时失败。

请注意，这只是此特定情况的一个示例。 我并不是说它总是应归咎于postgres。 也许下次发生时，它将是redis或Rabbitmq。

同样，它也使用群体。

我也有同样的问题。
在我工作的公司中，我们有一个Docker主机，其中有许多运行Java环境进行开发的容器，并且我不时无法删除图像或容器，无法检查图像或容器以及其他操作。

我在journalctl中收到许多消息，例如：
Jan 31 11:31:51 DOCKER-DEV dockerd[1882]: time="2018-01-31T11:31:51.598987401Z" level=warning msg="unknown container" container=540456eed4104723a0d4e9d4628d436ec381f978738f83dda16f22430cc60094 module=libcontainerd namespace=plugins.moby

我在Ubuntu 16.04.3 LTS中运行17.12 CE。

非常感谢您的时间和帮助。

@luisnabais在这里讨论有关Unknown container的问题： https :

@sorenhansendk谢谢，我已经知道这一点，我正在关注两个线程，我在17.12上都

更新-我已经设置了一个新的3节点集群（相同的VM模板）并手动安装了docker-18.02.0-ce的RC 1（https://download.docker.com/linux/static/test/x86_64/docker -18.02.0-ce-rc1.tgz），并且无法重现该问题。 另外，由于＃35891，我不再在日志中看到Unknown container消息，并且所有未定义的卷也都被删除了。 我将做一些更多的测试，以尝试找出哪个二进制文件具有修复功能。

至少对我来说，这个问题是断断续续的。

@richardjq是哪个版本？
它存在于所有版本的Docker中，因为该问题在runc https://github.com/opencontainers/runc/pull/1698中

12.17.0-这个

更新到最新的Docker（Mac OS）后，我也遇到了挂起同样的问题。 这是我的调试输出是否有帮助： https :

谢谢大家，非常有帮助。

@carlisia这个优势还是稳定的？ （或者，也许只是因为About疯狂，所以About屏幕中的版本会更好）。

@ cpuguy83稳定。 我只是从UI运行更新选项。

我不确定这是否有帮助，但是我注意到我认为我会加入到这次对话中的奇怪之处。 该问题再次发生（使用泊坞窗docker 17.12.0-ce），因此我查看了删除了堆栈/服务的节点上的守护程序日志，但是容器仍处于healthy状态。 docker ps显示此容器ID为8c5381ca6248 ，我在守护程序日志中grep输入该ID，我看到它以pid=94570开头，但是在调用stack rm命令，我在日志中看到一个重复3次的错误-该容器ID的Ignoring Exit Event, no such exec command found ，但是这三个日志消息中的每一个都有不同的PID，而且它们都不等于pid=94570 。 我还确认pid 94570是正确的PID（运行ps ax | grep 8c5381ca6248 ）。

知道为什么群会具有错误的PID吗？

这是守护程序日志的一个片段：

$ journalctl | grep 8c538
Feb 01 12:03:12 xxx dockerd[38879]: time="2018-02-01T12:03:12-05:00" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/8c5381ca6248dcbe199c56f87842c2cfd089e3ee5547895d7f037ed79e20dfc0/shim.sock" debug=false module="containerd/tasks" pid=94570
Feb 01 12:03:53 xxx dockerd[38879]: time="2018-02-01T12:03:53.823039243-05:00" level=warning msg="unknown container" container=8c5381ca6248dcbe199c56f87842c2cfd089e3ee5547895d7f037ed79e20dfc0 module=libcontainerd namespace=plugins.moby
Feb 01 12:03:53 xxx dockerd[38879]: time="2018-02-01T12:03:53.879628386-05:00" level=warning msg="unknown container" container=8c5381ca6248dcbe199c56f87842c2cfd089e3ee5547895d7f037ed79e20dfc0 module=libcontainerd namespace=plugins.moby
Feb 01 12:04:04 xxx dockerd[38879]: time="2018-02-01T12:04:04.143264783-05:00" level=warning msg="unknown container" container=8c5381ca6248dcbe199c56f87842c2cfd089e3ee5547895d7f037ed79e20dfc0 module=libcontainerd namespace=plugins.moby
Feb 01 12:04:04 xxx dockerd[38879]: time="2018-02-01T12:04:04.285288560-05:00" level=warning msg="unknown container" container=8c5381ca6248dcbe199c56f87842c2cfd089e3ee5547895d7f037ed79e20dfc0 module=libcontainerd namespace=plugins.moby
Feb 01 12:04:14 xxx dockerd[38879]: time="2018-02-01T12:04:14.685324737-05:00" level=warning msg="unknown container" container=8c5381ca6248dcbe199c56f87842c2cfd089e3ee5547895d7f037ed79e20dfc0 module=libcontainerd namespace=plugins.moby
Feb 01 12:04:24 xxx dockerd[38879]: time="2018-02-01T12:04:24.310513628-05:00" level=warning msg="Health check for container 8c5381ca6248dcbe199c56f87842c2cfd089e3ee5547895d7f037ed79e20dfc0 error: context cancelled"
Feb 01 12:04:34 xxx dockerd[38879]: time="2018-02-01T12:04:34.520599061-05:00" level=warning msg="unknown container" container=8c5381ca6248dcbe199c56f87842c2cfd089e3ee5547895d7f037ed79e20dfc0 module=libcontainerd namespace=plugins.moby
Feb 01 12:04:44 xxx dockerd[38879]: time="2018-02-01T12:04:44.332650759-05:00" level=warning msg="Health check for container 8c5381ca6248dcbe199c56f87842c2cfd089e3ee5547895d7f037ed79e20dfc0 error: context cancelled"
Feb 01 12:04:54 xxx dockerd[38879]: time="2018-02-01T12:04:54.666961688-05:00" level=warning msg="unknown container" container=8c5381ca6248dcbe199c56f87842c2cfd089e3ee5547895d7f037ed79e20dfc0 module=libcontainerd namespace=plugins.moby
Feb 01 12:05:04 xxx dockerd[38879]: time="2018-02-01T12:05:04.348300333-05:00" level=warning msg="Health check for container 8c5381ca6248dcbe199c56f87842c2cfd089e3ee5547895d7f037ed79e20dfc0 error: context cancelled"
Feb 01 12:05:14 xxx dockerd[38879]: time="2018-02-01T12:05:14.633386506-05:00" level=warning msg="unknown container" container=8c5381ca6248dcbe199c56f87842c2cfd089e3ee5547895d7f037ed79e20dfc0 module=libcontainerd namespace=plugins.moby
Feb 01 12:05:20 xxx dockerd[38879]: time="2018-02-01T12:05:20.813814668-05:00" level=warning msg="unknown container" container=8c5381ca6248dcbe199c56f87842c2cfd089e3ee5547895d7f037ed79e20dfc0 module=libcontainerd namespace=plugins.moby
Feb 01 12:05:20 xxx dockerd[38879]: time="2018-02-01T12:05:20.813896784-05:00" level=warning msg="Ignoring Exit Event, no such exec command found" container=8c5381ca6248dcbe199c56f87842c2cfd089e3ee5547895d7f037ed79e20dfc0 exec-id=9669e0bea74dbd90d56f61c3b8070fc043f6254d5b5b089d1562fdbfb154aa2b exec-pid=106625
Feb 01 12:05:20 xxx dockerd[38879]: time="2018-02-01T12:05:20.845301695-05:00" level=warning msg="unknown container" container=8c5381ca6248dcbe199c56f87842c2cfd089e3ee5547895d7f037ed79e20dfc0 module=libcontainerd namespace=plugins.moby
Feb 01 12:05:20 xxx dockerd[38879]: time="2018-02-01T12:05:20.845372559-05:00" level=warning msg="Ignoring Exit Event, no such exec command found" container=8c5381ca6248dcbe199c56f87842c2cfd089e3ee5547895d7f037ed79e20dfc0 exec-id=b637507387bcce7dcf3f270d48eaf75d4d5deccd00506d0ed656c78833ad8cda exec-pid=104564
Feb 01 12:05:20 xxx dockerd[38879]: time="2018-02-01T12:05:20.877888876-05:00" level=warning msg="unknown container" container=8c5381ca6248dcbe199c56f87842c2cfd089e3ee5547895d7f037ed79e20dfc0 module=libcontainerd namespace=plugins.moby
Feb 01 12:05:20 xxx dockerd[38879]: time="2018-02-01T12:05:20.952730879-05:00" level=warning msg="unknown container" container=8c5381ca6248dcbe199c56f87842c2cfd089e3ee5547895d7f037ed79e20dfc0 module=libcontainerd namespace=plugins.moby
Feb 01 12:05:20 xxx dockerd[38879]: time="2018-02-01T12:05:20.952805518-05:00" level=warning msg="Ignoring Exit Event, no such exec command found" container=8c5381ca6248dcbe199c56f87842c2cfd089e3ee5547895d7f037ed79e20dfc0 exec-id=2e5df9c74b29ca12fa9df0e14f095cf8524629564245107c4d8f0fb9d82adf9e exec-pid=103807

很抱歉继续增加此线程，但我一直在尝试。 无论问题是什么，它似乎都已在18.01.0-ce和18.02.0-ce-rc1之间的docker守护进程（ 17.12.0 ，我仅从分布在边缘和测试通道18.01 ， 18.02-rc1和18.02-rc2的二进制文件中替换了dockerd ，在每个版本上运行多个测试。

• dockerd 17.12.0-ce失败
• dockerd 18.01.0-ce失败
• dockerd 18.02.0-ce-rc1成功

明确地说，当我说“失败”时，我所谈论的问题是由docker stack deploy创建的容器在​​执行docker stack rm之后仍保持Up状态。 即使在18.02.0-ce-rc2中，仍然存在一个问题，其中docker stack rm会停止容器，但不会删除它们（容器保持退出状态）-不理想，但是我可以通过运行来解决docker stack rm命令在每个堆栈上完成后进行修剪（请注意，我等待30秒钟再发布修剪，以便在我接管之前给集群一些时间来完成清理工作）。

好消息是，它看起来像在下一版本中已修复，但我真的不想向我的团队推广RC。 我将浏览18.01和18.02-rc1之间的提交，以查看是否可以找到此修复程序。 @ cpuguy83和@thaJeztah ，任何想法都会浮现。 您想让我尝试或提供日志吗？ 如果可以帮助我找到可能的提交，则可以在fork中对其进行测试。

提醒您，通过这些测试需要花费时间，并且该问题无法始终如一地再现，因此即使我没有看到它在18.02（RC 1或2）中也发生过，但我不能百分百肯定它不会出现在某个点上-如果我看到此问题的任何发生，我将继续使用RC1进行测试并更新此线程，但是对于17.12-18.01，我已经看到此问题的发生率约为50％，所以我相当有信心它已在18.02中修复。

我同意你@foleymic！ 我已经将测试环境升级到了最新版本的Docker（候选发行版），并且关于停止/杀死容器的错误也得到了很好的结果。 我认为runc的更新版本已解决了问题👍

感谢您的报告。 我认为这完全由容器化的1.0.1和修补的runc组成。

@sorenhansendk-很高兴您也看到了积极的结果。 @ cpuguy83还认为这是完成与runc修复程序，但这绝对不是我的修复程序。 我目前正在运行所有内容，包括在17.12.0上的runc和在18.02.0-rc1上只有dockerd 。
相反，如果我仅升级runc我会看到问题（我尝试仅使用https://github.com/opencontainers/runc/pull/1698手动构建runc并从测试中获取版本和边缘发行版）。

@ cpuguy83-我仅与修补的dockerd一起运行。 Containered和Runc仍在17.12.0

码头工人信息：

...
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 89623f28b87a6004d4b785663257362d1658a729 (expected: 9b55aab90508bd389d7654c4baf173a981477d55)
runc version: b2567b37d7b75eb4cf325b77297b140ea686ce8f (expected: 9f9c96235cc97674e935002fc3d78361b696a69e)
...

是的，较新的容器随附18.02。
我们正在为17.12开发新的补丁程序。

那么dockerd包括容器化的吗？ 我认为这是一个单独的二进制文件-docker-containerd。 无论如何，很高兴听到您正在为17.12制作补丁程序-这是个好消息。 再次感谢您的帮助。

哦，所以您只从字面上升级了dockerd二进制文件。 喔好吧。 我将不得不仔细检查此处可能涉及哪些补丁，并确保它们已安装到17.12版本中。

:)-我当前正在运行Franken-docker！ 一切都是17.12.0，除了dockerd是18.02-rc1

仅供参考，我认为解决方法在于PR＃35748

我仍然能够通过版本18.02.0-ce-rc2-mac51重现该问题（22446）

linuxkit-025000000001:/# /usr/bin/containerd --version
containerd github.com/containerd/containerd v1.0.1 9b55aab90508bd389d7654c4baf173a981477d55

docker info

containerd version: 9b55aab90508bd389d7654c4baf173a981477d55
runc version: 9f9c96235cc97674e935002fc3d78361b696a69e

@ximarx感谢您的举报！ 您可以从docker获取堆叠转储并进行容器化吗？
使用docker你可以做

curl --unix-socket /var/run/docker.sock http:/./debug/pprof/goroutine?debug=2

对于Mac上的容器，这有点棘手...

 docker run -it --rm -v /:/host alpine /host/usr/local/bin/docker-containerd-ctr pprof --debug-socket /host/run/docker/containerd/docker-containerd-debug.sock goroutines

@ximarx-当您说问题仍然存在时，您是在谈论容器仍然保留但处于退出状态的容器吗，还是仍然看到它们处于运行状态？ 我只注意到在18.02版本中，我仍然看到容器仍然存在，但是没有一个容器处于运行状态，因此我可以在docker stack rm之后通过运行prune来解决。 这不是很理想，但是至少我放下所有东西后可以进入干净的状态-在使用18.02之前，我必须手动终止进程以删除那些仍在运行但不再与堆栈关联的容器。

顺便说一句，我在叉子中将PR＃35748合并到我的12.17上并构建了-自从应用该构建以来，我还没有看到问题。

您可以从docker获取堆叠转储并进行容器化吗？

@ cpuguy83 ：不幸的是我已经重启了

当您说问题仍然存在时，您是在谈论容器仍保留但处于退出状态的容器，还是仍然看到它们处于运行状态？

@foleymic Container仍处于运行状态，但docker-compose ps和docker stop ）

@ximarx-感谢您确认这一点，我在多个群集集群中看到了完全相同的行为，但是自从将它们升级到修补的版本以来，我们没有看到这个问题。 我看到的唯一区别是，我们在RHEL上与在Mac上运行。

这个问题已经解决了吗？
如果是这样，它将以哪个版本发布？

我相信固定版本已降为18.02，但我正在等待他们的确认

“稳定”通道17.12.0版本仍然存在此错误； 如果已修复，是否可以将PR反向移植到补丁程序版本17.12.1？ 如果人们必须完全恢复到17.09或求助于边缘发布，那么稳定的渠道就非常不稳定。

+1为补丁版本17.12.1

正在处理中。 谢谢！

抱歉，无法预热该线程-好像修复了;）-但我有一个快速的问题：
自从我们在Dockerfile中添加了HEALTHCHECK之后，我们在docker-ce-17.12上看到了完全相同的问题。 在其Dockerfile中未指定HEALTCHECK的容器会停止运行。
这可能与HEALTHCHECK相关联还是仅仅是巧合？

干杯
菲尔

@PhilPhonic是的，可以通过运行状况检查来触发

https://github.com/moby/moby/pull/36097 （添加到昨天的版本中）是否可以解决此问题？

我来自问题＃34213，不能完全确定问题是否相同，但是我被引导到这里。

对于我们来说，在完成这些修复后，docker容器仍然卡住，并且从未完成执行它，但是在这种情况下，我能够在不重新启动docker守护程序的情况下将其停止。

码头工人版本

Client:
 Version:   17.12.1-ce
 API version:   1.35
 Go version:    go1.9.4
 Git commit:    7390fc6
 Built: Tue Feb 27 22:17:56 2018
 OS/Arch:   linux/amd64

Server:
 Engine:
  Version:  17.12.1-ce
  API version:  1.35 (minimum version 1.12)
  Go version:   go1.9.4
  Git commit:   7390fc6
  Built:    Tue Feb 27 22:16:28 2018
  OS/Arch:  linux/amd64
  Experimental: false

码头工人信息

Containers: 1
 Running: 1
 Paused: 0
 Stopped: 0
Images: 1
Server Version: 17.12.1-ce
Storage Driver: overlay2
 Backing Filesystem: tmpfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 9b55aab90508bd389d7654c4baf173a981477d55
runc version: 9f9c96235cc97674e935002fc3d78361b696a69e
init version: 949e6fa
Security Options:
 apparmor
Kernel Version: 4.4.0-64-generic
Operating System: Ubuntu 14.04 LTS
OSType: linux
Architecture: x86_64
CPUs: 6
Total Memory: 19.8GiB
Name: vm64-3
ID: OHED:ZYGD:4ZNG:QWGC:7VQ5:2KVU:YXP3:CKCR:AMV5:FVFF:PMJ6:ICXM
Docker Root Dir: /tmp/ramdisk/docker
Debug Mode (client): false
Debug Mode (server): true
 File Descriptors: 27
 Goroutines: 47
 System Time: 2018-03-01T07:57:09.565797691Z
 EventsListeners: 0
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

泊坞窗检查预发布版本（卡住的容器）

[
    {
        "Id": "34d42f7a8246de8c6eb4b3d9f8fe3a62c1b0ac8ce7a800f918538b33c35d282a",
        "Created": "2018-02-28T19:22:54.823764324Z",
        "Path": "bash",
        "Args": [
            "-c",
            "./release_scripts/bootstrap_prerelease.sh /release_scripts/.rosinstall unused-version-argument git /release_scripts/.rosinstall erbium xenial kinetic index.yaml coverage true "
        ],
        "State": {
            "Status": "running",
            "Running": true,
            "Paused": false,
            "Restarting": false,
            "OOMKilled": false,
            "Dead": false,
            "Pid": 6495,
            "ExitCode": 0,
            "Error": "",
            "StartedAt": "2018-02-28T19:22:55.675976607Z",
            "FinishedAt": "0001-01-01T00:00:00Z"
        },
        "Image": "sha256:fa0591be5fc0613715fb2605264ff61967811262bdbc93d262faca23df1bf2fe",
        "ResolvConfPath": "/tmp/ramdisk/docker/containers/34d42f7a8246de8c6eb4b3d9f8fe3a62c1b0ac8ce7a800f918538b33c35d282a/resolv.conf",
        "HostnamePath": "/tmp/ramdisk/docker/containers/34d42f7a8246de8c6eb4b3d9f8fe3a62c1b0ac8ce7a800f918538b33c35d282a/hostname",
        "HostsPath": "/tmp/ramdisk/docker/containers/34d42f7a8246de8c6eb4b3d9f8fe3a62c1b0ac8ce7a800f918538b33c35d282a/hosts",
        "LogPath": "/tmp/ramdisk/docker/containers/34d42f7a8246de8c6eb4b3d9f8fe3a62c1b0ac8ce7a800f918538b33c35d282a/34d42f7a8246de8c6eb4b3d9f8fe3a62c1b0ac8ce7a800f918538b33c35d282a-json.log",
        "Name": "/prerelease",
        "RestartCount": 0,
        "Driver": "overlay2",
        "Platform": "linux",
        "MountLabel": "",
        "ProcessLabel": "",
        "AppArmorProfile": "unconfined",
        "ExecIDs": null,
        "HostConfig": {
            "Binds": [
                "/home/hudson/.hudson/workspace/Mega-Integration/release_scripts:/release_scripts",
                "/tmp/prerelease_out:/tmp/prerelease_out",
                "/home/hudson/pbuilder_ccache:/tmp/pbuilder_ccache",
                "/tmp/.X11-unix:/tmp/.X11-unix:rw",
                "/home/hudson/exchange:/home/user/exchange"
            ],
            "ContainerIDFile": "",
            "LogConfig": {
                "Type": "json-file",
                "Config": {}
            },
            "NetworkMode": "host",
            "PortBindings": {},
            "RestartPolicy": {
                "Name": "no",
                "MaximumRetryCount": 0
            },
            "AutoRemove": false,
            "VolumeDriver": "",
            "VolumesFrom": null,
            "CapAdd": null,
            "CapDrop": null,
            "Dns": [],
            "DnsOptions": [],
            "DnsSearch": [],
            "ExtraHosts": null,
            "GroupAdd": null,
            "IpcMode": "shareable",
            "Cgroup": "",
            "Links": null,
            "OomScoreAdj": 0,
            "PidMode": "",
            "Privileged": true,
            "PublishAllPorts": false,
            "ReadonlyRootfs": false,
            "SecurityOpt": [
                "label=disable"
            ],
            "UTSMode": "",
            "UsernsMode": "",
            "ShmSize": 67108864,
            "Runtime": "runc",
            "ConsoleSize": [
                0,
                0
            ],
            "Isolation": "",
            "CpuShares": 0,
            "Memory": 0,
            "NanoCpus": 0,
            "CgroupParent": "",
            "BlkioWeight": 0,
            "BlkioWeightDevice": [],
            "BlkioDeviceReadBps": null,
            "BlkioDeviceWriteBps": null,
            "BlkioDeviceReadIOps": null,
            "BlkioDeviceWriteIOps": null,
            "CpuPeriod": 0,
            "CpuQuota": 0,
            "CpuRealtimePeriod": 0,
            "CpuRealtimeRuntime": 0,
            "CpusetCpus": "",
            "CpusetMems": "",
            "Devices": [
                {
                    "PathOnHost": "/dev/snd",
                    "PathInContainer": "/dev/snd",
                    "CgroupPermissions": "rwm"
                }
            ],
            "DeviceCgroupRules": null,
            "DiskQuota": 0,
            "KernelMemory": 0,
            "MemoryReservation": 0,
            "MemorySwap": 0,
            "MemorySwappiness": null,
            "OomKillDisable": false,
            "PidsLimit": 0,
            "Ulimits": [
                {
                    "Name": "nofile",
                    "Hard": 10240,
                    "Soft": 10240
                }
            ],
            "CpuCount": 0,
            "CpuPercent": 0,
            "IOMaximumIOps": 0,
            "IOMaximumBandwidth": 0
        },
        "GraphDriver": {
            "Data": {
                "LowerDir": "/tmp/ramdisk/docker/overlay2/03efcd70b4e0faac571708859c2cfb0d43f7da1331702508980c6c49945ba1e4-init/diff:/tmp/ramdisk/docker/overlay2/b4f54a4bcf9693f50de7c5789597a12479edfd42d9b8fa2f449f9c6a3f09e580/diff:/tmp/ramdisk/docker/overlay2/ec49f1ad95aaa5088113b71e7894db1ce0229123457e8e63f76cde8d5501cc73/diff:/tmp/ramdisk/docker/overlay2/3bb76c8cef3b6466f1f57167c326a8b065e5f0133424f16b1c0be2510ec60574/diff:/tmp/ramdisk/docker/overlay2/77e87eeac5f5325031e0ababfe19e094c1a443593e4f849f5d79f3f15ef4a573/diff:/tmp/ramdisk/docker/overlay2/1ad76d631a387704468b365fddcc772596361c64e3085cd9efc50f514211024c/diff:/tmp/ramdisk/docker/overlay2/14ee3b713fb1cb48a8baa1c18c39bb85ebbd4acf23df18f146d1b2ddf2106772/diff:/tmp/ramdisk/docker/overlay2/0636682f487ae65c4123e0d281991fde9eeb207745556ead1aa605588df66053/diff:/tmp/ramdisk/docker/overlay2/635fe79b7fa395eb1171b4cb8a5b053643c152699e2e36711d219de92f32d44f/diff:/tmp/ramdisk/docker/overlay2/f5f559bda6b48218a145b81d505dc6bb7dd1dbd144c00eb8200884c20c541a16/diff:/tmp/ramdisk/docker/overlay2/fd6c90c91221f4395ee5d1fa6e6ce165d98cdb5e38edd00d4ef5aaf7bda9bdec/diff:/tmp/ramdisk/docker/overlay2/4198097e8c9bcdcc596434d269f39a2014049b9f62bbd9e259c6d0653371db6e/diff:/tmp/ramdisk/docker/overlay2/d9665613172c8ceb3cd6bfa5709bb05a38f5886322e9ba6f3b59604755670125/diff:/tmp/ramdisk/docker/overlay2/c3eec0a6317354680ed46e1883089fb345a91aada6d1bebc1585869e7497cfa2/diff:/tmp/ramdisk/docker/overlay2/f82163d218a43b50885139dfad1c5dedcd4d381fd76af4a38e0480d48e2ab32b/diff:/tmp/ramdisk/docker/overlay2/c72be8cddf53887e1c8c55207153d2b59e26e3327b60764d1dbe632d6db7b4ff/diff",
                "MergedDir": "/tmp/ramdisk/docker/overlay2/03efcd70b4e0faac571708859c2cfb0d43f7da1331702508980c6c49945ba1e4/merged",
                "UpperDir": "/tmp/ramdisk/docker/overlay2/03efcd70b4e0faac571708859c2cfb0d43f7da1331702508980c6c49945ba1e4/diff",
                "WorkDir": "/tmp/ramdisk/docker/overlay2/03efcd70b4e0faac571708859c2cfb0d43f7da1331702508980c6c49945ba1e4/work"
            },
            "Name": "overlay2"
        },
        "Mounts": [
            {
                "Type": "bind",
                "Source": "/home/hudson/.hudson/workspace/Mega-Integration/release_scripts",
                "Destination": "/release_scripts",
                "Mode": "",
                "RW": true,
                "Propagation": "rprivate"
            },
            {
                "Type": "bind",
                "Source": "/tmp/prerelease_out",
                "Destination": "/tmp/prerelease_out",
                "Mode": "",
                "RW": true,
                "Propagation": "rprivate"
            },
            {
                "Type": "bind",
                "Source": "/home/hudson/pbuilder_ccache",
                "Destination": "/tmp/pbuilder_ccache",
                "Mode": "",
                "RW": true,
                "Propagation": "rprivate"
            },
            {
                "Type": "bind",
                "Source": "/tmp/.X11-unix",
                "Destination": "/tmp/.X11-unix",
                "Mode": "rw",
                "RW": true,
                "Propagation": "rprivate"
            },
            {
                "Type": "bind",
                "Source": "/home/hudson/exchange",
                "Destination": "/home/user/exchange",
                "Mode": "",
                "RW": true,
                "Propagation": "rprivate"
            }
        ],
        "Config": {
            "Hostname": "vm64-3",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": true,
            "AttachStderr": true,
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "NIGHTLY_BUILD=1",
                "LOCAL_USER_ID=1001",
                "LOCAL_GROUP_ID=1001",
                "LOCAL_GROUP_NAME=hudson",
                "DISPLAY=:0",
                "QT_X11_NO_MITSHM=1",
                "PATH=/usr/local/nvidia/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64:"
            ],
            "Cmd": [
                "bash",
                "-c",
                "./release_scripts/bootstrap_prerelease.sh /release_scripts/.rosinstall unused-version-argument git /release_scripts/.rosinstall erbium xenial kinetic index.yaml coverage true "
            ],
            "Image": "gitlab:4567/dockers/pal_docker_images/core-erbium-internal-staging",
            "Volumes": null,
            "WorkingDir": "",
            "Entrypoint": null,
            "OnBuild": null,
            "Labels": {
                "com.nvidia.volumes.needed": "nvidia_driver",
                "maintainer": "[email protected]"
            }
        },
        "NetworkSettings": {
            "Bridge": "",
            "SandboxID": "57d9e5bf0c9e05f5351787befe065ef390dd39f39d2aa7681b28abfe4381a8d3",
            "HairpinMode": false,
            "LinkLocalIPv6Address": "",
            "LinkLocalIPv6PrefixLen": 0,
            "Ports": {},
            "SandboxKey": "/var/run/docker/netns/default",
            "SecondaryIPAddresses": null,
            "SecondaryIPv6Addresses": null,
            "EndpointID": "",
            "Gateway": "",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "IPAddress": "",
            "IPPrefixLen": 0,
            "IPv6Gateway": "",
            "MacAddress": "",
            "Networks": {
                "host": {
                    "IPAMConfig": null,
                    "Links": null,
                    "Aliases": null,
                    "NetworkID": "39e3e5618a9dcb24f03e831dbe1929bacdca472fefa3542122bc217429d6914d",
                    "EndpointID": "eabbf4a17b6f92b564ff0303cbc8d3ec67744f9d0b90fe21d49f5b7aebca1d5c",
                    "Gateway": "",
                    "IPAddress": "",
                    "IPPrefixLen": 0,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": "",
                    "DriverOpts": null
                }
            }
        }
    }
]

在具有3节点Rancher（v1.6.14）群集的Ubuntu 16.04.3 LTS上的17.12.0-ce上也经历了这些症状。 奇怪的是，虽然它们都运行相同的docker / ubuntu / rancher版本，但只有一个节点间歇性地（在1-7天之间）出现问题。 尽管发生在其上的节点恰好运行了最多数量的容器，因此具有最高的负载量。

计划在本周末将此集群升级到17.12.1-ce，以查看这是否有助于解决交叉问题

还有其他人在17.12.1-ce上有运气吗？

@mauriceteunissen 17.12.1-ce为我解决了这个问题！

可悲的是17.12.1-ce并没有为我解决问题。
我仍然无法使用Dockerfile中的HEALTHCHECK停止某些容器。

@ cpuguy83

复制它花了比预期更长的时间。 zip包含您所需的两个转储。

$ docker info
Containers: 21
 Running: 21
 Paused: 0
 Stopped: 0
Images: 42
Server Version: 18.03.0-ce-rc1
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host ipvlan macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: cfd04396dc68220d1cecbe686a6cc3aa5ce3667c
runc version: 6c55f98695e902427906eed2c799e566e3d3dfb5
init version: 949e6fa
Security Options:
 seccomp
  Profile: default
Kernel Version: 4.9.75-linuxkit-aufs
Operating System: Docker for Mac
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 3.363GiB
Name: linuxkit-025000000001
ID: EDQO:EXVD:FYS7:R47M:V4UC:H63J:NKHU:T466:DLMW:VXSO:C7XE:BROD
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): true
 File Descriptors: 185
 Goroutines: 197
 System Time: 2018-03-07T15:05:02.215912835Z
 EventsListeners: 2
HTTP Proxy: docker.for.mac.http.internal:3128
HTTPS Proxy: docker.for.mac.http.internal:3129
Registry: https://index.docker.io/v1/
Labels:
Experimental: true
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

dumps.zip

我经常在17.12.0-ce下发生这种情况。 我切换到18.03.0-ce-rc1-mac54，它的发生频率要低得多，尽管它仍然偶尔发生。 我碰巧碰到所有容器上都有HEALTHCHECKs。

嗨，很抱歉，如果这是一个已解决的问题，只是在等待修复程序进入稳定的版本，但是我不确定我是否有稍微不同的变化。 我正在以Swarm模式运行Docker For Windows，以匹配我的生产环境，并且部署的堆栈没有问题，但是单独运行的容器有时会陷入无法杀灭的状态。 这种情况至少持续了数周（可能不确定数月，我不确定）。

通常，我部署的是我正在处理的应用程序。 除此之外，我将项目绑定安装到容器中时通过docker run --rm命令运行开发任务。 例如，Webpack的构建命令为docker run --rm -t --env“ NODE_ENV = development” -v $ {pwd）：/ srv --workdir / srv node：alpine node_modules / .bin / webpack- config webpack.config.js-进度

有时，Webpack构建会冻结（大约10-12％），我不得不从中退出ctrl + c 。 我尝试将其放置30分钟左右，但它从未解冻（我的典型构建时间少于25秒）。 在这种情况下，当我按ctrl + c ，有时容器会像应有的那样死得很厉害，但有时它像什么也没有发生一样挂在身边。 当容器挂起时， docker ps显示它仍在运行，但是docker kill和docker rm只是挂起，直到我取消它们为止（再次，我尝试将其放置好20分钟或没有完成kill和rm命令）。

我怀疑存在内存问题，因此尝试在不使Windows不稳定的情况下尽我所能增加分配给Docker的内存，但这对此问题的发生频率没有影响。 然后，我尝试在构建运行时查看docker stats，它似乎达到约150Mb（在我的可用2.5Gb中）达到峰值。 即使在我为我的开发环境运行的所有其他容器中，总计的总数也不超过20％，所以看来OOM并不是问题。

另一点是，每当容器进入此状态时，如果我尝试在新容器中重新运行失败的命令（例如，重新运行我为Webpack构建发布的相同的docker run命令），则新容器只是挂起而没有日志输出，并且也变得无法杀死。 现在，这使我怀疑它与文件系统有关。

Docker for Windows的简单重启可使一切恢复到工作状态。 通常，重新启动后重复我的Webpack build命令可以正常工作。

我怀疑我也坚持这个问题。

我正在使用lsof / strace进行调试，发现容器卡住了，无法在stdout / stderr管道中写一些东西。

% sudo ls -l /var/run/docker/containerd/2456b59776d918e0e07ae9259a54229a8f0985ae7b6ad2be7d25fccf8fdd5b49
total 0
prwx------ 1 root root 0 Mar  8 21:46 0048bf0b619804896d3b70ccab859221bf8d318415b26cedfc096b80f4f9ed08-stderr
prwx------ 1 root root 0 Mar  8 21:46 0048bf0b619804896d3b70ccab859221bf8d318415b26cedfc096b80f4f9ed08-stdout
prwx------ 1 root root 0 Mar  8 20:10 init-stderr
prwx------ 1 root root 0 Mar  9 11:17 init-stdout

我发现2456b59776d918e0e07ae9259a54229a8f0985ae7b6ad2be7d25fccf8fdd5b49是容器ID，但是0048bf0b619804896d3b70ccab859221bf8d318415b26cedfc096b80f4f9ed08是未知的。
顺便说一句，我在读取未知的两个管道后发现容器出口，它们使用stdout-> stderr命令。

% sudo cat /var/run/docker/containerd/2456b59776d918e0e07ae9259a54229a8f0985ae7b6ad2be7d25fccf8fdd5b49/0048bf0b619804896d3b70ccab859221bf8d318415b26cedfc096b80f4f9ed08-stdout
% sudo cat /var/run/docker/containerd/2456b59776d918e0e07ae9259a54229a8f0985ae7b6ad2be7d25fccf8fdd5b49/0048bf0b619804896d3b70ccab859221bf8d318415b26cedfc096b80f4f9ed08-stderr

我怀疑dockerd包含某些竞争条件，但我找不到。

Docker在我的主机上运行平稳。 但是，由于我从Docker 17.12.0升级到17.12.1，因此有时会遇到无法停止或重新启动容器的问题，尤其是当它们具有运行状况检查设置时。 因此，当我发现此问题时，我感到很困惑，因为17.12.1应该可以解决此问题而不触发它。

奇怪的是，如果重新启动主机，则会在Dockerd中帮助很多“清理”工作，因此我可以再次停止或重新启动某些容器。 但是经过几天/几周的运行，它将开始失败。 对于那些失败的容器，都使用docker-compose或直接使用docker cli失败（因此docker-compose down或docker stop <name>同样失败，类似于重新启动）。

$ docker info
Containers: 13
 Running: 7
 Paused: 0
 Stopped: 6
Images: 93
Server Version: 17.12.1-ce
Storage Driver: btrfs
 Build Version: Btrfs v4.4
 Library Version: 101
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 9b55aab90508bd389d7654c4baf173a981477d55
runc version: 9f9c96235cc97674e935002fc3d78361b696a69e
init version: 949e6fa
Security Options:
 apparmor
 seccomp
  Profile: default
 userns
Kernel Version: 4.13.0-36-generic
Operating System: Ubuntu 16.04.4 LTS
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 7.544GiB
Name: malmur
Docker Root Dir: /var/lib/docker/235536.235536
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

我有类似的问题，版本是18.02。 我认为我可以通过在Jenkins中运行测试来轻松重现它。 每次我运行测试时都会发生。

这是我的信息。

码头工人停止5a28870c74e1

Error response from daemon: cannot stop container: 5a28870c74e1: Cannot kill container 5a28870c74e166152b4fa57a95691969eb0fb85185504dee43e495168e179bdd: connection error: desc = "transport: dial unix /var/run/docker/containerd/docker-containerd.sock: connect: connection refused": unknown

码头工人信息的输出：

Containers: 6
 Running: 3
 Paused: 0
 Stopped: 3
Images: 8
Server Version: 18.02.0-ce
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: N/A (expected: 9b55aab90508bd389d7654c4baf173a981477d55)
runc version: 9f9c96235cc97674e935002fc3d78361b696a69e
init version: 949e6fa
Security Options:
 seccomp
  Profile: default
Kernel Version: 3.10.0-693.11.6.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 992.3MiB
Name: docker
ID: 5WRP:X7CW:ATNU:2UAT:RSKA:4BX7:R4T6:B5DT:HGF4:YRTF:BXGK:NON5
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

码头工人版本

Client:
 Version:   18.02.0-ce
 API version:   1.36
 Go version:    go1.9.3
 Git commit:    fc4de44
 Built: Wed Feb  7 21:14:12 2018
 OS/Arch:   linux/amd64
 Experimental:  false
 Orchestrator:  swarm

Server:
 Engine:
  Version:  18.02.0-ce
  API version:  1.36 (minimum version 1.12)
  Go version:   go1.9.3
  Git commit:   fc4de44
  Built:    Wed Feb  7 21:17:42 2018
  OS/Arch:  linux/amd64
  Experimental: false

码头工人检查

 [
    {
        "Id": "5a28870c74e166152b4fa57a95691969eb0fb85185504dee43e495168e179bdd",
        "Created": "2018-03-13T09:38:27.546391905Z",
        "Path": "cat",
        "Args": [],
        "State": {
            "Status": "running",
            "Running": true,
            "Paused": false,
            "Restarting": false,
            "OOMKilled": false,
            "Dead": false,
            "Pid": 22458,
            "ExitCode": 0,
            "Error": "",
            "StartedAt": "2018-03-13T09:38:27.977365052Z",
            "FinishedAt": "0001-01-01T00:00:00Z"
        },
        "Image": "sha256:0d90898532210246893d448740ee6ad83ad351f71dc188d4a5705d193cb3a580",
        "ResolvConfPath": "/var/lib/docker/containers/5a28870c74e166152b4fa57a95691969eb0fb85185504dee43e495168e179bdd/resolv.conf",
        "HostnamePath": "/var/lib/docker/containers/5a28870c74e166152b4fa57a95691969eb0fb85185504dee43e495168e179bdd/hostname",
        "HostsPath": "/var/lib/docker/containers/5a28870c74e166152b4fa57a95691969eb0fb85185504dee43e495168e179bdd/hosts",
        "LogPath": "/var/lib/docker/containers/5a28870c74e166152b4fa57a95691969eb0fb85185504dee43e495168e179bdd/5a28870c74e166152b4fa57a95691969eb0fb85185504dee43e495168e179bdd-json.log",
        "Name": "/reverent_lovelace",
        "RestartCount": 0,
        "Driver": "overlay2",
        "Platform": "linux",
        "MountLabel": "",
        "ProcessLabel": "",
        "AppArmorProfile": "",
        "ExecIDs": [
            "6fab9a929239a9659834a3975f11e73662d238c56f2ab994bed0b0df35e45d3b"
        ],
        "HostConfig": {
            "Binds": null,
            "ContainerIDFile": "",
            "LogConfig": {
                "Type": "json-file",
                "Config": {}
            },
            "NetworkMode": "default",
            "PortBindings": {},
            "RestartPolicy": {
                "Name": "no",
                "MaximumRetryCount": 0
            },
            "AutoRemove": false,
            "VolumeDriver": "",
            "VolumesFrom": [
                "5049a01b4358233471daa74fb11996a6b8e3acfd9c00b33ea7a52a3c24264eaa"
            ],
            "CapAdd": null,
            "CapDrop": null,
            "Dns": [],
            "DnsOptions": [],
            "DnsSearch": [],
            "ExtraHosts": null,
            "GroupAdd": null,
            "IpcMode": "shareable",
            "Cgroup": "",
            "Links": null,
            "OomScoreAdj": 0,
            "PidMode": "",
            "Privileged": false,
            "PublishAllPorts": false,
            "ReadonlyRootfs": false,
            "SecurityOpt": null,
            "UTSMode": "",
            "UsernsMode": "",
            "ShmSize": 67108864,
            "Runtime": "runc",
            "ConsoleSize": [
                0,
                0
            ],
            "Isolation": "",
            "CpuShares": 0,
            "Memory": 0,
            "NanoCpus": 0,
            "CgroupParent": "",
            "BlkioWeight": 0,
            "BlkioWeightDevice": [],
            "BlkioDeviceReadBps": null,
            "BlkioDeviceWriteBps": null,
            "BlkioDeviceReadIOps": null,
            "BlkioDeviceWriteIOps": null,
            "CpuPeriod": 0,
            "CpuQuota": 0,
            "CpuRealtimePeriod": 0,
            "CpuRealtimeRuntime": 0,
            "CpusetCpus": "",
            "CpusetMems": "",
            "Devices": [],
            "DeviceCgroupRules": null,
            "DiskQuota": 0,
            "KernelMemory": 0,
            "MemoryReservation": 0,
            "MemorySwap": 0,
            "MemorySwappiness": null,
            "OomKillDisable": false,
            "PidsLimit": 0,
            "Ulimits": null,
            "CpuCount": 0,
            "CpuPercent": 0,
            "IOMaximumIOps": 0,
            "IOMaximumBandwidth": 0
        },
        "GraphDriver": {
            "Data": {
                "LowerDir": "/var/lib/docker/overlay2/78b8c2d7843b9374c55d939ccb028e27bedb1bbba34032c38609eb57f32e74a1-init/diff:/var/lib/docker/overlay2/7816f6aae89ff8e103e90e971e70cdc3a5bda0722d89fa361fce259e169c8aa8/diff:/var/lib/docker/overlay2/72f3b10b691778a8838270b9548a52dfabb11f28b2f520f1a24981dd416d3a25/diff:/var/lib/docker/overlay2/40cbb59d56e7583545b375636d8b99666071d3511c7acc1edc87f8e175ff5df0/diff:/var/lib/docker/overlay2/e0f0d655a2a8e3d518e966d0bd90d8e1de6ea06cd99f27ca7a5716c5be2faa19/diff:/var/lib/docker/overlay2/367dc44c14f8ad44319a439436966e9e1b00779b51c819307eb3ab47422f0b36/diff:/var/lib/docker/overlay2/e04892a8d5d028c713f10f0cdbb5d0be6854b4c90b3fa9c2bee3f0fc7217bad9/diff",
                "MergedDir": "/var/lib/docker/overlay2/78b8c2d7843b9374c55d939ccb028e27bedb1bbba34032c38609eb57f32e74a1/merged",
                "UpperDir": "/var/lib/docker/overlay2/78b8c2d7843b9374c55d939ccb028e27bedb1bbba34032c38609eb57f32e74a1/diff",
                "WorkDir": "/var/lib/docker/overlay2/78b8c2d7843b9374c55d939ccb028e27bedb1bbba34032c38609eb57f32e74a1/work"
            },
            "Name": "overlay2"
        },
        "Mounts": [
            {
                "Type": "bind",
                "Source": "/home/docker/jenkins/jenkins_home",
                "Destination": "/var/jenkins_home",
                "Mode": "",
                "RW": true,
                "Propagation": "rprivate"
            },
            {
                "Type": "bind",
                "Source": "/var/run/docker.sock",
                "Destination": "/var/run/docker.sock",
                "Mode": "",
                "RW": true,
                "Propagation": "rprivate"
            }
        ],
        "Config": {
            "Hostname": "5a28870c74e1",
            "Domainname": "",
            "User": "1000:1000",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "Tty": true,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "BUILD_NUMBER=12",
                "GIT_PREVIOUS_COMMIT=91873d6d120f13daf6ea243a7b8a97e0f982c6be",
                "JOB_NAME=pipeline test/master",
                "RUN_CHANGES_DISPLAY_URL=http://unconfigured-jenkins-location/job/pipeline%20test/job/master/12/display/redirect?page=changes",
                "BUILD_DISPLAY_NAME=#12",
                "BUILD_TAG=jenkins-pipeline test-master-12",
                "GIT_URL=https://github.com/miwa911/jenkins-pipeline.git",
                "JOB_DISPLAY_URL=http://unconfigured-jenkins-location/job/pipeline%20test/job/master/display/redirect",
                "NODE_NAME=master",
                "BUILD_ID=12",
                "GIT_BRANCH=master",
                "GIT_COMMIT=91873d6d120f13daf6ea243a7b8a97e0f982c6be",
                "HUDSON_HOME=/var/jenkins_home",
                "HUDSON_SERVER_COOKIE=76488b4fc4db183f",
                "JENKINS_SERVER_COOKIE=76488b4fc4db183f",
                "RUN_DISPLAY_URL=http://unconfigured-jenkins-location/job/pipeline%20test/job/master/12/display/redirect",
                "WORKSPACE=/var/jenkins_home/workspace/pipeline_test_master-U6GGLXKIF4VESPPHZ475TWN76LB4NUH5VZJ6SPHWOEU6N7DL5ICA",
                "BRANCH_NAME=master",
                "EXECUTOR_NUMBER=1",
                "GIT_PREVIOUS_SUCCESSFUL_COMMIT=145052192787cd88a307e67777367e6f3c90cb81",
                "JENKINS_NODE_COOKIE=911ca1f2-c7cc-41ba-a1d1-78df2155e65b",
                "JOB_BASE_NAME=master",
                "NODE_LABELS=master",
                "CLASSPATH=",
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "NPM_CONFIG_LOGLEVEL=info",
                "NODE_VERSION=6.3.1"
            ],
            "Cmd": [
                "cat"
            ],
            "Image": "node:6.3",
            "Volumes": null,
            "WorkingDir": "/var/jenkins_home/workspace/pipeline_test_master-U6GGLXKIF4VESPPHZ475TWN76LB4NUH5VZJ6SPHWOEU6N7DL5ICA",
            "Entrypoint": null,
            "OnBuild": null,
            "Labels": {}
        },
        "NetworkSettings": {
            "Bridge": "",
            "SandboxID": "16d1c596372cdd66415d9e738f493207f7fc8ee4444e936b08b40d273b05de1b",
            "HairpinMode": false,
            "LinkLocalIPv6Address": "",
            "LinkLocalIPv6PrefixLen": 0,
            "Ports": {},
            "SandboxKey": "/var/run/docker/netns/16d1c596372c",
            "SecondaryIPAddresses": null,
            "SecondaryIPv6Addresses": null,
            "EndpointID": "baec7581c0a8f19708e82d87ff98016b89309ef9b2c7717e92c28742f218fa4c",
            "Gateway": "172.17.0.1",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "IPAddress": "172.17.0.4",
            "IPPrefixLen": 16,
            "IPv6Gateway": "",
            "MacAddress": "02:42:ac:11:00:04",
            "Networks": {
                "bridge": {
                    "IPAMConfig": null,
                    "Links": null,
                    "Aliases": null,
                    "NetworkID": "669f5d230c893729c633095f59ad01885c689028f4379aba3621f239f734e46b",
                    "EndpointID": "baec7581c0a8f19708e82d87ff98016b89309ef9b2c7717e92c28742f218fa4c",
                    "Gateway": "172.17.0.1",
                    "IPAddress": "172.17.0.4",
                    "IPPrefixLen": 16,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": "02:42:ac:11:00:04",
                    "DriverOpts": null
                }
            }
        }
    }
]


[
    {
        "Id": "5049a01b4358233471daa74fb11996a6b8e3acfd9c00b33ea7a52a3c24264eaa",
        "Created": "2018-03-12T09:37:38.9422449Z",
        "Path": "/sbin/tini",
        "Args": [
            "--",
            "/usr/local/bin/jenkins.sh"
        ],
        "State": {
            "Status": "running",
            "Running": true,
            "Paused": false,
            "Restarting": false,
            "OOMKilled": false,
            "Dead": false,
            "Pid": 22088,
            "ExitCode": 0,
            "Error": "",
            "StartedAt": "2018-03-13T09:33:50.029083326Z",
            "FinishedAt": "2018-03-13T09:27:43.347300762Z"
        },
        "Image": "sha256:6844ee63019e341fa7c06a90ce1455200bad2f919738d12e44eaff03198c91d0",
        "ResolvConfPath": "/var/lib/docker/containers/5049a01b4358233471daa74fb11996a6b8e3acfd9c00b33ea7a52a3c24264eaa/resolv.conf",
        "HostnamePath": "/var/lib/docker/containers/5049a01b4358233471daa74fb11996a6b8e3acfd9c00b33ea7a52a3c24264eaa/hostname",
        "HostsPath": "/var/lib/docker/containers/5049a01b4358233471daa74fb11996a6b8e3acfd9c00b33ea7a52a3c24264eaa/hosts",
        "LogPath": "/var/lib/docker/containers/5049a01b4358233471daa74fb11996a6b8e3acfd9c00b33ea7a52a3c24264eaa/5049a01b4358233471daa74fb11996a6b8e3acfd9c00b33ea7a52a3c24264eaa-json.log",
        "Name": "/wonderful_carson",
        "RestartCount": 0,
        "Driver": "overlay2",
        "Platform": "linux",
        "MountLabel": "",
        "ProcessLabel": "",
        "AppArmorProfile": "",
        "ExecIDs": null,
        "HostConfig": {
            "Binds": [
                "/home/docker/jenkins/jenkins_home:/var/jenkins_home",
                "/var/run/docker.sock:/var/run/docker.sock"
            ],
            "ContainerIDFile": "",
            "LogConfig": {
                "Type": "json-file",
                "Config": {}
            },
            "NetworkMode": "default",
            "PortBindings": {},
            "RestartPolicy": {
                "Name": "no",
                "MaximumRetryCount": 0
            },
            "AutoRemove": false,
            "VolumeDriver": "",
            "VolumesFrom": null,
            "CapAdd": null,
            "CapDrop": null,
            "Dns": [],
            "DnsOptions": [],
            "DnsSearch": [],
            "ExtraHosts": null,
            "GroupAdd": null,
            "IpcMode": "shareable",
            "Cgroup": "",
            "Links": null,
            "OomScoreAdj": 0,
            "PidMode": "",
            "Privileged": false,
            "PublishAllPorts": false,
            "ReadonlyRootfs": false,
            "SecurityOpt": null,
            "UTSMode": "",
            "UsernsMode": "",
            "ShmSize": 67108864,
            "Runtime": "runc",
            "ConsoleSize": [
                0,
                0
            ],
            "Isolation": "",
            "CpuShares": 0,
            "Memory": 0,
            "NanoCpus": 0,
            "CgroupParent": "",
            "BlkioWeight": 0,
            "BlkioWeightDevice": [],
            "BlkioDeviceReadBps": null,
            "BlkioDeviceWriteBps": null,
            "BlkioDeviceReadIOps": null,
            "BlkioDeviceWriteIOps": null,
            "CpuPeriod": 0,
            "CpuQuota": 0,
            "CpuRealtimePeriod": 0,
            "CpuRealtimeRuntime": 0,
            "CpusetCpus": "",
            "CpusetMems": "",
            "Devices": [],
            "DeviceCgroupRules": null,
            "DiskQuota": 0,
            "KernelMemory": 0,
            "MemoryReservation": 0,
            "MemorySwap": 0,
            "MemorySwappiness": null,
            "OomKillDisable": false,
            "PidsLimit": 0,
            "Ulimits": null,
            "CpuCount": 0,
            "CpuPercent": 0,
            "IOMaximumIOps": 0,
            "IOMaximumBandwidth": 0
        },
        "GraphDriver": {
            "Data": {
                "LowerDir": "/var/lib/docker/overlay2/b3474fead85ba8317d2f19aecb9d13128c3660d9fa4b4a0d785480477a8e627f-init/diff:/var/lib/docker/overlay2/c1db031b078fccca2c4ecb9c08f02a12a166a19fbbb07d709fc7f909e543224b/diff:/var/lib/docker/overlay2/e87a5e9585cf46649068fd2804e9b0077586798e8817b635e9cdb27d604c9de3/diff:/var/lib/docker/overlay2/3aaf41fa890140b773a497f75e1e230249c9a50ef9e637807046bd2645d47383/diff:/var/lib/docker/overlay2/b3042667faad116fa7c1fda0ade770be4b7f3d980ec0c1fdaa3b7141590bce1a/diff:/var/lib/docker/overlay2/061f3bf09589109b981a0b1e4d1e033eda2e69ef0b93ee5d7b7219288392c75e/diff:/var/lib/docker/overlay2/ff2306ca7c4b29c85b102e4274968da89c04335ecbb436c348a2129aadaf5554/diff:/var/lib/docker/overlay2/f82ff9ece6eac85cacb3b39129ea89afe9f59b4729c85a06af7277f357a1800e/diff:/var/lib/docker/overlay2/486e2412d67fc87781f0bd6890402915bf1270ab6b9553c603427e671a2bb01d/diff:/var/lib/docker/overlay2/5a80be543bbabb904d8bc75cdee2788f58988b86bbe6e0b7cafa4f23a1afd08f/diff:/var/lib/docker/overlay2/a80d6b1c1895a11ac66cdb30aabf17cfe912f05b87260d435a53b2199b90bae5/diff:/var/lib/docker/overlay2/ee85443de96d6090613ca21303bb168e3d11f717424fbb2c198e8d9ad315ca7a/diff:/var/lib/docker/overlay2/b4bb1975a154aa01a6d293e6346bece8fca12ecf2390153a1c9b236aec5df008/diff:/var/lib/docker/overlay2/b57c6c33340dae437b79a51f24524821796e8dee73e1cb5499834e8cc08f3e81/diff:/var/lib/docker/overlay2/0e0e4140d22655500ea9580a3f5a3bf69b4a6e3f1880bf0896406aeea17cdb8a/diff:/var/lib/docker/overlay2/d49d5d848055333df33351aeeb4c7a9ed967515ef433ee875d5dbfc6edb14c17/diff:/var/lib/docker/overlay2/a87e0ee4bf32e82943b4e38375131867e0fa6fb5a524ad2a4835223588785a9d/diff:/var/lib/docker/overlay2/a6f8827918230ce42c9988c190c229030a62426de6ebc6de1e905573ef60ae9e/diff:/var/lib/docker/overlay2/5c6b6e85d0f568307e7514e51b736c50a5dbf12f3d8ba971067f4acfa02609bb/diff:/var/lib/docker/overlay2/a79d301bc255744c54ccdaa9c440494dfadec50106b0e3d6d0f7461c7a490742/diff:/var/lib/docker/overlay2/f6f53d9320fee3231a646cbc71bdffc571162368ee123ff3270ba39b2d70691a/diff:/var/lib/docker/overlay2/5930cbcb8f8133c83755f1e2c509a8cd51e7b5ba9a76dc0c581973129d3aa35d/diff",
                "MergedDir": "/var/lib/docker/overlay2/b3474fead85ba8317d2f19aecb9d13128c3660d9fa4b4a0d785480477a8e627f/merged",
                "UpperDir": "/var/lib/docker/overlay2/b3474fead85ba8317d2f19aecb9d13128c3660d9fa4b4a0d785480477a8e627f/diff",
                "WorkDir": "/var/lib/docker/overlay2/b3474fead85ba8317d2f19aecb9d13128c3660d9fa4b4a0d785480477a8e627f/work"
            },
            "Name": "overlay2"
        },
        "Mounts": [
            {
                "Type": "bind",
                "Source": "/home/docker/jenkins/jenkins_home",
                "Destination": "/var/jenkins_home",
                "Mode": "",
                "RW": true,
                "Propagation": "rprivate"
            },
            {
                "Type": "bind",
                "Source": "/var/run/docker.sock",
                "Destination": "/var/run/docker.sock",
                "Mode": "",
                "RW": true,
                "Propagation": "rprivate"
            }
        ],
        "Config": {
            "Hostname": "5049a01b4358",
            "Domainname": "",
            "User": "jenkins",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "ExposedPorts": {
                "50000/tcp": {},
                "8080/tcp": {}
            },
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "[email protected]",
                "LETSENCRYPT_HOST=jenkins.miwashop.com",
                "VIRTUAL_HOST=jenkins.miwashop.com",
                "VIRTUAL_PORT=8080",
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "LANG=C.UTF-8",
                "JAVA_HOME=/docker-java-home",
                "JAVA_VERSION=8u151",
                "JAVA_DEBIAN_VERSION=8u151-b12-1~deb9u1",
                "CA_CERTIFICATES_JAVA_VERSION=20170531+nmu1",
                "JENKINS_HOME=/var/jenkins_home",
                "JENKINS_SLAVE_AGENT_PORT=50000",
                "JENKINS_VERSION=2.111",
                "JENKINS_UC=https://updates.jenkins.io",
                "JENKINS_UC_EXPERIMENTAL=https://updates.jenkins.io/experimental",
                "COPY_REFERENCE_FILE_LOG=/var/jenkins_home/copy_reference_file.log"
            ],
            "Cmd": null,
            "ArgsEscaped": true,
            "Image": "jenkins/jenkins:latest",
            "Volumes": {
                "/var/jenkins_home": {}
            },
            "WorkingDir": "",
            "Entrypoint": [
                "/sbin/tini",
                "--",
                "/usr/local/bin/jenkins.sh"
            ],
            "OnBuild": null,
            "Labels": {}
        },
        "NetworkSettings": {
            "Bridge": "",
            "SandboxID": "1dcd94febb0a76606f80777174e45b3fd7bfc4d90676b9b2a3bb2c3226aa2904",
            "HairpinMode": false,
            "LinkLocalIPv6Address": "",
            "LinkLocalIPv6PrefixLen": 0,
            "Ports": {
                "50000/tcp": null,
                "8080/tcp": null
            },
            "SandboxKey": "/var/run/docker/netns/1dcd94febb0a",
            "SecondaryIPAddresses": null,
            "SecondaryIPv6Addresses": null,
            "EndpointID": "1bd793b1aad29922934758049725101de03b9284bb48774d3fd6012d7cbbbb2b",
            "Gateway": "172.17.0.1",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "IPAddress": "172.17.0.3",
            "IPPrefixLen": 16,
            "IPv6Gateway": "",
            "MacAddress": "02:42:ac:11:00:03",
            "Networks": {
                "bridge": {
                    "IPAMConfig": null,
                    "Links": null,
                    "Aliases": null,
                    "NetworkID": "669f5d230c893729c633095f59ad01885c689028f4379aba3621f239f734e46b",
                    "EndpointID": "1bd793b1aad29922934758049725101de03b9284bb48774d3fd6012d7cbbbb2b",
                    "Gateway": "172.17.0.1",
                    "IPAddress": "172.17.0.3",
                    "IPPrefixLen": 16,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": "02:42:ac:11:00:03",
                    "DriverOpts": null
                }
            }
        }
    }
]

@ miwa911基于;

containerd version: N/A (expected: 9b55aab90508bd389d7654c4baf173a981477d55)

看来容器化已退出/重新启动。 我看到那台机器没有很多内存；

Total Memory: 992.3MiB

在测试期间是否可能是内存不足，而内核OOM杀死了某些进程？ 我建议检查守护程序和系统日志，以查看它们是否包含更多信息。

@thaJeztah谢谢thaJeztah，
我是新手，所以我不知道在哪里检查。 我已经附加了我的stacktrace（curl --unix-socket /var/run/docker.sock http：// localhost / debug / pprof / goroutine？debug = 2）。

goroutine_debug.log

我想我有一个相关的问题

上周，我将开发环境升级到了最新的17.12.1-ce，内部版本7390fc6，这是我第一次看到此错误。

我的开发人员尝试更新应用程序，并且swarm无法删除群集上特定节点上以前版本的旧容器。 我发现是因为开发人员开始间歇性地抱怨白页综合症。

当我在服务上执行docker service ps时，这就是我看到的内容： https :
旧容器正在运行，但处于关闭状态。

在节点上，我看到容器好像在以一种健康的方式运行：
https://www.screencast.com/t/ABKVYxNUQ

从“ docker service ls”中，我有更多的容器超出预期
https://www.screencast.com/t/0Po8Sqs0Jr

我尝试在节点上的容器上运行docker kill和docker inspect，但是它不起作用。
我在dmesg中没有任何特定的消息传递程序。

从现在开始，我能说的就是全部，我将删除堆栈并再次启动它，以便使开发人员能够继续他们的工作。

希望能帮助到你

编辑：

• Stack rm无法解决问题，僵尸容器仍在节点上
• 将节点可用性设置为耗尽并不能解决问题，节点上只剩下了僵尸容器
• 服务docker重新启动不响应
• 最终，重新启动了该节点，所有容器不再存在。

在此过程中，我在节点上看到了这样的错误

ar 13 10:04:10服务器名称dockerd：time =“ 2018-03-13T10：04：10.406196465-04：00” level = error msg =“加载容器失败/f5d6bb74d6b37871b72b5f27d46f8705a6b66cba7afb50706bbf68b​​764facb24/config.v2.json：无此类文件或目录”
3月13日10:04:10服务器名称dockerd：time =“ 2018-03-13T10：04：10.408039262-04：00” level = error msg =“加载容器失败/fd5ac869991b263a28c36bddf9b2847a8a26e2b7d59fa033f85e9616b0b7cb7a/config.v2.json：无此类文件或目录”

编辑2：
找到其他人也有同样的问题： https :

/ cc @stevvooe @dmcgowan ，以防您知道要从包装的一面寻找的东西

@mauriceteunissen我们有17.12.1-ce的问题

想知道是否使用Tini： All Tini does is spawn a single child (Tini is meant to be run in a container), and wait for it to exit all the while reaping zombies and performing signal forwarding.

https://github.com/krallin/tini这个问题和https://github.com/moby/moby/issues/34213将得到修复！

@loretoparisi对此表示怀疑。

但是您总是设置docker run --init来查看... --init使用tini。

@ cpuguy83我是说，由于在与Java有关的所有问题和发现（请参阅https://github.com/moby/moby/issues/34213）中，主要问题可以解决JVM中的僵尸进程（例如没有捕获未捕获的异常的Java多线程应用程序）和/或锁定的I / O资源（大多数是打开了某些套接字/文件的Java进程）或相同的问题，但是在通过child_process启动的某些C ++可执行文件上叉子。 但这只是一个假设，因为我无法以肯定的方式复制它...

我们在17.12.1-ce上也有同样的问题

随着时间的流逝，容器进入docker ps和docker inspect挂起的状态。
强制群重新部署服务会使容器进入僵尸状态（所需状态：正在关闭，当前状态：正在运行）。

docker kill不起作用。 杀死容器的一种方法是ps aux | grep [container_id] ，然后kill [process_id]

我是否需要提供任何信息？

@mhaamann如果

您可以通过点击/debug/pprof/goroutines?debug=2端点来完成此操作
例：

curl --unix-socket /var/run/docker.sock http://./debug/pprof/goroutines?debug=2

当然@ cpuguy83。
我将对其进行描述，以确保我做对了。 两个终端打开。
在第一个中，我执行了docker inspect [container_id] 。 等待几秒钟以确保其卡住。 （通常检查会在1秒内返回结果。 ）
然后在第二个终端中，我执行了：

curl --unix-socket /var/run/docker.sock http://./debug/pprof/goroutines?debug=2

跟踪可以在这里找到： https :

@mhaamann您可以运行以下命令来获取容器堆栈吗？

docker run -it --rm -v /run/docker/containerd:/run/docker/containerd docker:17.12.1 docker-containerd-ctr pprof --debug-socket /run/docker/containerd/docker-containerd-debug.sock goroutines

或者，如果您在主机上，则可以运行：

docker-containerd-ctr pprof --debug-socket /run/docker/containerd/docker-containerd-debug.sock goroutines

（顺便说一句）相关的固定goroutine似乎是对容器化的调用： https: //gist.github.com/mhaamann/f63ad6a0b8f70a8c47e030fae1cdd1f6#file -stack-trace-from-dockerd-L5213-L5241

这是使用以下命令从主机生成的堆栈： docker-containerd-ctr pprof --debug-socket /run/docker/containerd/docker-containerd-debug.sock goroutines

https://gist.github.com/mhaamann/215c3c6f77af89972b6eace4d09866fc

@mhaamann谢谢！ 深层发掘...

看起来好像卡在了填充程序中，无法获取容器的状态。
您是否可以在垫片上触发堆栈跟踪？ kill -SIGUSR1 ${PID_OF_SHIM}
这应该生成堆栈跟踪并传播到dockerd日志。
您应该能够确定pid是什么，因为它是容器进程的父进程。

@ cpuguy83

我们使用ps faux | grep docker找到垫片的父级。 在这里复制了相关部分：

root      2624  2.3  2.5 5621140 829884 ?      Ssl  Feb28 782:08 /usr/bin/dockerd
root      2634  0.3  0.1 1694516 47684 ?       Ssl  Feb28 105:32  \_ docker-containerd --config /var/run/docker/containerd/containerd.toml
...

然后我们执行了kill -SIGUSR1 2634 。 日志已附在此处：

https://gist.github.com/mhaamann/e245e9a8dedf264ae0343af1322bdf66

@mhaamann谢谢，但是我认为我们需要从docker-containerd-shim转储，而不是docker-containerd

我再次陷入那个问题。 这次尝试从17.12.1升级到18.03.0。 升级过程被卡住，大多数容器仍在运行（因为应用程序仍在运行，但是docker ps被卡住了）。

我已经完成了docker-containerd套接字的转储，这是要点： https :

我不知道如何进行docker-containerd-shim的转储。

@jcberthon谢谢，
要从docker-containerd-shim获得堆栈转储，请执行kill -s SIGUSR1 <docker-containerd-shim-pid> 。 这应该在dockerd的日志中生成堆栈跟踪。

@ cpuguy83大家好，我不得不重启主机（在看到您的消息之前），因为重启docker.service无效，并且终止进程也无助于随后重启容器。 因此，我经历了完整的重启周期，而不是摆弄直到我回到干净状态。

因此，我需要等待下一次锁定，才能报告docker-containerd-shim的堆栈转储。 我现在在18.03.0上...

无论如何，谢谢您迅速回到我身边:-)

我在17.12.1上已经21天了，此后我（包括今天）发生了两次。 我很少有时间进行故障排除，因为它运行在关键任务箱上，通常可以立即重启。 正如@jcberthon指出的那样，在这种情况下，重启

如果我可以获得需要运行的命令和需要转储的日志的确切顺序，则可以在下次发生时捕获它们，但是我还利用了重新启动并升级到18.03.0的需要。希望今天能为我解决这个问题。

docker -containerd-shim上的kill -SIGUSR1不会生成堆栈跟踪。 没发生什么事。

它应该在dockerd的日志中吗？

在2018年3月24日星期六，下午2：15，Matthias Haamann通知@ github.com
写道：

@ cpuguy83 https://github.com/cpuguy83在垫片上杀死-SIGUSR1
不生成堆栈跟踪。 没发生什么事。

-
您收到此邮件是因为有人提到您。
直接回复此电子邮件，在GitHub上查看
https://github.com/moby/moby/issues/35933#issuecomment-375913621或静音
线程
https://github.com/notifications/unsubscribe-auth/AAwxZtxfxj8i-AmbzffFqR1SSLIvYzbcks5tho1YgaJpZM4RTNnH
。

-

• 布莱恩·高夫

到目前为止，有人能在生产中正常运行17.12.1吗？

在过去（差不多）五天内（即当我升级到Docker CE 18.03.0时），我没有遇到此问题。

这并不意味着它已在18.03.0中解决，现在说还为时过早。 但是至少这种情况很少发生。 至少每隔2或3天出现问题。 🤞

@jcberthon真的对结果感到好奇。 原因我看到有人也对18.03.0有问题。 @JnMik我们决定降级到17.09.1，直到此问题解决为止，因为它在17.12和18.02上经常发生。

我们还坚持使用17.09.1，因为较新的版本不适用于我们。

有趣的是，对于我最初的问题，解决方案是更新到18.02。 至少到目前为止还算不错。

就在今天，我用docker 17.12.1创建了一个全新的集群，并且遇到了这个问题。
8小时后，在滚动更新服务时，我注意到一个容器卡在“运行中”，并且我不能docker inspect 。

这是来自容器垫片的转储，如@ cpuguy83所述获得。

https://gist.github.com/mion00/7ff75e04d6ac21de353cef3e8c64fcd1

编辑：从容器垫片，从另一个也被卡住的容器中更新（希望）正确的堆栈跟踪。

@ mion00看起来像是容器化的转储，而不是容器化的垫片:(

@ cpuguy83我更新了以前的注释，其中使用了从其他容器生成的新堆栈跟踪。

不幸的是，它又发生了，但是经过一个多星期的完美工作之后，与17.12.1。相比这是一次胜利。 抄送： @jordijansen

无论如何，我试图为docker-containerd-shim生成转储。 我已经列出了所有此类进程，并选择了一个进程，该进程对应于一个卡住的容器（我无法在此容器上执行docker日志，docker exec等）。 因此，我做了sudo kill -s SIGUSER1 <pid> ，但是，当我尝试检查dockerd的日志（使用sudo journalctl -u docker.service ）时，最后一条日志消息的日期已超过12小时。 所以我怀疑dockerd卡住了并且不再记录任何东西了...所以没有转储...除非某处有未知文件...

我已经用最新信息更新了要点（https://gist.github.com/jcberthon/143c3e6b7c9e5fc8f18c9204ca1bedf6）。 最后，我添加了我运行的命令，以尝试找出应该发送SIGUSR1信号的docker-containerd-shim。 我的卡住的容器正在运行MongoDB数据库，因此我寻找它的PID，然后使用proc FS，寻找它所属的cgroup。 通过使用cgroup ID，我确定了docker-containerd-shim PID。

@jcberthon我也有同样的问题，dockerd没有记录堆栈跟踪。 我解决了将"debug": true到/etc/docker/daemon.json的json配置中，并执行service docker reload来获取新配置的问题。 这将在dockerd中启用调试日志，然后您可以在kill -s SIGUSER1之后找到堆栈跟踪

谢谢@ mion00，确实是通过添加此选项并重新加载守护程序，我现在可以在执行sudo journalctl -u docker.service时看到一些日志

@ mion00 @jcberthon您想尝试一下https://github.com/moby/moby/pull/36751中的补丁，让我知道是否有帮助？

18.03.0仍受影响

在Mac的18.03.0上仍然有此问题...较不频繁...但仍然存在。

$ docker run -it --rm -v /run/docker/containerd:/run/docker/containerd docker:18.03.0 docker-containerd-ctr pprof --debug-socket /run/docker/containerd/docker-containerd-debug.sock goroutines
goroutine 764393 [running]:
runtime/pprof.writeGoroutineStacks(0x145e720, 0xc420741ce0, 0x0, 0xc420419ad0)
    /usr/local/go/src/runtime/pprof/pprof.go:608 +0xa9
runtime/pprof.writeGoroutine(0x145e720, 0xc420741ce0, 0x2, 0x30, 0x1026340)
    /usr/local/go/src/runtime/pprof/pprof.go:597 +0x46
runtime/pprof.(*Profile).WriteTo(0x1448700, 0x145e720, 0xc420741ce0, 0x2, 0xc420741ce0, 0xc4201b8750)
    /usr/local/go/src/runtime/pprof/pprof.go:310 +0x3ad
net/http/pprof.handler.ServeHTTP(0xc42052f4b1, 0x9, 0x1469220, 0xc420741ce0, 0xc4207e0000)
    /usr/local/go/src/net/http/pprof/pprof.go:237 +0x1ba
net/http/pprof.Index(0x1469220, 0xc420741ce0, 0xc4207e0000)
    /usr/local/go/src/net/http/pprof/pprof.go:248 +0x1dd
net/http.HandlerFunc.ServeHTTP(0x10b0ee8, 0x1469220, 0xc420741ce0, 0xc4207e0000)
    /usr/local/go/src/net/http/server.go:1918 +0x46
net/http.(*ServeMux).ServeHTTP(0xc4201b8750, 0x1469220, 0xc420741ce0, 0xc4207e0000)
    /usr/local/go/src/net/http/server.go:2254 +0x132
net/http.serverHandler.ServeHTTP(0xc4201c0410, 0x1469220, 0xc420741ce0, 0xc4207e0000)
    /usr/local/go/src/net/http/server.go:2619 +0xb6
net/http.(*conn).serve(0xc4206f6280, 0x1469fe0, 0xc4206e26c0)
    /usr/local/go/src/net/http/server.go:1801 +0x71f
created by net/http.(*Server).Serve
    /usr/local/go/src/net/http/server.go:2720 +0x28a

goroutine 1 [chan receive, 5006 minutes]:
main.main.func1(0xc4200aedc0, 0xc4200aedc0, 0xc4201adb4f)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/cmd/containerd/main.go:132 +0x871
github.com/containerd/containerd/vendor/github.com/urfave/cli.HandleAction(0xf9bc00, 0x10b0b60, 0xc4200aedc0, 0xc420058ba0, 0x0)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/github.com/urfave/cli/app.go:502 +0xd4
github.com/containerd/containerd/vendor/github.com/urfave/cli.(*App).Run(0xc420182a80, 0xc420010090, 0x3, 0x3, 0x0, 0x0)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/github.com/urfave/cli/app.go:268 +0x655
main.main()
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/cmd/containerd/main.go:135 +0x53d

goroutine 13 [select, 12 minutes]:
main.handleSignals.func1(0xc420058c60, 0xc420058c00, 0x146a0a0, 0xc4201739b0, 0xc420070420)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/cmd/containerd/main_unix.go:30 +0xf7
created by main.handleSignals
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/cmd/containerd/main_unix.go:27 +0x8b

goroutine 12 [syscall, 12 minutes]:
os/signal.signal_recv(0x14640a0)
    /usr/local/go/src/runtime/sigqueue.go:131 +0xa8
os/signal.loop()
    /usr/local/go/src/os/signal/signal_unix.go:22 +0x24
created by os/signal.init.0
    /usr/local/go/src/os/signal/signal_unix.go:28 +0x43

goroutine 14 [select, 5006 minutes, locked to thread]:
runtime.gopark(0x10b1368, 0x0, 0xba26f3, 0x6, 0x18, 0x1)
    /usr/local/go/src/runtime/proc.go:287 +0x132
runtime.selectgo(0xc420200f50, 0xc4200704e0)
    /usr/local/go/src/runtime/select.go:395 +0x114f
runtime.ensureSigM.func1()
    /usr/local/go/src/runtime/signal_unix.go:511 +0x226
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:2337 +0x1

goroutine 34 [select, 1 minutes]:
github.com/containerd/containerd/vendor/github.com/docker/go-events.(*Broadcaster).run(0xc4201a23c0)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/github.com/docker/go-events/broadcast.go:117 +0x414
created by github.com/containerd/containerd/vendor/github.com/docker/go-events.NewBroadcaster
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/github.com/docker/go-events/broadcast.go:39 +0x1b1

goroutine 35 [select, 1 minutes]:
github.com/containerd/containerd/gc/scheduler.(*gcScheduler).run(0xc4201ec3c0, 0x146a0a0, 0xc42024eb10)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/gc/scheduler/scheduler.go:243 +0x21d
created by github.com/containerd/containerd/gc/scheduler.init.0.func1
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/gc/scheduler/scheduler.go:107 +0x4bf

goroutine 36 [syscall, 1 minutes]:
syscall.Syscall6(0xe8, 0x5, 0xc4206439b8, 0x80, 0xffffffffffffffff, 0x0, 0x0, 0x1, 0x80, 0x0)
    /usr/local/go/src/syscall/asm_linux_amd64.s:44 +0x5
github.com/containerd/containerd/vendor/golang.org/x/sys/unix.EpollWait(0x5, 0xc4206439b8, 0x80, 0x80, 0xffffffffffffffff, 0x1, 0x0, 0x0)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/golang.org/x/sys/unix/zsyscall_linux_amd64.go:1518 +0x79
github.com/containerd/containerd/metrics/cgroups.(*oomCollector).start(0xc42020d3c0)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/metrics/cgroups/oom.go:98 +0x7d
created by github.com/containerd/containerd/metrics/cgroups.newOOMCollector
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/metrics/cgroups/oom.go:34 +0x125

goroutine 50 [IO wait, 1 minutes]:
internal/poll.runtime_pollWait(0x7f9762b56f70, 0x72, 0xffffffffffffffff)
    /usr/local/go/src/runtime/netpoll.go:173 +0x59
internal/poll.(*pollDesc).wait(0xc4202ee118, 0x72, 0xc420203b00, 0x0, 0x0)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0xb0
internal/poll.(*pollDesc).waitRead(0xc4202ee118, 0xffffffffffffff00, 0x0, 0x0)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3f
internal/poll.(*FD).Accept(0xc4202ee100, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/internal/poll/fd_unix.go:335 +0x1e4
net.(*netFD).accept(0xc4202ee100, 0xc4206f6300, 0xfb2160, 0xc420203d78)
    /usr/local/go/src/net/fd_unix.go:238 +0x44
net.(*UnixListener).accept(0xc4202c7920, 0x7f8e9a, 0x45ad10, 0xc420203dc0)
    /usr/local/go/src/net/unixsock_posix.go:162 +0x34
net.(*UnixListener).Accept(0xc4202c7920, 0x10b0d00, 0xc4206f6280, 0x146a0a0, 0xc4201b8840)
    /usr/local/go/src/net/unixsock.go:241 +0x4b
net/http.(*Server).Serve(0xc4201c0410, 0x1468fe0, 0xc4202c7920, 0x0, 0x0)
    /usr/local/go/src/net/http/server.go:2695 +0x1b4
net/http.Serve(0x1468fe0, 0xc4202c7920, 0x145e360, 0xc4201b8750, 0x10b0f00, 0xc420268720)
    /usr/local/go/src/net/http/server.go:2323 +0x75
github.com/containerd/containerd/server.(*Server).ServeDebug(0xc4201aef40, 0x1468fe0, 0xc4202c7920, 0xc420268738, 0x0)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/server/server.go:159 +0x1c8
github.com/containerd/containerd/server.(*Server).ServeDebug-fm(0x1468fe0, 0xc4202c7920, 0xc4202c7920, 0x0)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/cmd/containerd/main.go:115 +0x40
main.serve.func1(0x1468fe0, 0xc4202c7920, 0xc4202bb620, 0x146a0a0, 0xc4202c79e0, 0xc4202c0380, 0x37)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/cmd/containerd/main.go:146 +0x77
created by main.serve
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/cmd/containerd/main.go:144 +0x1c8

goroutine 51 [IO wait, 1 minutes]:
internal/poll.runtime_pollWait(0x7f9762b56eb0, 0x72, 0xffffffffffffffff)
    /usr/local/go/src/runtime/netpoll.go:173 +0x59
internal/poll.(*pollDesc).wait(0xc4202ee298, 0x72, 0xc420033b00, 0x0, 0x0)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0xb0
internal/poll.(*pollDesc).waitRead(0xc4202ee298, 0xffffffffffffff00, 0x0, 0x0)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3f
internal/poll.(*FD).Accept(0xc4202ee280, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/internal/poll/fd_unix.go:335 +0x1e4
net.(*netFD).accept(0xc4202ee280, 0xc4202b8008, 0x0, 0x0)
    /usr/local/go/src/net/fd_unix.go:238 +0x44
net.(*UnixListener).accept(0xc4202c7a70, 0x89339b, 0x45ad10, 0xc420033da0)
    /usr/local/go/src/net/unixsock_posix.go:162 +0x34
net.(*UnixListener).Accept(0xc4202c7a70, 0x10b07e8, 0xc4201f8140, 0x146d6c0, 0xc4202b8008)
    /usr/local/go/src/net/unixsock.go:241 +0x4b
github.com/containerd/containerd/vendor/google.golang.org/grpc.(*Server).Serve(0xc4201f8140, 0x1468fe0, 0xc4202c7a70, 0x0, 0x0)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/server.go:463 +0x198
github.com/containerd/containerd/server.(*Server).ServeGRPC(0xc4201aef40, 0x1468fe0, 0xc4202c7a70, 0xc420268f38, 0x0)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/server/server.go:138 +0x55
github.com/containerd/containerd/server.(*Server).ServeGRPC-fm(0x1468fe0, 0xc4202c7a70, 0xc4202c7a70, 0x0)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/cmd/containerd/main.go:129 +0x40
main.serve.func1(0x1468fe0, 0xc4202c7a70, 0xc4202bb730, 0x146a0a0, 0xc4202c7b30, 0xc4202c0480, 0x31)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/cmd/containerd/main.go:146 +0x77
created by main.serve
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/cmd/containerd/main.go:144 +0x1c8

goroutine 52 [IO wait, 1 minutes]:
internal/poll.runtime_pollWait(0x7f9762b56df0, 0x72, 0x0)
    /usr/local/go/src/runtime/netpoll.go:173 +0x59
internal/poll.(*pollDesc).wait(0xc4202ee618, 0x72, 0xffffffffffffff00, 0x1460960, 0x145adf0)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0xb0
internal/poll.(*pollDesc).waitRead(0xc4202ee618, 0xc4203f2000, 0x8000, 0x8000)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3f
internal/poll.(*FD).Read(0xc4202ee600, 0xc4203f2000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
    /usr/local/go/src/internal/poll/fd_unix.go:126 +0x18c
net.(*netFD).Read(0xc4202ee600, 0xc4203f2000, 0x8000, 0x8000, 0x11, 0x0, 0x0)
    /usr/local/go/src/net/fd_unix.go:202 +0x54
net.(*conn).Read(0xc4202b8028, 0xc4203f2000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/net.go:176 +0x6f
bufio.(*Reader).Read(0xc4202da660, 0xc4203021f8, 0x9, 0x9, 0x9, 0x0, 0x0)
    /usr/local/go/src/bufio/bufio.go:213 +0x30d
io.ReadAtLeast(0x145c760, 0xc4202da660, 0xc4203021f8, 0x9, 0x9, 0x9, 0x4acb760201fcbb8, 0x5ac36c5e, 0xc4201fcbc0)
    /usr/local/go/src/io/io.go:309 +0x88
io.ReadFull(0x145c760, 0xc4202da660, 0xc4203021f8, 0x9, 0x9, 0x111349c75d3a6, 0x14a7a60, 0xbea8f8f784a7d948)
    /usr/local/go/src/io/io.go:327 +0x5a
github.com/containerd/containerd/vendor/golang.org/x/net/http2.readFrameHeader(0xc4203021f8, 0x9, 0x9, 0x145c760, 0xc4202da660, 0x0, 0x7070e0900000000, 0xc4202e6798, 0xc4201fcce8)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/golang.org/x/net/http2/frame.go:237 +0x7d
github.com/containerd/containerd/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc4203021c0, 0xc4201c8ce0, 0xc4201c8ce0, 0x0, 0x0)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/golang.org/x/net/http2/frame.go:492 +0xa6
github.com/containerd/containerd/vendor/google.golang.org/grpc/transport.(*http2Server).HandleStreams(0xc420404000, 0xc4203ef9e0, 0x10b0820)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/transport/http2_server.go:393 +0x317
github.com/containerd/containerd/vendor/google.golang.org/grpc.(*Server).serveStreams(0xc4201f8140, 0x146d1e0, 0xc420404000)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/server.go:568 +0x142
github.com/containerd/containerd/vendor/google.golang.org/grpc.(*Server).serveHTTP2Transport(0xc4201f8140, 0x146d6c0, 0xc4202b8028, 0x0, 0x0)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/server.go:561 +0x473
github.com/containerd/containerd/vendor/google.golang.org/grpc.(*Server).handleRawConn(0xc4201f8140, 0x146d6c0, 0xc4202b8028)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/server.go:526 +0x499
created by github.com/containerd/containerd/vendor/google.golang.org/grpc.(*Server).Serve
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/server.go:495 +0x5bb

goroutine 53 [select, 1 minutes]:
github.com/containerd/containerd/vendor/google.golang.org/grpc/transport.loopyWriter(0x7f97623111a8, 0xc4203ed840, 0xc4203ef980, 0xc42041cfb8)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/transport/transport.go:750 +0x2e6
github.com/containerd/containerd/vendor/google.golang.org/grpc/transport.newHTTP2Server.func1(0xc420404000)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/transport/http2_server.go:227 +0x60
created by github.com/containerd/containerd/vendor/google.golang.org/grpc/transport.newHTTP2Server
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/transport/http2_server.go:226 +0x8fb

goroutine 54 [select, 88 minutes]:
github.com/containerd/containerd/vendor/google.golang.org/grpc/transport.(*http2Server).keepalive(0xc420404000)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/transport/http2_server.go:935 +0x266
created by github.com/containerd/containerd/vendor/google.golang.org/grpc/transport.newHTTP2Server
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/transport/http2_server.go:230 +0x920

goroutine 56 [select, 5006 minutes]:
github.com/containerd/containerd/services/events.(*service).Subscribe(0xc4201b0088, 0xc4203e93c0, 0x146d000, 0xc4203f0550, 0x0, 0x0)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/services/events/service.go:61 +0x213
github.com/containerd/containerd/api/services/events/v1._Events_Subscribe_Handler(0xff7e20, 0xc4201b0088, 0x146c580, 0xc4203e93a0, 0xc4202e6d20, 0xc4202a2000)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/api/services/events/v1/events.pb.go:288 +0x110
github.com/containerd/containerd/vendor/github.com/grpc-ecosystem/go-grpc-prometheus.StreamServerInterceptor(0xff7e20, 0xc4201b0088, 0x146c640, 0xc4202ee800, 0xc4203e9380, 0x10af408, 0x0, 0x0)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/github.com/grpc-ecosystem/go-grpc-prometheus/server.go:40 +0x13b
github.com/containerd/containerd/vendor/google.golang.org/grpc.(*Server).processStreamingRPC(0xc4201f8140, 0x146d1e0, 0xc4200aef20, 0xc4202f0400, 0xc4202c7470, 0x1447fc0, 0x0, 0x0, 0x0)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/server.go:940 +0x2ea
github.com/containerd/containerd/vendor/google.golang.org/grpc.(*Server).handleStream(0xc4201f8140, 0x146d1e0, 0xc4200aef20, 0xc4202f0400, 0x0)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/server.go:1027 +0x14c3
github.com/containerd/containerd/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc4200157f0, 0xc4201f8140, 0x146d1e0, 0xc4200aef20, 0xc4202f0400)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/server.go:572 +0xa1
created by github.com/containerd/containerd/vendor/google.golang.org/grpc.(*Server).serveStreams.func1
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/server.go:570 +0xa3

goroutine 15 [IO wait, 74 minutes]:
internal/poll.runtime_pollWait(0x7f9762b56d30, 0x72, 0x0)
    /usr/local/go/src/runtime/netpoll.go:173 +0x59
internal/poll.(*pollDesc).wait(0xc420012498, 0x72, 0xffffffffffffff00, 0x1460960, 0x145adf0)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0xb0
internal/poll.(*pollDesc).waitRead(0xc420012498, 0xc420428000, 0x8000, 0x8000)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3f
internal/poll.(*FD).Read(0xc420012480, 0xc420428000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
    /usr/local/go/src/internal/poll/fd_unix.go:126 +0x18c
net.(*netFD).Read(0xc420012480, 0xc420428000, 0x8000, 0x8000, 0x11, 0x0, 0x0)
    /usr/local/go/src/net/fd_unix.go:202 +0x54
net.(*conn).Read(0xc42000e138, 0xc420428000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/net.go:176 +0x6f
bufio.(*Reader).Read(0xc420058cc0, 0xc4200ac3b8, 0x9, 0x9, 0x9, 0x0, 0x0)
    /usr/local/go/src/bufio/bufio.go:213 +0x30d
io.ReadAtLeast(0x145c760, 0xc420058cc0, 0xc4200ac3b8, 0x9, 0x9, 0x9, 0xc420038bb8, 0x400f10, 0xc420038c67)
    /usr/local/go/src/io/io.go:309 +0x88
io.ReadFull(0x145c760, 0xc420058cc0, 0xc4200ac3b8, 0x9, 0x9, 0x83ce2d, 0xc42048414c, 0xc42043a000)
    /usr/local/go/src/io/io.go:327 +0x5a
github.com/containerd/containerd/vendor/golang.org/x/net/http2.readFrameHeader(0xc4200ac3b8, 0x9, 0x9, 0x145c760, 0xc420058cc0, 0x0, 0x0, 0xc420484140, 0xc420038ce8)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/golang.org/x/net/http2/frame.go:237 +0x7d
github.com/containerd/containerd/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc4200ac380, 0xc420484140, 0xc420484140, 0x0, 0x0)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/golang.org/x/net/http2/frame.go:492 +0xa6
github.com/containerd/containerd/vendor/google.golang.org/grpc/transport.(*http2Server).HandleStreams(0xc4200aef20, 0xc420173bc0, 0x10b0820)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/transport/http2_server.go:393 +0x317
github.com/containerd/containerd/vendor/google.golang.org/grpc.(*Server).serveStreams(0xc4201f8140, 0x146d1e0, 0xc4200aef20)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/server.go:568 +0x142
github.com/containerd/containerd/vendor/google.golang.org/grpc.(*Server).serveHTTP2Transport(0xc4201f8140, 0x146d6c0, 0xc42000e138, 0x0, 0x0)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/server.go:561 +0x473
github.com/containerd/containerd/vendor/google.golang.org/grpc.(*Server).handleRawConn(0xc4201f8140, 0x146d6c0, 0xc42000e138)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/server.go:526 +0x499
created by github.com/containerd/containerd/vendor/google.golang.org/grpc.(*Server).Serve
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/server.go:495 +0x5bb

goroutine 16 [select, 74 minutes]:
github.com/containerd/containerd/vendor/google.golang.org/grpc/transport.loopyWriter(0x7f97623111a8, 0xc420054cc0, 0xc420173b60, 0xc42041dfb8)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/transport/transport.go:750 +0x2e6
github.com/containerd/containerd/vendor/google.golang.org/grpc/transport.newHTTP2Server.func1(0xc4200aef20)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/transport/http2_server.go:227 +0x60
created by github.com/containerd/containerd/vendor/google.golang.org/grpc/transport.newHTTP2Server
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/transport/http2_server.go:226 +0x8fb

goroutine 66 [select, 74 minutes]:
github.com/containerd/containerd/vendor/google.golang.org/grpc/transport.(*http2Server).keepalive(0xc4200aef20)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/transport/http2_server.go:935 +0x266
created by github.com/containerd/containerd/vendor/google.golang.org/grpc/transport.newHTTP2Server
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/transport/http2_server.go:230 +0x920

goroutine 57 [semacquire, 5006 minutes]:
sync.runtime_notifyListWait(0xc4203edbd0, 0xc400000000)
    /usr/local/go/src/runtime/sema.go:507 +0x114
sync.(*Cond).Wait(0xc4203edbc0)
    /usr/local/go/src/sync/cond.go:56 +0x82
github.com/containerd/containerd/vendor/github.com/docker/go-events.(*Queue).next(0xc4204382d0, 0x0, 0x0)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/github.com/docker/go-events/queue.go:103 +0x87
github.com/containerd/containerd/vendor/github.com/docker/go-events.(*Queue).run(0xc4204382d0)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/github.com/docker/go-events/queue.go:68 +0x34
created by github.com/containerd/containerd/vendor/github.com/docker/go-events.NewQueue
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/github.com/docker/go-events/queue.go:29 +0x14d

goroutine 58 [select, 5006 minutes]:
github.com/containerd/containerd/events/exchange.(*Exchange).Subscribe.func3(0xc420438330, 0xc4203e93e0, 0xc42040a120, 0x1469fe0, 0xc4203edb80, 0xc4202da7e0)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/events/exchange/exchange.go:155 +0x15f
created by github.com/containerd/containerd/events/exchange.(*Exchange).Subscribe
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/events/exchange/exchange.go:149 +0x291

goroutine 67 [select, 1 minutes]:
github.com/containerd/containerd/services/events.(*service).Subscribe(0xc4201b0088, 0xc4203e9700, 0x146d000, 0xc4203f06f0, 0x0, 0x0)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/services/events/service.go:61 +0x213
github.com/containerd/containerd/api/services/events/v1._Events_Subscribe_Handler(0xff7e20, 0xc4201b0088, 0x146c580, 0xc4203e96e0, 0xc4202e7090, 0xc420029400)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/api/services/events/v1/events.pb.go:288 +0x110
github.com/containerd/containerd/vendor/github.com/grpc-ecosystem/go-grpc-prometheus.StreamServerInterceptor(0xff7e20, 0xc4201b0088, 0x146c640, 0xc4202eeb80, 0xc4203e96c0, 0x10af408, 0x0, 0xc420268ec8)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/github.com/grpc-ecosystem/go-grpc-prometheus/server.go:40 +0x13b
github.com/containerd/containerd/vendor/google.golang.org/grpc.(*Server).processStreamingRPC(0xc4201f8140, 0x146d1e0, 0xc420404840, 0xc42019a200, 0xc4202c7470, 0x1447fc0, 0x0, 0x0, 0x0)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/server.go:940 +0x2ea
github.com/containerd/containerd/vendor/google.golang.org/grpc.(*Server).handleStream(0xc4201f8140, 0x146d1e0, 0xc420404840, 0xc42019a200, 0x0)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/server.go:1027 +0x14c3
github.com/containerd/containerd/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc4202c8860, 0xc4201f8140, 0x146d1e0, 0xc420404840, 0xc42019a200)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/server.go:572 +0xa1
created by github.com/containerd/containerd/vendor/google.golang.org/grpc.(*Server).serveStreams.func1
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/server.go:570 +0xa3

goroutine 60 [IO wait, 1 minutes]:
internal/poll.runtime_pollWait(0x7f9762b56c70, 0x72, 0x0)
    /usr/local/go/src/runtime/netpoll.go:173 +0x59
internal/poll.(*pollDesc).wait(0xc4202ee918, 0x72, 0xffffffffffffff00, 0x1460960, 0x145adf0)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0xb0
internal/poll.(*pollDesc).waitRead(0xc4202ee918, 0xc420458000, 0x8000, 0x8000)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3f
internal/poll.(*FD).Read(0xc4202ee900, 0xc420458000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
    /usr/local/go/src/internal/poll/fd_unix.go:126 +0x18c
net.(*netFD).Read(0xc4202ee900, 0xc420458000, 0x8000, 0x8000, 0x11, 0x0, 0x0)
    /usr/local/go/src/net/fd_unix.go:202 +0x54
net.(*conn).Read(0xc4202b8040, 0xc420458000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/net.go:176 +0x6f
bufio.(*Reader).Read(0xc4202daae0, 0xc4203023b8, 0x9, 0x9, 0x9, 0x0, 0x0)
    /usr/local/go/src/bufio/bufio.go:213 +0x30d
io.ReadAtLeast(0x145c760, 0xc4202daae0, 0xc4203023b8, 0x9, 0x9, 0x9, 0x7add264201febb8, 0x5ac36c5e, 0xc4201febc0)
    /usr/local/go/src/io/io.go:309 +0x88
io.ReadFull(0x145c760, 0xc4202daae0, 0xc4203023b8, 0x9, 0x9, 0x111349f76e9fa, 0x14a7a60, 0xbea8f8f787ab9e54)
    /usr/local/go/src/io/io.go:327 +0x5a
github.com/containerd/containerd/vendor/golang.org/x/net/http2.readFrameHeader(0xc4203023b8, 0x9, 0x9, 0x145c760, 0xc4202daae0, 0x0, 0x7070e0900000000, 0xc4202e6fb8, 0xc4201fece8)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/golang.org/x/net/http2/frame.go:237 +0x7d
github.com/containerd/containerd/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc420302380, 0xc4202bf260, 0xc4202bf260, 0x0, 0x0)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/golang.org/x/net/http2/frame.go:492 +0xa6
github.com/containerd/containerd/vendor/google.golang.org/grpc/transport.(*http2Server).HandleStreams(0xc420404840, 0xc420438900, 0x10b0820)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/transport/http2_server.go:393 +0x317
github.com/containerd/containerd/vendor/google.golang.org/grpc.(*Server).serveStreams(0xc4201f8140, 0x146d1e0, 0xc420404840)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/server.go:568 +0x142
github.com/containerd/containerd/vendor/google.golang.org/grpc.(*Server).serveHTTP2Transport(0xc4201f8140, 0x146d6c0, 0xc4202b8040, 0x0, 0x0)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/server.go:561 +0x473
github.com/containerd/containerd/vendor/google.golang.org/grpc.(*Server).handleRawConn(0xc4201f8140, 0x146d6c0, 0xc4202b8040)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/server.go:526 +0x499
created by github.com/containerd/containerd/vendor/google.golang.org/grpc.(*Server).Serve
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/server.go:495 +0x5bb

goroutine 61 [select, 1 minutes]:
github.com/containerd/containerd/vendor/google.golang.org/grpc/transport.loopyWriter(0x7f97623111a8, 0xc4203eddc0, 0xc4204388a0, 0xc42046cfb8)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/transport/transport.go:750 +0x2e6
github.com/containerd/containerd/vendor/google.golang.org/grpc/transport.newHTTP2Server.func1(0xc420404840)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/transport/http2_server.go:227 +0x60
created by github.com/containerd/containerd/vendor/google.golang.org/grpc/transport.newHTTP2Server
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/transport/http2_server.go:226 +0x8fb

goroutine 62 [select, 74 minutes]:
github.com/containerd/containerd/vendor/google.golang.org/grpc/transport.(*http2Server).keepalive(0xc420404840)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/transport/http2_server.go:935 +0x266
created by github.com/containerd/containerd/vendor/google.golang.org/grpc/transport.newHTTP2Server
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/transport/http2_server.go:230 +0x920

goroutine 63 [semacquire, 1 minutes]:
sync.runtime_notifyListWait(0xc4203edf50, 0xc400009f35)
    /usr/local/go/src/runtime/sema.go:507 +0x114
sync.(*Cond).Wait(0xc4203edf40)
    /usr/local/go/src/sync/cond.go:56 +0x82
github.com/containerd/containerd/vendor/github.com/docker/go-events.(*Queue).next(0xc4204389c0, 0x0, 0x0)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/github.com/docker/go-events/queue.go:103 +0x87
github.com/containerd/containerd/vendor/github.com/docker/go-events.(*Queue).run(0xc4204389c0)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/github.com/docker/go-events/queue.go:68 +0x34
created by github.com/containerd/containerd/vendor/github.com/docker/go-events.NewQueue
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/github.com/docker/go-events/queue.go:29 +0x14d

goroutine 64 [select, 1 minutes]:
github.com/containerd/containerd/events/exchange.(*Exchange).Subscribe.func3(0xc420438a20, 0xc4203e9720, 0xc42040a3c0, 0x1469fe0, 0xc4203edf00, 0xc4202daf00)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/events/exchange/exchange.go:155 +0x15f
created by github.com/containerd/containerd/events/exchange.(*Exchange).Subscribe
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/events/exchange/exchange.go:149 +0x291

goroutine 592290 [select, 42 minutes]:
github.com/containerd/containerd/vendor/github.com/stevvooe/ttrpc.(*Client).run(0xc4201a6360)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/github.com/stevvooe/ttrpc/client.go:171 +0x342
created by github.com/containerd/containerd/vendor/github.com/stevvooe/ttrpc.NewClient
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/github.com/stevvooe/ttrpc/client.go:46 +0x2ab

goroutine 764394 [IO wait, 1 minutes]:
internal/poll.runtime_pollWait(0x7f9762b56970, 0x72, 0x0)
    /usr/local/go/src/runtime/netpoll.go:173 +0x59
internal/poll.(*pollDesc).wait(0xc4202ee698, 0x72, 0xffffffffffffff00, 0x1460960, 0x145adf0)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0xb0
internal/poll.(*pollDesc).waitRead(0xc4202ee698, 0xc4205a7600, 0x1, 0x1)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3f
internal/poll.(*FD).Read(0xc4202ee680, 0xc4205a7661, 0x1, 0x1, 0x0, 0x0, 0x0)
    /usr/local/go/src/internal/poll/fd_unix.go:126 +0x18c
net.(*netFD).Read(0xc4202ee680, 0xc4205a7661, 0x1, 0x1, 0x0, 0xc420545a00, 0x42b91b)
    /usr/local/go/src/net/fd_unix.go:202 +0x54
net.(*conn).Read(0xc42000e398, 0xc4205a7661, 0x1, 0x1, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/net.go:176 +0x6f
net/http.(*connReader).backgroundRead(0xc4205a7650)
    /usr/local/go/src/net/http/server.go:660 +0x64
created by net/http.(*connReader).startBackgroundRead
    /usr/local/go/src/net/http/server.go:656 +0xda

goroutine 671746 [select, 42 minutes]:
github.com/containerd/containerd/vendor/github.com/stevvooe/ttrpc.(*Client).dispatch(0xc4201a6360, 0x146a0a0, 0xc4204eee70, 0xc420345a80, 0xc4201daa60, 0x0, 0x0)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/github.com/stevvooe/ttrpc/client.go:102 +0x282
github.com/containerd/containerd/vendor/github.com/stevvooe/ttrpc.(*Client).Call(0xc4201a6360, 0x146a0a0, 0xc4204eee70, 0xbbc2cb, 0x25, 0xba0ec1, 0x5, 0x1046a60, 0xc4201a1ee0, 0x1046b40, ...)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/github.com/stevvooe/ttrpc/client.go:73 +0x15d
github.com/containerd/containerd/linux/shim/v1.(*shimClient).State(0xc42000e068, 0x146a0a0, 0xc4204eee70, 0xc4201a1ee0, 0x0, 0xc42016b298, 0xc42059f578)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/linux/shim/v1/shim.pb.go:1729 +0xbf
github.com/containerd/containerd/linux.(*Process).State(0xc4201daa40, 0x146a0a0, 0xc4204eee70, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/linux/process.go:45 +0xc7
github.com/containerd/containerd/services/tasks.processFromContainerd(0x7f9762311168, 0xc4204eee70, 0x146d0c0, 0xc4201daa40, 0x40, 0x146d0c0, 0xc4201daa40)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/services/tasks/service.go:225 +0xbe
github.com/containerd/containerd/services/tasks.(*service).Get(0xc4202c6e70, 0x7f9762311168, 0xc4204eee70, 0xc4201da800, 0xc4202c6e70, 0xbbd46b, 0x3)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/services/tasks/service.go:268 +0xef
github.com/containerd/containerd/api/services/tasks/v1._Tasks_Get_Handler.func1(0x7f9762311168, 0xc4204eee70, 0x10444c0, 0xc4201da800, 0xc420079ae0, 0x14cdcb0, 0xf87860, 0xc4201a1e30)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/api/services/tasks/v1/tasks.pb.go:680 +0x88
github.com/containerd/containerd/vendor/github.com/grpc-ecosystem/go-grpc-prometheus.UnaryServerInterceptor(0x7f9762311168, 0xc4204eee70, 0x10444c0, 0xc4201da800, 0xc4201da820, 0xc4201da840, 0x50, 0x48, 0xc4201da7e0, 0xc42059f968)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/github.com/grpc-ecosystem/go-grpc-prometheus/server.go:29 +0xd4
github.com/containerd/containerd/server.interceptor(0x7f9762311168, 0xc4204eedb0, 0x10444c0, 0xc4201da800, 0xc4201da820, 0xc4201da840, 0x4354a6, 0xc42059f9e0, 0x41228a, 0x50)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/server/server.go:267 +0x1d2
github.com/containerd/containerd/api/services/tasks/v1._Tasks_Get_Handler(0x1097ee0, 0xc4202c6e70, 0x7f9762311168, 0xc42044fbc0, 0xc420079900, 0x10af920, 0x0, 0x0, 0xc42001d900, 0x46)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/api/services/tasks/v1/tasks.pb.go:682 +0x16f
github.com/containerd/containerd/vendor/google.golang.org/grpc.(*Server).processUnaryRPC(0xc4201f8140, 0x146d1e0, 0xc420404840, 0xc42021d700, 0xc4202c77d0, 0x1452dc0, 0x0, 0x0, 0x0)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/server.go:826 +0xab6
github.com/containerd/containerd/vendor/google.golang.org/grpc.(*Server).handleStream(0xc4201f8140, 0x146d1e0, 0xc420404840, 0xc42021d700, 0x0)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/server.go:1023 +0x152a
github.com/containerd/containerd/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc4202c8860, 0xc4201f8140, 0x146d1e0, 0xc420404840, 0xc42021d700)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/server.go:572 +0xa1
created by github.com/containerd/containerd/vendor/google.golang.org/grpc.(*Server).serveStreams.func1
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/server.go:570 +0xa3

goroutine 764368 [syscall, 1 minutes]:
syscall.Syscall6(0xf7, 0x1, 0x36fe, 0xc42049d5b8, 0x1000004, 0x0, 0x0, 0x146d1e0, 0xc420404840, 0xc4204f9c00)
    /usr/local/go/src/syscall/asm_linux_amd64.s:44 +0x5
os.(*Process).blockUntilWaitable(0xc4207ebe90, 0x0, 0xc42049d6b0, 0x5268aa)
    /usr/local/go/src/os/wait_waitid.go:31 +0xa7
os.(*Process).wait(0xc4207ebe90, 0xc42049d770, 0x8943d5, 0x146a0a0)
    /usr/local/go/src/os/exec_unix.go:22 +0x44
os.(*Process).Wait(0xc4207ebe90, 0xc420523400, 0x42b91b, 0xc400000008)
    /usr/local/go/src/os/exec.go:115 +0x2d
os/exec.(*Cmd).Wait(0xc4203138c0, 0x45e611, 0xc4202c8860)
    /usr/local/go/src/os/exec/exec.go:446 +0x64
github.com/containerd/containerd/linux/shim/client.WithStart.func1.2(0xc4203138c0, 0xc420568320)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/linux/shim/client/client.go:59 +0x2d
created by github.com/containerd/containerd/linux/shim/client.WithStart.func1
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/linux/shim/client/client.go:58 +0x2d5

goroutine 592291 [IO wait, 42 minutes]:
internal/poll.runtime_pollWait(0x7f9762b56af0, 0x72, 0x0)
    /usr/local/go/src/runtime/netpoll.go:173 +0x59
internal/poll.(*pollDesc).wait(0xc4201b4398, 0x72, 0xffffffffffffff00, 0x1460960, 0x145adf0)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0xb0
internal/poll.(*pollDesc).waitRead(0xc4201b4398, 0xc4202ea000, 0x1000, 0x1000)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3f
internal/poll.(*FD).Read(0xc4201b4380, 0xc4202ea000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /usr/local/go/src/internal/poll/fd_unix.go:126 +0x18c
net.(*netFD).Read(0xc4201b4380, 0xc4202ea000, 0x1000, 0x1000, 0xc420201e78, 0x43ebf4, 0xc42040a720)
    /usr/local/go/src/net/fd_unix.go:202 +0x54
net.(*conn).Read(0xc42000e060, 0xc4202ea000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/net.go:176 +0x6f
bufio.(*Reader).Read(0xc4201a6300, 0xc4202fa060, 0xa, 0xa, 0xc420201fac, 0x0, 0xc420201fa8)
    /usr/local/go/src/bufio/bufio.go:213 +0x30d
io.ReadAtLeast(0x145c760, 0xc4201a6300, 0xc4202fa060, 0xa, 0xa, 0xa, 0xc42021e300, 0x0, 0xc420201f48)
    /usr/local/go/src/io/io.go:309 +0x88
io.ReadFull(0x145c760, 0xc4201a6300, 0xc4202fa060, 0xa, 0xa, 0x2, 0x2, 0x0)
    /usr/local/go/src/io/io.go:327 +0x5a
github.com/containerd/containerd/vendor/github.com/stevvooe/ttrpc.readMessageHeader(0xc4202fa060, 0xa, 0xa, 0x145c760, 0xc4201a6300, 0xc420201f48, 0x2, 0x2, 0xc420201fa8)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/github.com/stevvooe/ttrpc/channel.go:38 +0x62
github.com/containerd/containerd/vendor/github.com/stevvooe/ttrpc.(*channel).recv(0xc4202fa040, 0x146a020, 0xc420014048, 0x0, 0x2, 0xc4201b6070, 0x6, 0x6, 0x0, 0x0)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/github.com/stevvooe/ttrpc/channel.go:86 +0x6f
github.com/containerd/containerd/vendor/github.com/stevvooe/ttrpc.(*Client).run.func1(0xc42040a780, 0xc4201a6360, 0xc42003ba20, 0xc42040a720)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/github.com/stevvooe/ttrpc/client.go:144 +0x9a
created by github.com/containerd/containerd/vendor/github.com/stevvooe/ttrpc.(*Client).run
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/github.com/stevvooe/ttrpc/client.go:139 +0x164

goroutine 764369 [select, 1 minutes]:
github.com/containerd/containerd/vendor/github.com/stevvooe/ttrpc.(*Client).run(0xc42065c540)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/github.com/stevvooe/ttrpc/client.go:171 +0x342
created by github.com/containerd/containerd/vendor/github.com/stevvooe/ttrpc.NewClient
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/github.com/stevvooe/ttrpc/client.go:46 +0x2ab

goroutine 671647 [select, 42 minutes]:
github.com/containerd/containerd/vendor/github.com/stevvooe/ttrpc.(*Client).dispatch(0xc4201a6360, 0x146a0a0, 0xc42024e5a0, 0xc4203440c0, 0xc420306260, 0x0, 0x0)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/github.com/stevvooe/ttrpc/client.go:102 +0x282
github.com/containerd/containerd/vendor/github.com/stevvooe/ttrpc.(*Client).Call(0xc4201a6360, 0x146a0a0, 0xc42024e5a0, 0xbbc2cb, 0x25, 0xba0ebc, 0x5, 0x10468a0, 0xc4201ae640, 0x1046980, ...)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/github.com/stevvooe/ttrpc/client.go:73 +0x15d
github.com/containerd/containerd/linux/shim/v1.(*shimClient).Start(0xc42000e068, 0x146a0a0, 0xc42024e5a0, 0xc4201ae640, 0x1001ae0, 0x7f9762311101, 0xc420571660)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/linux/shim/v1/shim.pb.go:1745 +0xbf
github.com/containerd/containerd/linux.(*Process).Start(0xc4203061e0, 0x146a0a0, 0xc42024e5a0, 0x146a0a0, 0xc42024e5a0)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/linux/process.go:109 +0x91
github.com/containerd/containerd/services/tasks.(*service).Start(0xc4202c6e70, 0x7f9762311168, 0xc42024e5a0, 0xc42000c560, 0xc4202c6e70, 0xbbea93, 0x5)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/services/tasks/service.go:175 +0x14e
github.com/containerd/containerd/api/services/tasks/v1._Tasks_Start_Handler.func1(0x7f9762311168, 0xc42024e5a0, 0x1044f40, 0xc42000c560, 0xc4202f64b0, 0x14cdcb0, 0xf87860, 0xc4201ae630)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/api/services/tasks/v1/tasks.pb.go:626 +0x8b
github.com/containerd/containerd/vendor/github.com/grpc-ecosystem/go-grpc-prometheus.UnaryServerInterceptor(0x7f9762311168, 0xc42024e5a0, 0x1044f40, 0xc42000c560, 0xc42000c580, 0xc42000c5a0, 0x50, 0x48, 0xc42000c540, 0xc420571968)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/github.com/grpc-ecosystem/go-grpc-prometheus/server.go:29 +0xd4
github.com/containerd/containerd/server.interceptor(0x7f9762311168, 0xc42024e4b0, 0x1044f40, 0xc42000c560, 0xc42000c580, 0xc42000c5a0, 0x8000000000000000, 0xc4205719e0, 0x41228a, 0x50)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/server/server.go:267 +0x1d2
github.com/containerd/containerd/api/services/tasks/v1._Tasks_Start_Handler(0x1097ee0, 0xc4202c6e70, 0x7f9762311168, 0xc420471a70, 0xc4202f62d0, 0x10af920, 0x0, 0x0, 0x0, 0x145e9e0)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/api/services/tasks/v1/tasks.pb.go:628 +0x16f
github.com/containerd/containerd/vendor/google.golang.org/grpc.(*Server).processUnaryRPC(0xc4201f8140, 0x146d1e0, 0xc420404840, 0xc4202f0000, 0xc4202c77d0, 0x1452d78, 0x0, 0x0, 0x0)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/server.go:826 +0xab6
github.com/containerd/containerd/vendor/google.golang.org/grpc.(*Server).handleStream(0xc4201f8140, 0x146d1e0, 0xc420404840, 0xc4202f0000, 0x0)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/server.go:1023 +0x152a
github.com/containerd/containerd/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc4202c8860, 0xc4201f8140, 0x146d1e0, 0xc420404840, 0xc4202f0000)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/server.go:572 +0xa1
created by github.com/containerd/containerd/vendor/google.golang.org/grpc.(*Server).serveStreams.func1
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/server.go:570 +0xa3

goroutine 592225 [syscall, 68 minutes]:
syscall.Syscall6(0xf7, 0x1, 0xb0e, 0xc42026a5b8, 0x1000004, 0x0, 0x0, 0x146d1e0, 0xc420404840, 0xc4202f5000)
    /usr/local/go/src/syscall/asm_linux_amd64.s:44 +0x5
os.(*Process).blockUntilWaitable(0xc420016d50, 0x0, 0xc42026a6b0, 0x5268aa)
    /usr/local/go/src/os/wait_waitid.go:31 +0xa7
os.(*Process).wait(0xc420016d50, 0xc42026a770, 0x8943d5, 0x146a0a0)
    /usr/local/go/src/os/exec_unix.go:22 +0x44
os.(*Process).Wait(0xc420016d50, 0xc42034b300, 0x42b91b, 0xc400000008)
    /usr/local/go/src/os/exec.go:115 +0x2d
os/exec.(*Cmd).Wait(0xc420262160, 0x45e611, 0xc4202c8860)
    /usr/local/go/src/os/exec/exec.go:446 +0x64
github.com/containerd/containerd/linux/shim/client.WithStart.func1.2(0xc420262160, 0xc420440280)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/linux/shim/client/client.go:59 +0x2d
created by github.com/containerd/containerd/linux/shim/client.WithStart.func1
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/linux/shim/client/client.go:58 +0x2d5

goroutine 764402 [IO wait, 1 minutes]:
internal/poll.runtime_pollWait(0x7f9762b56bb0, 0x72, 0x0)
    /usr/local/go/src/runtime/netpoll.go:173 +0x59
internal/poll.(*pollDesc).wait(0xc4201b4898, 0x72, 0xffffffffffffff00, 0x1460960, 0x145adf0)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0xb0
internal/poll.(*pollDesc).waitRead(0xc4201b4898, 0xc42073d000, 0x1000, 0x1000)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3f
internal/poll.(*FD).Read(0xc4201b4880, 0xc42073d000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /usr/local/go/src/internal/poll/fd_unix.go:126 +0x18c
net.(*netFD).Read(0xc4201b4880, 0xc42073d000, 0x1000, 0x1000, 0xc42041ee78, 0x43ebf4, 0xc4201be7e0)
    /usr/local/go/src/net/fd_unix.go:202 +0x54
net.(*conn).Read(0xc4201b0370, 0xc42073d000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/net.go:176 +0x6f
bufio.(*Reader).Read(0xc42065c4e0, 0xc420774420, 0xa, 0xa, 0xc42041efac, 0x0, 0xc42041efa8)
    /usr/local/go/src/bufio/bufio.go:213 +0x30d
io.ReadAtLeast(0x145c760, 0xc42065c4e0, 0xc420774420, 0xa, 0xa, 0xa, 0xc4201f6100, 0x0, 0xc42041ef48)
    /usr/local/go/src/io/io.go:309 +0x88
io.ReadFull(0x145c760, 0xc42065c4e0, 0xc420774420, 0xa, 0xa, 0x2, 0x2, 0x0)
    /usr/local/go/src/io/io.go:327 +0x5a
github.com/containerd/containerd/vendor/github.com/stevvooe/ttrpc.readMessageHeader(0xc420774420, 0xa, 0xa, 0x145c760, 0xc42065c4e0, 0xc42041ef48, 0x2, 0x2, 0xc42041efa8)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/github.com/stevvooe/ttrpc/channel.go:38 +0x62
github.com/containerd/containerd/vendor/github.com/stevvooe/ttrpc.(*channel).recv(0xc420774400, 0x146a020, 0xc420014048, 0x0, 0x2, 0xc4204a7060, 0x6, 0xb, 0x0, 0x0)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/github.com/stevvooe/ttrpc/channel.go:86 +0x6f
github.com/containerd/containerd/vendor/github.com/stevvooe/ttrpc.(*Client).run.func1(0xc4201be840, 0xc42065c540, 0xc420265220, 0xc4201be7e0)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/github.com/stevvooe/ttrpc/client.go:144 +0x9a
created by github.com/containerd/containerd/vendor/github.com/stevvooe/ttrpc.(*Client).run
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/github.com/stevvooe/ttrpc/client.go:139 +0x164

再说一次（以便更容易地滤除卡住的瞬态）

$ docker run -it --rm -v /run/docker/containerd:/run/docker/containerd docker:18.03.0 docker-containerd-ctr pprof --debug-socket /run/docker/containerd/docker-containerd-debug.sock goroutines
goroutine 764428 [running]:
runtime/pprof.writeGoroutineStacks(0x145e720, 0xc4200ac7e0, 0x0, 0xc4205ccad0)
    /usr/local/go/src/runtime/pprof/pprof.go:608 +0xa9
runtime/pprof.writeGoroutine(0x145e720, 0xc4200ac7e0, 0x2, 0x30, 0x1026340)
    /usr/local/go/src/runtime/pprof/pprof.go:597 +0x46
runtime/pprof.(*Profile).WriteTo(0x1448700, 0x145e720, 0xc4200ac7e0, 0x2, 0xc4200ac7e0, 0xc4201b8750)
    /usr/local/go/src/runtime/pprof/pprof.go:310 +0x3ad
net/http/pprof.handler.ServeHTTP(0xc4205620d1, 0x9, 0x1469220, 0xc4200ac7e0, 0xc42057b200)
    /usr/local/go/src/net/http/pprof/pprof.go:237 +0x1ba
net/http/pprof.Index(0x1469220, 0xc4200ac7e0, 0xc42057b200)
    /usr/local/go/src/net/http/pprof/pprof.go:248 +0x1dd
net/http.HandlerFunc.ServeHTTP(0x10b0ee8, 0x1469220, 0xc4200ac7e0, 0xc42057b200)
    /usr/local/go/src/net/http/server.go:1918 +0x46
net/http.(*ServeMux).ServeHTTP(0xc4201b8750, 0x1469220, 0xc4200ac7e0, 0xc42057b200)
    /usr/local/go/src/net/http/server.go:2254 +0x132
net/http.serverHandler.ServeHTTP(0xc4201c0410, 0x1469220, 0xc4200ac7e0, 0xc42057b200)
    /usr/local/go/src/net/http/server.go:2619 +0xb6
net/http.(*conn).serve(0xc4206328c0, 0x1469fe0, 0xc4202fa300)
    /usr/local/go/src/net/http/server.go:1801 +0x71f
created by net/http.(*Server).Serve
    /usr/local/go/src/net/http/server.go:2720 +0x28a

goroutine 1 [chan receive, 5006 minutes]:
main.main.func1(0xc4200aedc0, 0xc4200aedc0, 0xc4201adb4f)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/cmd/containerd/main.go:132 +0x871
github.com/containerd/containerd/vendor/github.com/urfave/cli.HandleAction(0xf9bc00, 0x10b0b60, 0xc4200aedc0, 0xc420058ba0, 0x0)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/github.com/urfave/cli/app.go:502 +0xd4
github.com/containerd/containerd/vendor/github.com/urfave/cli.(*App).Run(0xc420182a80, 0xc420010090, 0x3, 0x3, 0x0, 0x0)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/github.com/urfave/cli/app.go:268 +0x655
main.main()
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/cmd/containerd/main.go:135 +0x53d

goroutine 13 [select, 12 minutes]:
main.handleSignals.func1(0xc420058c60, 0xc420058c00, 0x146a0a0, 0xc4201739b0, 0xc420070420)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/cmd/containerd/main_unix.go:30 +0xf7
created by main.handleSignals
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/cmd/containerd/main_unix.go:27 +0x8b

goroutine 12 [syscall, 12 minutes]:
os/signal.signal_recv(0x14640a0)
    /usr/local/go/src/runtime/sigqueue.go:131 +0xa8
os/signal.loop()
    /usr/local/go/src/os/signal/signal_unix.go:22 +0x24
created by os/signal.init.0
    /usr/local/go/src/os/signal/signal_unix.go:28 +0x43

goroutine 14 [select, 5006 minutes, locked to thread]:
runtime.gopark(0x10b1368, 0x0, 0xba26f3, 0x6, 0x18, 0x1)
    /usr/local/go/src/runtime/proc.go:287 +0x132
runtime.selectgo(0xc420200f50, 0xc4200704e0)
    /usr/local/go/src/runtime/select.go:395 +0x114f
runtime.ensureSigM.func1()
    /usr/local/go/src/runtime/signal_unix.go:511 +0x226
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:2337 +0x1

goroutine 34 [select]:
github.com/containerd/containerd/vendor/github.com/docker/go-events.(*Broadcaster).run(0xc4201a23c0)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/github.com/docker/go-events/broadcast.go:117 +0x414
created by github.com/containerd/containerd/vendor/github.com/docker/go-events.NewBroadcaster
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/github.com/docker/go-events/broadcast.go:39 +0x1b1

goroutine 35 [select]:
github.com/containerd/containerd/gc/scheduler.(*gcScheduler).run(0xc4201ec3c0, 0x146a0a0, 0xc42024eb10)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/gc/scheduler/scheduler.go:243 +0x21d
created by github.com/containerd/containerd/gc/scheduler.init.0.func1
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/gc/scheduler/scheduler.go:107 +0x4bf

goroutine 36 [syscall]:
syscall.Syscall6(0xe8, 0x5, 0xc4206439b8, 0x80, 0xffffffffffffffff, 0x0, 0x0, 0x1, 0x80, 0x0)
    /usr/local/go/src/syscall/asm_linux_amd64.s:44 +0x5
github.com/containerd/containerd/vendor/golang.org/x/sys/unix.EpollWait(0x5, 0xc4206439b8, 0x80, 0x80, 0xffffffffffffffff, 0x1, 0x0, 0x0)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/golang.org/x/sys/unix/zsyscall_linux_amd64.go:1518 +0x79
github.com/containerd/containerd/metrics/cgroups.(*oomCollector).start(0xc42020d3c0)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/metrics/cgroups/oom.go:98 +0x7d
created by github.com/containerd/containerd/metrics/cgroups.newOOMCollector
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/metrics/cgroups/oom.go:34 +0x125

goroutine 50 [IO wait]:
internal/poll.runtime_pollWait(0x7f9762b56f70, 0x72, 0xffffffffffffffff)
    /usr/local/go/src/runtime/netpoll.go:173 +0x59
internal/poll.(*pollDesc).wait(0xc4202ee118, 0x72, 0xc420203b00, 0x0, 0x0)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0xb0
internal/poll.(*pollDesc).waitRead(0xc4202ee118, 0xffffffffffffff00, 0x0, 0x0)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3f
internal/poll.(*FD).Accept(0xc4202ee100, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/internal/poll/fd_unix.go:335 +0x1e4
net.(*netFD).accept(0xc4202ee100, 0xc420632940, 0xfb2160, 0xc420203d78)
    /usr/local/go/src/net/fd_unix.go:238 +0x44
net.(*UnixListener).accept(0xc4202c7920, 0x7f8e9a, 0x45ad10, 0xc420203dc0)
    /usr/local/go/src/net/unixsock_posix.go:162 +0x34
net.(*UnixListener).Accept(0xc4202c7920, 0x10b0d00, 0xc4206328c0, 0x146a0a0, 0xc4201b8840)
    /usr/local/go/src/net/unixsock.go:241 +0x4b
net/http.(*Server).Serve(0xc4201c0410, 0x1468fe0, 0xc4202c7920, 0x0, 0x0)
    /usr/local/go/src/net/http/server.go:2695 +0x1b4
net/http.Serve(0x1468fe0, 0xc4202c7920, 0x145e360, 0xc4201b8750, 0x10b0f00, 0xc420268720)
    /usr/local/go/src/net/http/server.go:2323 +0x75
github.com/containerd/containerd/server.(*Server).ServeDebug(0xc4201aef40, 0x1468fe0, 0xc4202c7920, 0xc420268738, 0x0)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/server/server.go:159 +0x1c8
github.com/containerd/containerd/server.(*Server).ServeDebug-fm(0x1468fe0, 0xc4202c7920, 0xc4202c7920, 0x0)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/cmd/containerd/main.go:115 +0x40
main.serve.func1(0x1468fe0, 0xc4202c7920, 0xc4202bb620, 0x146a0a0, 0xc4202c79e0, 0xc4202c0380, 0x37)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/cmd/containerd/main.go:146 +0x77
created by main.serve
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/cmd/containerd/main.go:144 +0x1c8

goroutine 51 [IO wait]:
internal/poll.runtime_pollWait(0x7f9762b56eb0, 0x72, 0xffffffffffffffff)
    /usr/local/go/src/runtime/netpoll.go:173 +0x59
internal/poll.(*pollDesc).wait(0xc4202ee298, 0x72, 0xc420033b00, 0x0, 0x0)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0xb0
internal/poll.(*pollDesc).waitRead(0xc4202ee298, 0xffffffffffffff00, 0x0, 0x0)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3f
internal/poll.(*FD).Accept(0xc4202ee280, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/internal/poll/fd_unix.go:335 +0x1e4
net.(*netFD).accept(0xc4202ee280, 0xc4202b8000, 0x0, 0x0)
    /usr/local/go/src/net/fd_unix.go:238 +0x44
net.(*UnixListener).accept(0xc4202c7a70, 0x89339b, 0x45ad10, 0xc420033da0)
    /usr/local/go/src/net/unixsock_posix.go:162 +0x34
net.(*UnixListener).Accept(0xc4202c7a70, 0x10b07e8, 0xc4201f8140, 0x146d6c0, 0xc4202b8000)
    /usr/local/go/src/net/unixsock.go:241 +0x4b
github.com/containerd/containerd/vendor/google.golang.org/grpc.(*Server).Serve(0xc4201f8140, 0x1468fe0, 0xc4202c7a70, 0x0, 0x0)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/server.go:463 +0x198
github.com/containerd/containerd/server.(*Server).ServeGRPC(0xc4201aef40, 0x1468fe0, 0xc4202c7a70, 0xc420268f38, 0x0)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/server/server.go:138 +0x55
github.com/containerd/containerd/server.(*Server).ServeGRPC-fm(0x1468fe0, 0xc4202c7a70, 0xc4202c7a70, 0x0)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/cmd/containerd/main.go:129 +0x40
main.serve.func1(0x1468fe0, 0xc4202c7a70, 0xc4202bb730, 0x146a0a0, 0xc4202c7b30, 0xc4202c0480, 0x31)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/cmd/containerd/main.go:146 +0x77
created by main.serve
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/cmd/containerd/main.go:144 +0x1c8

goroutine 52 [IO wait]:
internal/poll.runtime_pollWait(0x7f9762b56df0, 0x72, 0x0)
    /usr/local/go/src/runtime/netpoll.go:173 +0x59
internal/poll.(*pollDesc).wait(0xc4202ee618, 0x72, 0xffffffffffffff00, 0x1460960, 0x145adf0)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0xb0
internal/poll.(*pollDesc).waitRead(0xc4202ee618, 0xc4203f2000, 0x8000, 0x8000)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3f
internal/poll.(*FD).Read(0xc4202ee600, 0xc4203f2000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
    /usr/local/go/src/internal/poll/fd_unix.go:126 +0x18c
net.(*netFD).Read(0xc4202ee600, 0xc4203f2000, 0x8000, 0x8000, 0x11, 0x0, 0x0)
    /usr/local/go/src/net/fd_unix.go:202 +0x54
net.(*conn).Read(0xc4202b8028, 0xc4203f2000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/net.go:176 +0x6f
bufio.(*Reader).Read(0xc4202da660, 0xc4203021f8, 0x9, 0x9, 0x9, 0x0, 0x0)
    /usr/local/go/src/bufio/bufio.go:213 +0x30d
io.ReadAtLeast(0x145c760, 0xc4202da660, 0xc4203021f8, 0x9, 0x9, 0x9, 0x227a0234201fcbb8, 0x5ac36c69, 0xc4201fcbc0)
    /usr/local/go/src/io/io.go:309 +0x88
io.ReadFull(0x145c760, 0xc4202da660, 0xc4203021f8, 0x9, 0x9, 0x1113749e9ca22, 0x14a7a60, 0xbea8f8fa62751fd0)
    /usr/local/go/src/io/io.go:327 +0x5a
github.com/containerd/containerd/vendor/golang.org/x/net/http2.readFrameHeader(0xc4203021f8, 0x9, 0x9, 0x145c760, 0xc4202da660, 0x0, 0x7070e0900000000, 0xc4202e6798, 0xc4201fcce8)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/golang.org/x/net/http2/frame.go:237 +0x7d
github.com/containerd/containerd/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc4203021c0, 0xc4202bff20, 0xc4202bff20, 0x0, 0x0)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/golang.org/x/net/http2/frame.go:492 +0xa6
github.com/containerd/containerd/vendor/google.golang.org/grpc/transport.(*http2Server).HandleStreams(0xc420404000, 0xc4203ef9e0, 0x10b0820)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/transport/http2_server.go:393 +0x317
github.com/containerd/containerd/vendor/google.golang.org/grpc.(*Server).serveStreams(0xc4201f8140, 0x146d1e0, 0xc420404000)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/server.go:568 +0x142
github.com/containerd/containerd/vendor/google.golang.org/grpc.(*Server).serveHTTP2Transport(0xc4201f8140, 0x146d6c0, 0xc4202b8028, 0x0, 0x0)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/server.go:561 +0x473
github.com/containerd/containerd/vendor/google.golang.org/grpc.(*Server).handleRawConn(0xc4201f8140, 0x146d6c0, 0xc4202b8028)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/server.go:526 +0x499
created by github.com/containerd/containerd/vendor/google.golang.org/grpc.(*Server).Serve
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/server.go:495 +0x5bb

goroutine 53 [select]:
github.com/containerd/containerd/vendor/google.golang.org/grpc/transport.loopyWriter(0x7f97623111a8, 0xc4203ed840, 0xc4203ef980, 0xc42041cfb8)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/transport/transport.go:750 +0x2e6
github.com/containerd/containerd/vendor/google.golang.org/grpc/transport.newHTTP2Server.func1(0xc420404000)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/transport/http2_server.go:227 +0x60
created by github.com/containerd/containerd/vendor/google.golang.org/grpc/transport.newHTTP2Server
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/transport/http2_server.go:226 +0x8fb

goroutine 54 [select, 88 minutes]:
github.com/containerd/containerd/vendor/google.golang.org/grpc/transport.(*http2Server).keepalive(0xc420404000)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/transport/http2_server.go:935 +0x266
created by github.com/containerd/containerd/vendor/google.golang.org/grpc/transport.newHTTP2Server
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/transport/http2_server.go:230 +0x920

goroutine 56 [select, 5006 minutes]:
github.com/containerd/containerd/services/events.(*service).Subscribe(0xc4201b0088, 0xc4203e93c0, 0x146d000, 0xc4203f0550, 0x0, 0x0)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/services/events/service.go:61 +0x213
github.com/containerd/containerd/api/services/events/v1._Events_Subscribe_Handler(0xff7e20, 0xc4201b0088, 0x146c580, 0xc4203e93a0, 0xc4202e6d20, 0xc4202a2000)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/api/services/events/v1/events.pb.go:288 +0x110
github.com/containerd/containerd/vendor/github.com/grpc-ecosystem/go-grpc-prometheus.StreamServerInterceptor(0xff7e20, 0xc4201b0088, 0x146c640, 0xc4202ee800, 0xc4203e9380, 0x10af408, 0x0, 0x0)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/github.com/grpc-ecosystem/go-grpc-prometheus/server.go:40 +0x13b
github.com/containerd/containerd/vendor/google.golang.org/grpc.(*Server).processStreamingRPC(0xc4201f8140, 0x146d1e0, 0xc4200aef20, 0xc4202f0400, 0xc4202c7470, 0x1447fc0, 0x0, 0x0, 0x0)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/server.go:940 +0x2ea
github.com/containerd/containerd/vendor/google.golang.org/grpc.(*Server).handleStream(0xc4201f8140, 0x146d1e0, 0xc4200aef20, 0xc4202f0400, 0x0)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/server.go:1027 +0x14c3
github.com/containerd/containerd/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc4200157f0, 0xc4201f8140, 0x146d1e0, 0xc4200aef20, 0xc4202f0400)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/server.go:572 +0xa1
created by github.com/containerd/containerd/vendor/google.golang.org/grpc.(*Server).serveStreams.func1
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/server.go:570 +0xa3

goroutine 15 [IO wait, 74 minutes]:
internal/poll.runtime_pollWait(0x7f9762b56d30, 0x72, 0x0)
    /usr/local/go/src/runtime/netpoll.go:173 +0x59
internal/poll.(*pollDesc).wait(0xc420012498, 0x72, 0xffffffffffffff00, 0x1460960, 0x145adf0)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0xb0
internal/poll.(*pollDesc).waitRead(0xc420012498, 0xc420428000, 0x8000, 0x8000)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3f
internal/poll.(*FD).Read(0xc420012480, 0xc420428000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
    /usr/local/go/src/internal/poll/fd_unix.go:126 +0x18c
net.(*netFD).Read(0xc420012480, 0xc420428000, 0x8000, 0x8000, 0x11, 0x0, 0x0)
    /usr/local/go/src/net/fd_unix.go:202 +0x54
net.(*conn).Read(0xc42000e138, 0xc420428000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/net.go:176 +0x6f
bufio.(*Reader).Read(0xc420058cc0, 0xc4200ac3b8, 0x9, 0x9, 0x9, 0x0, 0x0)
    /usr/local/go/src/bufio/bufio.go:213 +0x30d
io.ReadAtLeast(0x145c760, 0xc420058cc0, 0xc4200ac3b8, 0x9, 0x9, 0x9, 0xc420038bb8, 0x400f10, 0xc420038c67)
    /usr/local/go/src/io/io.go:309 +0x88
io.ReadFull(0x145c760, 0xc420058cc0, 0xc4200ac3b8, 0x9, 0x9, 0x83ce2d, 0xc42048414c, 0xc42043a000)
    /usr/local/go/src/io/io.go:327 +0x5a
github.com/containerd/containerd/vendor/golang.org/x/net/http2.readFrameHeader(0xc4200ac3b8, 0x9, 0x9, 0x145c760, 0xc420058cc0, 0x0, 0x0, 0xc420484140, 0xc420038ce8)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/golang.org/x/net/http2/frame.go:237 +0x7d
github.com/containerd/containerd/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc4200ac380, 0xc420484140, 0xc420484140, 0x0, 0x0)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/golang.org/x/net/http2/frame.go:492 +0xa6
github.com/containerd/containerd/vendor/google.golang.org/grpc/transport.(*http2Server).HandleStreams(0xc4200aef20, 0xc420173bc0, 0x10b0820)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/transport/http2_server.go:393 +0x317
github.com/containerd/containerd/vendor/google.golang.org/grpc.(*Server).serveStreams(0xc4201f8140, 0x146d1e0, 0xc4200aef20)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/server.go:568 +0x142
github.com/containerd/containerd/vendor/google.golang.org/grpc.(*Server).serveHTTP2Transport(0xc4201f8140, 0x146d6c0, 0xc42000e138, 0x0, 0x0)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/server.go:561 +0x473
github.com/containerd/containerd/vendor/google.golang.org/grpc.(*Server).handleRawConn(0xc4201f8140, 0x146d6c0, 0xc42000e138)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/server.go:526 +0x499
created by github.com/containerd/containerd/vendor/google.golang.org/grpc.(*Server).Serve
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/server.go:495 +0x5bb

goroutine 16 [select, 74 minutes]:
github.com/containerd/containerd/vendor/google.golang.org/grpc/transport.loopyWriter(0x7f97623111a8, 0xc420054cc0, 0xc420173b60, 0xc42041dfb8)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/transport/transport.go:750 +0x2e6
github.com/containerd/containerd/vendor/google.golang.org/grpc/transport.newHTTP2Server.func1(0xc4200aef20)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/transport/http2_server.go:227 +0x60
created by github.com/containerd/containerd/vendor/google.golang.org/grpc/transport.newHTTP2Server
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/transport/http2_server.go:226 +0x8fb

goroutine 66 [select, 74 minutes]:
github.com/containerd/containerd/vendor/google.golang.org/grpc/transport.(*http2Server).keepalive(0xc4200aef20)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/transport/http2_server.go:935 +0x266
created by github.com/containerd/containerd/vendor/google.golang.org/grpc/transport.newHTTP2Server
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/transport/http2_server.go:230 +0x920

goroutine 57 [semacquire, 5006 minutes]:
sync.runtime_notifyListWait(0xc4203edbd0, 0xc400000000)
    /usr/local/go/src/runtime/sema.go:507 +0x114
sync.(*Cond).Wait(0xc4203edbc0)
    /usr/local/go/src/sync/cond.go:56 +0x82
github.com/containerd/containerd/vendor/github.com/docker/go-events.(*Queue).next(0xc4204382d0, 0x0, 0x0)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/github.com/docker/go-events/queue.go:103 +0x87
github.com/containerd/containerd/vendor/github.com/docker/go-events.(*Queue).run(0xc4204382d0)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/github.com/docker/go-events/queue.go:68 +0x34
created by github.com/containerd/containerd/vendor/github.com/docker/go-events.NewQueue
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/github.com/docker/go-events/queue.go:29 +0x14d

goroutine 58 [select, 5006 minutes]:
github.com/containerd/containerd/events/exchange.(*Exchange).Subscribe.func3(0xc420438330, 0xc4203e93e0, 0xc42040a120, 0x1469fe0, 0xc4203edb80, 0xc4202da7e0)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/events/exchange/exchange.go:155 +0x15f
created by github.com/containerd/containerd/events/exchange.(*Exchange).Subscribe
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/events/exchange/exchange.go:149 +0x291

goroutine 67 [select]:
github.com/containerd/containerd/services/events.(*service).Subscribe(0xc4201b0088, 0xc4203e9700, 0x146d000, 0xc4203f06f0, 0x0, 0x0)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/services/events/service.go:61 +0x213
github.com/containerd/containerd/api/services/events/v1._Events_Subscribe_Handler(0xff7e20, 0xc4201b0088, 0x146c580, 0xc4203e96e0, 0xc4202e7090, 0xc420029400)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/api/services/events/v1/events.pb.go:288 +0x110
github.com/containerd/containerd/vendor/github.com/grpc-ecosystem/go-grpc-prometheus.StreamServerInterceptor(0xff7e20, 0xc4201b0088, 0x146c640, 0xc4202eeb80, 0xc4203e96c0, 0x10af408, 0x0, 0xc420268ec8)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/github.com/grpc-ecosystem/go-grpc-prometheus/server.go:40 +0x13b
github.com/containerd/containerd/vendor/google.golang.org/grpc.(*Server).processStreamingRPC(0xc4201f8140, 0x146d1e0, 0xc420404840, 0xc42019a200, 0xc4202c7470, 0x1447fc0, 0x0, 0x0, 0x0)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/server.go:940 +0x2ea
github.com/containerd/containerd/vendor/google.golang.org/grpc.(*Server).handleStream(0xc4201f8140, 0x146d1e0, 0xc420404840, 0xc42019a200, 0x0)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/server.go:1027 +0x14c3
github.com/containerd/containerd/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc4202c8860, 0xc4201f8140, 0x146d1e0, 0xc420404840, 0xc42019a200)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/server.go:572 +0xa1
created by github.com/containerd/containerd/vendor/google.golang.org/grpc.(*Server).serveStreams.func1
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/server.go:570 +0xa3

goroutine 60 [IO wait]:
internal/poll.runtime_pollWait(0x7f9762b56c70, 0x72, 0x0)
    /usr/local/go/src/runtime/netpoll.go:173 +0x59
internal/poll.(*pollDesc).wait(0xc4202ee918, 0x72, 0xffffffffffffff00, 0x1460960, 0x145adf0)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0xb0
internal/poll.(*pollDesc).waitRead(0xc4202ee918, 0xc420458000, 0x8000, 0x8000)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3f
internal/poll.(*FD).Read(0xc4202ee900, 0xc420458000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
    /usr/local/go/src/internal/poll/fd_unix.go:126 +0x18c
net.(*netFD).Read(0xc4202ee900, 0xc420458000, 0x8000, 0x8000, 0x11, 0x0, 0x0)
    /usr/local/go/src/net/fd_unix.go:202 +0x54
net.(*conn).Read(0xc4202b8040, 0xc420458000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/net.go:176 +0x6f
bufio.(*Reader).Read(0xc4202daae0, 0xc4203023b8, 0x9, 0x9, 0x9, 0x0, 0x0)
    /usr/local/go/src/bufio/bufio.go:213 +0x30d
io.ReadAtLeast(0x145c760, 0xc4202daae0, 0xc4203023b8, 0x9, 0x9, 0x9, 0x22856124201febb8, 0x5ac36c69, 0xc4201febc0)
    /usr/local/go/src/io/io.go:309 +0x88
io.ReadFull(0x145c760, 0xc4202daae0, 0xc4203023b8, 0x9, 0x9, 0x1113749f52782, 0x14a7a60, 0xbea8f8fa62803e4c)
    /usr/local/go/src/io/io.go:327 +0x5a
github.com/containerd/containerd/vendor/golang.org/x/net/http2.readFrameHeader(0xc4203023b8, 0x9, 0x9, 0x145c760, 0xc4202daae0, 0x0, 0x7070e0900000000, 0xc4202e6fb8, 0xc4201fece8)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/golang.org/x/net/http2/frame.go:237 +0x7d
github.com/containerd/containerd/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc420302380, 0xc420204d20, 0xc420204d20, 0x0, 0x0)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/golang.org/x/net/http2/frame.go:492 +0xa6
github.com/containerd/containerd/vendor/google.golang.org/grpc/transport.(*http2Server).HandleStreams(0xc420404840, 0xc420438900, 0x10b0820)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/transport/http2_server.go:393 +0x317
github.com/containerd/containerd/vendor/google.golang.org/grpc.(*Server).serveStreams(0xc4201f8140, 0x146d1e0, 0xc420404840)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/server.go:568 +0x142
github.com/containerd/containerd/vendor/google.golang.org/grpc.(*Server).serveHTTP2Transport(0xc4201f8140, 0x146d6c0, 0xc4202b8040, 0x0, 0x0)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/server.go:561 +0x473
github.com/containerd/containerd/vendor/google.golang.org/grpc.(*Server).handleRawConn(0xc4201f8140, 0x146d6c0, 0xc4202b8040)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/server.go:526 +0x499
created by github.com/containerd/containerd/vendor/google.golang.org/grpc.(*Server).Serve
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/server.go:495 +0x5bb

goroutine 61 [select]:
github.com/containerd/containerd/vendor/google.golang.org/grpc/transport.loopyWriter(0x7f97623111a8, 0xc4203eddc0, 0xc4204388a0, 0xc42046cfb8)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/transport/transport.go:750 +0x2e6
github.com/containerd/containerd/vendor/google.golang.org/grpc/transport.newHTTP2Server.func1(0xc420404840)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/transport/http2_server.go:227 +0x60
created by github.com/containerd/containerd/vendor/google.golang.org/grpc/transport.newHTTP2Server
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/transport/http2_server.go:226 +0x8fb

goroutine 62 [select, 74 minutes]:
github.com/containerd/containerd/vendor/google.golang.org/grpc/transport.(*http2Server).keepalive(0xc420404840)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/transport/http2_server.go:935 +0x266
created by github.com/containerd/containerd/vendor/google.golang.org/grpc/transport.newHTTP2Server
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/transport/http2_server.go:230 +0x920

goroutine 63 [semacquire]:
sync.runtime_notifyListWait(0xc4203edf50, 0xc400009f39)
    /usr/local/go/src/runtime/sema.go:507 +0x114
sync.(*Cond).Wait(0xc4203edf40)
    /usr/local/go/src/sync/cond.go:56 +0x82
github.com/containerd/containerd/vendor/github.com/docker/go-events.(*Queue).next(0xc4204389c0, 0x0, 0x0)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/github.com/docker/go-events/queue.go:103 +0x87
github.com/containerd/containerd/vendor/github.com/docker/go-events.(*Queue).run(0xc4204389c0)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/github.com/docker/go-events/queue.go:68 +0x34
created by github.com/containerd/containerd/vendor/github.com/docker/go-events.NewQueue
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/github.com/docker/go-events/queue.go:29 +0x14d

goroutine 64 [select]:
github.com/containerd/containerd/events/exchange.(*Exchange).Subscribe.func3(0xc420438a20, 0xc4203e9720, 0xc42040a3c0, 0x1469fe0, 0xc4203edf00, 0xc4202daf00)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/events/exchange/exchange.go:155 +0x15f
created by github.com/containerd/containerd/events/exchange.(*Exchange).Subscribe
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/events/exchange/exchange.go:149 +0x291

goroutine 592290 [select, 43 minutes]:
github.com/containerd/containerd/vendor/github.com/stevvooe/ttrpc.(*Client).run(0xc4201a6360)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/github.com/stevvooe/ttrpc/client.go:171 +0x342
created by github.com/containerd/containerd/vendor/github.com/stevvooe/ttrpc.NewClient
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/github.com/stevvooe/ttrpc/client.go:46 +0x2ab

goroutine 671746 [select, 43 minutes]:
github.com/containerd/containerd/vendor/github.com/stevvooe/ttrpc.(*Client).dispatch(0xc4201a6360, 0x146a0a0, 0xc4204eee70, 0xc420345a80, 0xc4201daa60, 0x0, 0x0)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/github.com/stevvooe/ttrpc/client.go:102 +0x282
github.com/containerd/containerd/vendor/github.com/stevvooe/ttrpc.(*Client).Call(0xc4201a6360, 0x146a0a0, 0xc4204eee70, 0xbbc2cb, 0x25, 0xba0ec1, 0x5, 0x1046a60, 0xc4201a1ee0, 0x1046b40, ...)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/github.com/stevvooe/ttrpc/client.go:73 +0x15d
github.com/containerd/containerd/linux/shim/v1.(*shimClient).State(0xc42000e068, 0x146a0a0, 0xc4204eee70, 0xc4201a1ee0, 0x0, 0xc42016b298, 0xc42059f578)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/linux/shim/v1/shim.pb.go:1729 +0xbf
github.com/containerd/containerd/linux.(*Process).State(0xc4201daa40, 0x146a0a0, 0xc4204eee70, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/linux/process.go:45 +0xc7
github.com/containerd/containerd/services/tasks.processFromContainerd(0x7f9762311168, 0xc4204eee70, 0x146d0c0, 0xc4201daa40, 0x40, 0x146d0c0, 0xc4201daa40)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/services/tasks/service.go:225 +0xbe
github.com/containerd/containerd/services/tasks.(*service).Get(0xc4202c6e70, 0x7f9762311168, 0xc4204eee70, 0xc4201da800, 0xc4202c6e70, 0xbbd46b, 0x3)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/services/tasks/service.go:268 +0xef
github.com/containerd/containerd/api/services/tasks/v1._Tasks_Get_Handler.func1(0x7f9762311168, 0xc4204eee70, 0x10444c0, 0xc4201da800, 0xc420079ae0, 0x14cdcb0, 0xf87860, 0xc4201a1e30)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/api/services/tasks/v1/tasks.pb.go:680 +0x88
github.com/containerd/containerd/vendor/github.com/grpc-ecosystem/go-grpc-prometheus.UnaryServerInterceptor(0x7f9762311168, 0xc4204eee70, 0x10444c0, 0xc4201da800, 0xc4201da820, 0xc4201da840, 0x50, 0x48, 0xc4201da7e0, 0xc42059f968)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/github.com/grpc-ecosystem/go-grpc-prometheus/server.go:29 +0xd4
github.com/containerd/containerd/server.interceptor(0x7f9762311168, 0xc4204eedb0, 0x10444c0, 0xc4201da800, 0xc4201da820, 0xc4201da840, 0x4354a6, 0xc42059f9e0, 0x41228a, 0x50)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/server/server.go:267 +0x1d2
github.com/containerd/containerd/api/services/tasks/v1._Tasks_Get_Handler(0x1097ee0, 0xc4202c6e70, 0x7f9762311168, 0xc42044fbc0, 0xc420079900, 0x10af920, 0x0, 0x0, 0xc42001d900, 0x46)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/api/services/tasks/v1/tasks.pb.go:682 +0x16f
github.com/containerd/containerd/vendor/google.golang.org/grpc.(*Server).processUnaryRPC(0xc4201f8140, 0x146d1e0, 0xc420404840, 0xc42021d700, 0xc4202c77d0, 0x1452dc0, 0x0, 0x0, 0x0)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/server.go:826 +0xab6
github.com/containerd/containerd/vendor/google.golang.org/grpc.(*Server).handleStream(0xc4201f8140, 0x146d1e0, 0xc420404840, 0xc42021d700, 0x0)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/server.go:1023 +0x152a
github.com/containerd/containerd/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc4202c8860, 0xc4201f8140, 0x146d1e0, 0xc420404840, 0xc42021d700)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/server.go:572 +0xa1
created by github.com/containerd/containerd/vendor/google.golang.org/grpc.(*Server).serveStreams.func1
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/server.go:570 +0xa3

goroutine 764452 [select]:
github.com/containerd/containerd/vendor/github.com/stevvooe/ttrpc.(*Client).run(0xc420169500)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/github.com/stevvooe/ttrpc/client.go:171 +0x342
created by github.com/containerd/containerd/vendor/github.com/stevvooe/ttrpc.NewClient
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/github.com/stevvooe/ttrpc/client.go:46 +0x2ab

goroutine 592291 [IO wait, 43 minutes]:
internal/poll.runtime_pollWait(0x7f9762b56af0, 0x72, 0x0)
    /usr/local/go/src/runtime/netpoll.go:173 +0x59
internal/poll.(*pollDesc).wait(0xc4201b4398, 0x72, 0xffffffffffffff00, 0x1460960, 0x145adf0)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0xb0
internal/poll.(*pollDesc).waitRead(0xc4201b4398, 0xc4202ea000, 0x1000, 0x1000)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3f
internal/poll.(*FD).Read(0xc4201b4380, 0xc4202ea000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /usr/local/go/src/internal/poll/fd_unix.go:126 +0x18c
net.(*netFD).Read(0xc4201b4380, 0xc4202ea000, 0x1000, 0x1000, 0xc420201e78, 0x43ebf4, 0xc42040a720)
    /usr/local/go/src/net/fd_unix.go:202 +0x54
net.(*conn).Read(0xc42000e060, 0xc4202ea000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/net.go:176 +0x6f
bufio.(*Reader).Read(0xc4201a6300, 0xc4202fa060, 0xa, 0xa, 0xc420201fac, 0x0, 0xc420201fa8)
    /usr/local/go/src/bufio/bufio.go:213 +0x30d
io.ReadAtLeast(0x145c760, 0xc4201a6300, 0xc4202fa060, 0xa, 0xa, 0xa, 0xc42021e300, 0x0, 0xc420201f48)
    /usr/local/go/src/io/io.go:309 +0x88
io.ReadFull(0x145c760, 0xc4201a6300, 0xc4202fa060, 0xa, 0xa, 0x2, 0x2, 0x0)
    /usr/local/go/src/io/io.go:327 +0x5a
github.com/containerd/containerd/vendor/github.com/stevvooe/ttrpc.readMessageHeader(0xc4202fa060, 0xa, 0xa, 0x145c760, 0xc4201a6300, 0xc420201f48, 0x2, 0x2, 0xc420201fa8)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/github.com/stevvooe/ttrpc/channel.go:38 +0x62
github.com/containerd/containerd/vendor/github.com/stevvooe/ttrpc.(*channel).recv(0xc4202fa040, 0x146a020, 0xc420014048, 0x0, 0x2, 0xc4201b6070, 0x6, 0x6, 0x0, 0x0)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/github.com/stevvooe/ttrpc/channel.go:86 +0x6f
github.com/containerd/containerd/vendor/github.com/stevvooe/ttrpc.(*Client).run.func1(0xc42040a780, 0xc4201a6360, 0xc42003ba20, 0xc42040a720)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/github.com/stevvooe/ttrpc/client.go:144 +0x9a
created by github.com/containerd/containerd/vendor/github.com/stevvooe/ttrpc.(*Client).run
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/github.com/stevvooe/ttrpc/client.go:139 +0x164

goroutine 764451 [syscall]:
syscall.Syscall6(0xf7, 0x1, 0x3778, 0xc42026d5b8, 0x1000004, 0x0, 0x0, 0x146d1e0, 0xc420404840, 0xc42019af00)
    /usr/local/go/src/syscall/asm_linux_amd64.s:44 +0x5
os.(*Process).blockUntilWaitable(0xc42052ed80, 0x0, 0xc42026d6b0, 0x5268aa)
    /usr/local/go/src/os/wait_waitid.go:31 +0xa7
os.(*Process).wait(0xc42052ed80, 0xc42026d770, 0x8943d5, 0x146a0a0)
    /usr/local/go/src/os/exec_unix.go:22 +0x44
os.(*Process).Wait(0xc42052ed80, 0xc420128d00, 0x42b91b, 0xc400000008)
    /usr/local/go/src/os/exec.go:115 +0x2d
os/exec.(*Cmd).Wait(0xc4200ae580, 0x45e611, 0xc4202c8860)
    /usr/local/go/src/os/exec/exec.go:446 +0x64
github.com/containerd/containerd/linux/shim/client.WithStart.func1.2(0xc4200ae580, 0xc4202f6410)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/linux/shim/client/client.go:59 +0x2d
created by github.com/containerd/containerd/linux/shim/client.WithStart.func1
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/linux/shim/client/client.go:58 +0x2d5

goroutine 764429 [IO wait]:
internal/poll.runtime_pollWait(0x7f9762b56bb0, 0x72, 0x0)
    /usr/local/go/src/runtime/netpoll.go:173 +0x59
internal/poll.(*pollDesc).wait(0xc420230818, 0x72, 0xffffffffffffff00, 0x1460960, 0x145adf0)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0xb0
internal/poll.(*pollDesc).waitRead(0xc420230818, 0xc4204def00, 0x1, 0x1)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3f
internal/poll.(*FD).Read(0xc420230800, 0xc4204defa1, 0x1, 0x1, 0x0, 0x0, 0x0)
    /usr/local/go/src/internal/poll/fd_unix.go:126 +0x18c
net.(*netFD).Read(0xc420230800, 0xc4204defa1, 0x1, 0x1, 0x0, 0xc4204deb00, 0x42b91b)
    /usr/local/go/src/net/fd_unix.go:202 +0x54
net.(*conn).Read(0xc4201b0230, 0xc4204defa1, 0x1, 0x1, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/net.go:176 +0x6f
net/http.(*connReader).backgroundRead(0xc4204def90)
    /usr/local/go/src/net/http/server.go:660 +0x64
created by net/http.(*connReader).startBackgroundRead
    /usr/local/go/src/net/http/server.go:656 +0xda

goroutine 671647 [select, 43 minutes]:
github.com/containerd/containerd/vendor/github.com/stevvooe/ttrpc.(*Client).dispatch(0xc4201a6360, 0x146a0a0, 0xc42024e5a0, 0xc4203440c0, 0xc420306260, 0x0, 0x0)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/github.com/stevvooe/ttrpc/client.go:102 +0x282
github.com/containerd/containerd/vendor/github.com/stevvooe/ttrpc.(*Client).Call(0xc4201a6360, 0x146a0a0, 0xc42024e5a0, 0xbbc2cb, 0x25, 0xba0ebc, 0x5, 0x10468a0, 0xc4201ae640, 0x1046980, ...)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/github.com/stevvooe/ttrpc/client.go:73 +0x15d
github.com/containerd/containerd/linux/shim/v1.(*shimClient).Start(0xc42000e068, 0x146a0a0, 0xc42024e5a0, 0xc4201ae640, 0x1001ae0, 0x7f9762311101, 0xc420571660)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/linux/shim/v1/shim.pb.go:1745 +0xbf
github.com/containerd/containerd/linux.(*Process).Start(0xc4203061e0, 0x146a0a0, 0xc42024e5a0, 0x146a0a0, 0xc42024e5a0)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/linux/process.go:109 +0x91
github.com/containerd/containerd/services/tasks.(*service).Start(0xc4202c6e70, 0x7f9762311168, 0xc42024e5a0, 0xc42000c560, 0xc4202c6e70, 0xbbea93, 0x5)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/services/tasks/service.go:175 +0x14e
github.com/containerd/containerd/api/services/tasks/v1._Tasks_Start_Handler.func1(0x7f9762311168, 0xc42024e5a0, 0x1044f40, 0xc42000c560, 0xc4202f64b0, 0x14cdcb0, 0xf87860, 0xc4201ae630)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/api/services/tasks/v1/tasks.pb.go:626 +0x8b
github.com/containerd/containerd/vendor/github.com/grpc-ecosystem/go-grpc-prometheus.UnaryServerInterceptor(0x7f9762311168, 0xc42024e5a0, 0x1044f40, 0xc42000c560, 0xc42000c580, 0xc42000c5a0, 0x50, 0x48, 0xc42000c540, 0xc420571968)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/github.com/grpc-ecosystem/go-grpc-prometheus/server.go:29 +0xd4
github.com/containerd/containerd/server.interceptor(0x7f9762311168, 0xc42024e4b0, 0x1044f40, 0xc42000c560, 0xc42000c580, 0xc42000c5a0, 0x8000000000000000, 0xc4205719e0, 0x41228a, 0x50)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/server/server.go:267 +0x1d2
github.com/containerd/containerd/api/services/tasks/v1._Tasks_Start_Handler(0x1097ee0, 0xc4202c6e70, 0x7f9762311168, 0xc420471a70, 0xc4202f62d0, 0x10af920, 0x0, 0x0, 0x0, 0x145e9e0)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/api/services/tasks/v1/tasks.pb.go:628 +0x16f
github.com/containerd/containerd/vendor/google.golang.org/grpc.(*Server).processUnaryRPC(0xc4201f8140, 0x146d1e0, 0xc420404840, 0xc4202f0000, 0xc4202c77d0, 0x1452d78, 0x0, 0x0, 0x0)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/server.go:826 +0xab6
github.com/containerd/containerd/vendor/google.golang.org/grpc.(*Server).handleStream(0xc4201f8140, 0x146d1e0, 0xc420404840, 0xc4202f0000, 0x0)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/server.go:1023 +0x152a
github.com/containerd/containerd/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc4202c8860, 0xc4201f8140, 0x146d1e0, 0xc420404840, 0xc4202f0000)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/server.go:572 +0xa1
created by github.com/containerd/containerd/vendor/google.golang.org/grpc.(*Server).serveStreams.func1
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/server.go:570 +0xa3

goroutine 592225 [syscall, 68 minutes]:
syscall.Syscall6(0xf7, 0x1, 0xb0e, 0xc42026a5b8, 0x1000004, 0x0, 0x0, 0x146d1e0, 0xc420404840, 0xc4202f5000)
    /usr/local/go/src/syscall/asm_linux_amd64.s:44 +0x5
os.(*Process).blockUntilWaitable(0xc420016d50, 0x0, 0xc42026a6b0, 0x5268aa)
    /usr/local/go/src/os/wait_waitid.go:31 +0xa7
os.(*Process).wait(0xc420016d50, 0xc42026a770, 0x8943d5, 0x146a0a0)
    /usr/local/go/src/os/exec_unix.go:22 +0x44
os.(*Process).Wait(0xc420016d50, 0xc42034b300, 0x42b91b, 0xc400000008)
    /usr/local/go/src/os/exec.go:115 +0x2d
os/exec.(*Cmd).Wait(0xc420262160, 0x45e611, 0xc4202c8860)
    /usr/local/go/src/os/exec/exec.go:446 +0x64
github.com/containerd/containerd/linux/shim/client.WithStart.func1.2(0xc420262160, 0xc420440280)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/linux/shim/client/client.go:59 +0x2d
created by github.com/containerd/containerd/linux/shim/client.WithStart.func1
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/linux/shim/client/client.go:58 +0x2d5

goroutine 764453 [IO wait]:
internal/poll.runtime_pollWait(0x7f9762b567f0, 0x72, 0x0)
    /usr/local/go/src/runtime/netpoll.go:173 +0x59
internal/poll.(*pollDesc).wait(0xc4202ee498, 0x72, 0xffffffffffffff00, 0x1460960, 0x145adf0)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0xb0
internal/poll.(*pollDesc).waitRead(0xc4202ee498, 0xc420777000, 0x1000, 0x1000)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3f
internal/poll.(*FD).Read(0xc4202ee480, 0xc420777000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /usr/local/go/src/internal/poll/fd_unix.go:126 +0x18c
net.(*netFD).Read(0xc4202ee480, 0xc420777000, 0x1000, 0x1000, 0xc42041fe78, 0x43ebf4, 0xc4201be300)
    /usr/local/go/src/net/fd_unix.go:202 +0x54
net.(*conn).Read(0xc42000e1f0, 0xc420777000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/net.go:176 +0x6f
bufio.(*Reader).Read(0xc420169440, 0xc4203453e0, 0xa, 0xa, 0xc42041ffac, 0x0, 0xc42041ffa8)
    /usr/local/go/src/bufio/bufio.go:213 +0x30d
io.ReadAtLeast(0x145c760, 0xc420169440, 0xc4203453e0, 0xa, 0xa, 0xa, 0xc4200b4180, 0x0, 0xc42041ff48)
    /usr/local/go/src/io/io.go:309 +0x88
io.ReadFull(0x145c760, 0xc420169440, 0xc4203453e0, 0xa, 0xa, 0x2, 0x2, 0x0)
    /usr/local/go/src/io/io.go:327 +0x5a
github.com/containerd/containerd/vendor/github.com/stevvooe/ttrpc.readMessageHeader(0xc4203453e0, 0xa, 0xa, 0x145c760, 0xc420169440, 0xc42041ff48, 0x2, 0x2, 0xc42041ffa8)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/github.com/stevvooe/ttrpc/channel.go:38 +0x62
github.com/containerd/containerd/vendor/github.com/stevvooe/ttrpc.(*channel).recv(0xc4203453c0, 0x146a020, 0xc420014048, 0x0, 0x2, 0xc42073a000, 0x6, 0x1b9, 0x0, 0x0)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/github.com/stevvooe/ttrpc/channel.go:86 +0x6f
github.com/containerd/containerd/vendor/github.com/stevvooe/ttrpc.(*Client).run.func1(0xc4201be360, 0xc420169500, 0xc420664ec0, 0xc4201be300)
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/github.com/stevvooe/ttrpc/client.go:144 +0x9a
created by github.com/containerd/containerd/vendor/github.com/stevvooe/ttrpc.(*Client).run
    /tmp/tmp.AD0Uaz9KtF/src/github.com/containerd/containerd/vendor/github.com/stevvooe/ttrpc/client.go:139 +0x164

$ docker info
Containers: 5
 Running: 1
 Paused: 0
 Stopped: 4
Images: 323
Server Version: 18.03.0-ce
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host ipvlan macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: cfd04396dc68220d1cecbe686a6cc3aa5ce3667c
runc version: 4fc53a81fb7c994640722ac585fa9ca548971871
init version: 949e6fa
Security Options:
 seccomp
  Profile: default
Kernel Version: 4.9.87-linuxkit-aufs
Operating System: Docker for Mac
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 7.786GiB
Name: linuxkit-025000000001
ID: UUJI:BHCD:4SFP:CCSM:H5QT:VZQL:E6AH:UXGU:5VMJ:P525:VYIK:YWOH
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): true
 File Descriptors: 42
 Goroutines: 70
 System Time: 2018-04-03T11:59:08.2610107Z
 EventsListeners: 2
HTTP Proxy: docker.for.mac.http.internal:3128
HTTPS Proxy: docker.for.mac.http.internal:3129
Registry: https://index.docker.io/v1/
Labels:
Experimental: true
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

$ docker version
Client:
 Version:   18.03.0-ce
 API version:   1.37
 Go version:    go1.9.4
 Git commit:    0520e24
 Built: Wed Mar 21 23:06:22 2018
 OS/Arch:   darwin/amd64
 Experimental:  false
 Orchestrator:  swarm

Server:
 Engine:
  Version:  18.03.0-ce
  API version:  1.37 (minimum version 1.12)
  Go version:   go1.9.4
  Git commit:   0520e24
  Built:    Wed Mar 21 23:14:32 2018
  OS/Arch:  linux/amd64
  Experimental: true

我得到的图像始终是docker.elastic.co/elasticsearch/elasticsearch:6.0.1 （或任何elasticsearch图像...我一直使用6.0.1）

我每天有23张其他图片我会多次启动和停止，但是elasticsearch一直死着……而且它不会是同一个Elasticsearch容器实例（因为我运行的不止一个，但是全部关闭了）图片）...

挂起时间可以是启动后的1-2分钟内到2-3小时之间的任何时间...

我还看到了似乎在多个群体中出现的问题。 在确定根本原因方面有任何进展吗？

人们会回滚到以前的版本吗？（已提及17.09）

该问题被标记为需要更多信息，还需要什么其他信息？

@timdau我们仍在生产中17.09，因为这些“不可阻挡的容器”对我们而言是最稳定的版本

在docker-CE 17.12.0（在3个集群中）中，我也遇到了同样的情况，我回滚到17.09。
令人难以置信的是，Docker现在在两个LTS版本中都存在此类严重错误，并且不予修复...
我知道也许很难复制，但这在很多人身上都发生了……

这是因为现在有EE版本，而该版本EE 2.2.x（Docker 17.06.x）也在努力吗？

这里有多个问题在起作用，并且有多个修复程序可以解决不同领域的问题。

我在docker-CE 17.12.0中也发生了同样的情况

17.12.1已经存在一段时间了。 它不能解决所有问题，但可以解决一些问题。
请更新。
18.03.0中还有其他修复程序可用，但值得等待的18.03.1可能很快就会发布。

由于我们了解此问题尚未解决且正在解决中，因此该问题仍未解决。
如果您想提供帮助，除了缩小案例范围之外，还有多种方法可以做出贡献……例如，特定/一致的复制步骤，来自更新的Docker实例的堆栈跟踪（并且容器化和容器化垫片也很有帮助）等。

来到这里，做出虚假的声明是愚蠢的姿态根本没有帮助。

我在其他集群中有这个问题：
docker service ls

ID NAME MODE REPLICAS IMAGE PORTS
2uy2rdh3cu7e arxx_rxx replicated 5/4 xx/arxx_rxx:latest-SNAP *:80->80/tcp,*:443->443/tcp
从4 ....中查看5个容器。（始终设置为4个副本）

docker service ps arxx_rxx
仅显示4个正在运行

docker ps
同时显示无法停止的额外容器
69364e4293d1 xx/arxx_rxx:latest-SNAP "java -jar app-all.j…" 13 days ago Up 13 days (healthy) 80/tcp, 443/tcp arxx_rxx.1.bayrllx65489r7e1vh5te3plp

与该容器有关的所有命令都挂起
docker inspect 69364e4293d1

容器也破坏了服务。

¿您能告诉我为您提供查找错误所需信息的确切命令吗？

Ubuntu 16.04
Docker 17.12.0-CE

如果您有Skype /环聊，我可以让您自己做。
@ cpuguy83

我们遇到了同样的问题。 其中一个容器挂起，因此其他命令（例如docker rmi / logs）不起作用。

猫/ etc / redhat-release

CentOS Linux版本7.4.1708（核心）

码头工人版本

客户：
版本：17.12.0-ce
API版本：1.35
Go版本：go1.9.2
Git提交：c97c6d6
建成：2017年12月27日星期三20:10:14
操作系统/ Arch：linux / amd64

服务器：
发动机：
版本：17.12.0-ce
API版本：1.35（最低版本1.12）
Go版本：go1.9.2
Git提交：c97c6d6
建成：2017年12月27日星期三20:12:46
操作系统/ Arch：linux / amd64
实验性：错误

古鲁丁

（curl --unix-socket /var/run/docker.sock http：/./ debug / pprof / goroutine？debug = 2）
goroutine.log

我在日志中看到一些大的IO等待。
此刻，我们切换回docker版本：17.09.1-ce

任何想法将不胜感激。

@victorvarza参见之前的评论： https :

我已经对基础架构进行了一些更改，以使自己有很多时间在下一次在生产系统上发生这种情况时收集日志/信息，这给我自己带来了很多好处。

我目前在运行docker-ce 18.03.1和Linux Kernel 4.13.0-39-generic x86_64的Ubuntu 16.04.4 LTS上。

有人可以确认这就是为了提供足够的信息来解决此问题而需要收集的所有信息？

1. docker inspect {container-id} > docker-inspect-container.log
2. ps -aux | grep {container-id}获取docker-containerd-shim pid
3. 要从docker-containerd-shim获取堆栈转储，请执行kill -s SIGUSR1 {docker-containerd-shim-pid} 。 这应该在dockerd的日志中生成堆栈跟踪。
4. sudo journalctl -u docker.service --since today > docker-service-log.txt
5. docker info
6. docker version

这里同样的问题。 容器被卡住。 无法停止，杀死，rm等。
在附件中提供了所有相关信息。

Client:
 Version:   18.03.0-ce
 API version:   1.37
 Go version:    go1.9.4
 Git commit:    0520e24
 Built: Wed Mar 21 23:04:48 2018
 OS/Arch:   linux/arm64
 Experimental:  false
 Orchestrator:  swarm

Server:
 Engine:
  Version:  18.03.0-ce
  API version:  1.37 (minimum version 1.12)
  Go version:   go1.9.4
  Git commit:   0520e24
  Built:    Wed Mar 21 23:10:22 2018
  OS/Arch:  linux/arm64
  Experimental: false

strace-pid.log
docker-version.txt
docker-service-log.txt
docker-inspect-container.log
docker-info.txt
docker-containerd-shim-cmd-pid.txt
contanerd-stack.log

我在18.03.0-ce中遇到了同样的问题，但看起来这与主机安装的NFS卷有关。 我已经在该线程中阅读了一些类似的设置。 NFS可能是这里的实际问题吗？

我们的设置中没有NFS，并且面临相同的问题。

18.03.1对此有所缓解。 如果该版本仍存在问题，请告知我们。

上次我检查时， @ cpuguy83用于AWS的Docker仍在18.03.0上。 发行说明中列出的最新版本是18.03.0。 我很想检查。 知道何时将适用于AWS的Docker稳定版升级到18.03.1吗？

小心升级群集集群，错误：＃36961，您的集群可能会崩溃。

@ cpuguy83 18.03.1在发布页面尚不存在： https ://docs.docker.com/release-notes/docker-ce/还是我瞎了？

发布页面上还没有18.03.1： https ：//docs.docker.com/release-notes/docker-ce/还是我瞎了？

该文档已过期，您可以在此处查看： https :
11天前发布。

@marcomsousa感谢您注意到这一点； 现在，发布说明也已添加到docs网站上； https://docs.docker.com/release-notes/docker-ce/#18031 -ce-2018-04-26

@ cpuguy83是否有与此问题相关的所有问题的清单？ 这样，我们就可以确定何时解决此问题及其安全升级方法。

@timdau通过https://github.com/containerd/containerd/commit/d235ae9439c7a22b96a0d66cb15dd8808d978e8f缓解了

此提交containerd / containerd @ d235ae9在容器1.0.3中释放。
Docker-ce 18.03.1包含此版本的容器。

因此，我们需要测试此错误是否已在18.03.1版本中修复

好像18.03.1为我解决了这个问题。 我在本地使用了一周，但是没有遇到问题，否则在一天之内就可以轻松重现。

18.03.1版本似乎解决了此问题。 （或缓解说@ cpuguy83）

我在4个集群中进行了测试。

谢谢大家的确认； 我将继续解决此问题。

如果您仍然在Docker 18.03.1或更高版本上遇到此问题; 请打开一个新的细节

@mavogel我在冻结

我在18.06.1-ce和18.09版本中都使用的（临时）解决方案与@casperWWW类似。 就我而言，我降低了在容器内执行的应用程序的日志级别，并且它们停止挂起。

因此，看来容器无法释放分配的I / O资源。

我没有使用docker-compose也遇到同样的问题。 我正在使用docker swarm。 不过，同样的事情，我偶尔会得到一些容器，这些容器既无法使docker swarm集群也无法停止使用docker CLI。 这导致docker swarm最终收集了超出其无法缩减规模的副本。 有时，这些副本仍可以为请求提供服务并接收流量。 删除容器的唯一方法是在受影响的节点上重新启动docker。

我和你一样，除了在受影响的节点上重启docker以外，还有其他解决方法吗？

我没有使用docker-compose也遇到同样的问题。 我正在使用docker swarm。 不过，同样的事情，我偶尔会得到一些容器，这些容器既无法使docker swarm集群也无法停止使用docker CLI。 这导致docker swarm最终收集了超出其无法缩减规模的副本。 有时，这些副本仍可以为请求提供服务并接收流量。 删除容器的唯一方法是在受影响的节点上重新启动docker。

我和你一样，除了在受影响的节点上重启docker以外，还有其他解决方法吗？

请参阅我之前在这里发送的评论-https : //github.com/moby/moby/issues/35933#issuecomment -419863541
希望对您有帮助。

我没有使用docker-compose也遇到同样的问题。 我正在使用docker swarm。 不过，同样的事情，我偶尔会得到一些容器，这些容器既无法使docker swarm集群也无法停止使用docker CLI。 这导致docker swarm最终收集了超出其无法缩减规模的副本。 有时，这些副本仍可以为请求提供服务并接收流量。 删除容器的唯一方法是在受影响的节点上重新启动docker。

我的Docker版本v17.12.1。
我遇到同样的问题。 这导致我在不同映像版本上的服务负载平衡，并且容器计数超过了副本集。 我认为这是docker的一个大错误。 它严重影响了我的生产服务。 请帮忙解决。 @thaJeztah

一年多以前，Docker 17.12已经达到停产期。 您可以在当前版本上进行复制吗？