Moby: How to deal with the case that Docker data file size reaches the 100G threshold?

Created on 29 Mar 2016 · 3Comments · Source: moby/moby

Output of docker version:

Client:
 Version:      1.8.2-el7.centos
 API version:  1.20
 Package Version: docker-1.8.2-10.el7.centos.x86_64
 Go version:   go1.4.2
 Git commit:   a01dc02/1.8.2
 Built:        
 OS/Arch:      linux/amd64

Server:
 Version:      1.8.2-el7.centos
 API version:  1.20
 Package Version: 
 Go version:   go1.4.2
 Git commit:   a01dc02/1.8.2
 Built:        
 OS/Arch:      linux/amd64

Output of docker info:

Containers: 0
Images: 130
Storage Driver: devicemapper
 Pool Name: docker-253:0-3221586422-pool
 Pool Blocksize: 65.54 kB
 Backing Filesystem: xfs
 Data file: /dev/loop4
 Metadata file: /dev/loop5
 Data Space Used: 107.4 GB
 Data Space Total: 107.4 GB
 Data Space Available: 0 B
 Metadata Space Used: 60.92 MB
 Metadata Space Total: 2.147 GB
 Metadata Space Available: 2.087 GB
 Udev Sync Supported: true
 Deferred Removal Enabled: false
 Data loop file: /var/lib/docker/devicemapper/devicemapper/data
 Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
 Library Version: 1.02.107-RHEL7 (2015-12-01)
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 3.10.0-229.el7.x86_64
Operating System: CentOS Linux 7 (Core)
CPUs: 12
Total Memory: 31.2 GiB
Name: IP-5-14
ID: DNOS:FC2P:2WH4:OSYL:L2CH:U7HZ:MFL2:ZID3:SYTX:JWKP:TGIN:YYPB
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled

Additional environment details (AWS, VirtualBox, physical, etc.):

Steps to reproduce the issue:
i wanna run my docker image called gateway using the cmd below.(a no-daemon entrypoint has been set in the gateway's Dockerfile)

docker run -d -P gateway

Describe the results you received:
the container can run as usual

Describe the results you expected:

[root@IP-5-14 devicemapper]# docker run -d -P gateway
Error response from daemon: Error running DeviceCreate (createSnapDevice) dm_task_run failed

Additional information you deem important (e.g. issue happens only occasionally):

as the docker info shows, the data space has been raised up to the threshold 100G.
when I delete the container using docker rm -f $(docker ps -a -q). All the containers had been removed but reported the error below:

Error response from daemon: Cannot destroy container 5d5eed10468b: Driver devicemapper failed to remove root filesystem 5d5eed10468b809475b0eb23bda167e1a962d32092a348936d56e27417dbf578: Error running DeleteDevice dm_task_run failed

i used docker ps -a to ensure that all containers had been removed after i did the step 2. the cmd result is empty. but error occur(below) when i delete a NONE image

[root@IP-5-14 devicemapper]# docker rmi 6fdebd7b0eb5   
Error response from daemon: Conflict, cannot delete because 6fdebd7b0eb5 is held by an ongoing pull or build
Error: failed to remove images: [6fdebd7b0eb5]

Can someone give me a favor? Thx a lot

Source

surlymo

Most helpful comment

You need to increase the pool allowed for your containers. To do this, you will need to remove your var/lib/docker which will destroy all your containers and images.

sudo service docker stop
sudo rm -rf /var/lib/docker
sudo dd if=/dev/zero of=/var/lib/docker/devicemapper/devicemapper/data bs=1G count=0 seek=300
This will make your data space used 300GB

Does this solve your issue? Also, try stopping docker processes, start docker again, and see if you can remove the images

ghost on 29 Mar 2016

👍3 👎1

All 3 comments

You need to increase the pool allowed for your containers. To do this, you will need to remove your var/lib/docker which will destroy all your containers and images.

sudo service docker stop
sudo rm -rf /var/lib/docker
sudo dd if=/dev/zero of=/var/lib/docker/devicemapper/devicemapper/data bs=1G count=0 seek=300
This will make your data space used 300GB

Does this solve your issue? Also, try stopping docker processes, start docker again, and see if you can remove the images

ghost on 29 Mar 2016

👍3 👎1

Devicemapper running out of space can be very tricky indeed; We're tracking issues around this in https://github.com/docker/docker/issues/20272.

For Docker 1.11, there's a new option to specify a minimum amount of free space to be kept, and would prevent you from arriving in this situation; see https://github.com/docker/docker/pull/20786

I'm going to close this issue, because I think this is not a bug, but a support question, but feel free to continue the discussion here