Elasticsearch: Check available disk space before starting a build

Created on 19 Sep 2016 · 3Comments · Source: elastic/elasticsearch

It would be super nice to check with gradle before starting tests (or on every run whatever the task is) that we have enough hard disk space (more than 10%).

Otherwise, you can have some tests failing.

For example, I was running gradle check for a PR not related to netty3 and it was always failing in netty3 module with:

  2> REPRODUCE WITH: gradle :modules:transport-netty3:integTest -Dtests.seed=E0B565B5C6B00017 -Dtests.class=org.elasticsearch.http.netty3.Netty3ClientYamlTestSuiteIT -Dtests.method="test {yaml=indices.get_alias/10_basic/Get aliases via /{index}/_alias/name,name}" -Dtests.security.manager=true -Dtests.locale=ar-KW -Dtests.timezone=CET
ERROR   30.1s | Netty3ClientYamlTestSuiteIT.test {yaml=indices.get_alias/10_basic/Get aliases via /{index}/_alias/name,name} <<< FAILURES!
   > Throwable #1: java.lang.RuntimeException: Failure at [indices.get_alias/10_basic:5]: listener timeout after waiting for [30000] ms
   >    at __randomizedtesting.SeedInfo.seed([E0B565B5C6B00017:68E15A6F684C6DEF]:0)
   >    at org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.executeSection(ESClientYamlSuiteTestCase.java:327)
   >    at org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.test(ESClientYamlSuiteTestCase.java:300)
   >    at java.lang.Thread.run(Thread.java:745)
   > Caused by: java.io.IOException: listener timeout after waiting for [30000] ms
   >    at org.elasticsearch.client.RestClient$SyncResponseListener.get(RestClient.java:615)
   >    at org.elasticsearch.client.RestClient.performRequest(RestClient.java:212)
   >    at org.elasticsearch.client.RestClient.performRequest(RestClient.java:185)
   >    at org.elasticsearch.test.rest.yaml.ClientYamlTestClient.callApi(ClientYamlTestClient.java:227)
   >    at org.elasticsearch.test.rest.yaml.ClientYamlTestExecutionContext.callApiInternal(ClientYamlTestExecutionContext.java:107)
   >    at org.elasticsearch.test.rest.yaml.ClientYamlTestExecutionContext.callApi(ClientYamlTestExecutionContext.java:72)
   >    at org.elasticsearch.test.rest.yaml.section.DoSection.execute(DoSection.java:128)
   >    at org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.executeSection(ESClientYamlSuiteTestCase.java:325)
   >    ... 38 more

When I looked at modules/transport-netty3/build/cluster/integTest\ node0/elasticsearch-6.0.0-alpha1-SNAPSHOT/logs/modules_transport-netty3_integTest.log, I saw:

[2016-09-19T10:32:20,078][WARN ][o.e.c.r.a.DiskThresholdMonitor] [tofpR8o] high disk watermark [90%] exceeded on [tofpR8o_Tv-QZ9t2AWtjug][tofpR8o][/Users/dpilato/Documents/Elasticsearch/dev/es-gradle/elasticsearch/modules/transport-netty3/build/cluster/integTest node0/elasticsearch-6.0.0-alpha1-SNAPSHOT/data/nodes/0] free: 2.9gb[0.6%], shards will be relocated away from this node

I have 3Gb free disk space though.

So if we can fail or warn at the beginning of our integration tests, that would help to identify faster the source of the "issue".

:DeliverBuild Delivery

Source

dadoonet

Most helpful comment

I don't think we should add things like this to the build. The check wouldn't be reliable because it's subject to time-of-check-time-of-use problems, and 90% probably isn't even the right cutoff starting from a clean build (because artifacts and logs and bunch of other things will get created pushing the disk usage over 90% before the dependent tests here run).

The right thing to do when an integration test fails is read the logs and that will show the issue, as you've done here.