Hibernate-reactive: tests stopped working on Docker desktop

Created on 4 Oct 2020 · 23Comments · Source: hibernate/hibernate-reactive

I've no idea what's going on here, but it seems to have coincided with me updating Docker desktop on Mac to the latest version:

Gradle Test Executor 2 > org.hibernate.reactive.FilterTest > testFilterCollectionFetch FAILED
    org.testcontainers.containers.ContainerLaunchException: Container startup failed
        at org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:330)
        at org.testcontainers.containers.GenericContainer.start(GenericContainer.java:311)
        at org.hibernate.reactive.containers.PostgreSQLDatabase.getJdbcUrl(PostgreSQLDatabase.java:39)
        at org.hibernate.reactive.containers.DatabaseConfiguration.getJdbcUrl(DatabaseConfiguration.java:78)
        at org.hibernate.reactive.BaseReactiveTest.constructConfiguration(BaseReactiveTest.java:88)
        at org.hibernate.reactive.FilterTest.constructConfiguration(FilterTest.java:26)

        Caused by:
        org.testcontainers.containers.ContainerFetchException: Can't get Docker image: RemoteDockerImage(imageName=postgres:13.0, imagePullPolicy=DefaultPullPolicy())
            at org.testcontainers.containers.GenericContainer.getDockerImageName(GenericContainer.java:1279)
            at org.testcontainers.containers.GenericContainer.logger(GenericContainer.java:613)
            at org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:320)
            ... 5 more

            Caused by:
            java.lang.IllegalStateException: Can not connect to Ryuk at localhost:32768
                at org.testcontainers.utility.ResourceReaper.start(ResourceReaper.java:176)
                at org.testcontainers.DockerClientFactory.client(DockerClientFactory.java:168)
                at org.testcontainers.LazyDockerClient.getDockerClient(LazyDockerClient.java:14)
                at org.testcontainers.LazyDockerClient.listImagesCmd(LazyDockerClient.java:12)
                at org.testcontainers.images.LocalImagesCache.maybeInitCache(LocalImagesCache.java:68)
                at org.testcontainers.images.LocalImagesCache.get(LocalImagesCache.java:32)
                at org.testcontainers.images.AbstractImagePullPolicy.shouldPull(AbstractImagePullPolicy.java:18)
                at org.testcontainers.images.RemoteDockerImage.resolve(RemoteDockerImage.java:59)
                at org.testcontainers.images.RemoteDockerImage.resolve(RemoteDockerImage.java:26)
                at org.testcontainers.utility.LazyFuture.getResolvedValue(LazyFuture.java:20)
                at org.testcontainers.utility.LazyFuture.get(LazyFuture.java:27)
                at org.testcontainers.containers.GenericContainer.getDockerImageName(GenericContainer.java:1277)
                ... 7 more

    java.lang.NullPointerException
        at org.hibernate.reactive.BaseReactiveTest.after(BaseReactiveTest.java:156)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at io.vertx.ext.unit.junit.VertxUnitRunner.invokeTestMethod(VertxUnitRunner.java:95)
        at io.vertx.ext.unit.junit.VertxUnitRunner.lambda$invokeExplosively$0(VertxUnitRunner.java:114)
        at io.vertx.ext.unit.impl.TestContextImpl.run(TestContextImpl.java:90)
        at io.vertx.ext.unit.junit.VertxUnitRunner.invokeExplosively(VertxUnitRunner.java:130)
        at io.vertx.ext.unit.junit.VertxUnitRunner.access$000(VertxUnitRunner.java:39)
        at io.vertx.ext.unit.junit.VertxUnitRunner$3.evaluate(VertxUnitRunner.java:217)
        at io.vertx.ext.unit.junit.Timeout$1.evaluate(Timeout.java:45)
        at org.junit.rules.RunRules.evaluate(RunRules.java:20)
        at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
        at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
        at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
        at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
        at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
        at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
        at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
        at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
        at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
        at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:110)
        at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:58)
        at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:38)
        at org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:62)
        at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
        at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
        at org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:33)
        at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:94)
        at com.sun.proxy.$Proxy2.processTestClass(Unknown Source)
        at org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:119)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
        at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
        at org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:182)
        at org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:164)
        at org.gradle.internal.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:414)
        at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
        at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:48)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at org.gradle.internal.concurrent.ThreadFactoryImpl$ManagedThreadRunnable.run(ThreadFactoryImpl.java:56)
        at java.lang.Thread.run(Thread.java:748)

Any ideas @Sanne, @DavideD ?

bug testing

Source

gavinking

Most helpful comment

Changing to testcontainers 1.15.0-rc2 seems to have fixed it for me.

gavinking on 4 Oct 2020

👍3

All 23 comments

What is a Ryuk and why should it be running at localhost:32768?

gavinking on 4 Oct 2020

Oh, wait, I just found this: https://github.com/testcontainers/testcontainers-java/issues/3241

gavinking on 4 Oct 2020

PosgreSQL recently released version 13, so I reconfigured our test images to use image postgres:13.0 . Sounds like you're unable to download it?

Weird, it's supposedly no different from fetching the previous versions.

Sanne on 4 Oct 2020

PosgreSQL recently released version 13, so I reconfigured our test images to use image postgres:13.0 . Sounds like you're unable to download it?

I think—but I'm not sure—that I already had it working on Postgres 13 up until I updated Docker.

gavinking on 4 Oct 2020

First time I hear about Ryuk .. from the very few doc references I found, it seems like an optional local server one is supposed to start?
It's responsible for fetching images on behalf of Testcontainers - so not having a new container (such as postgresql 13) would trigger this.

Seems a bug in Testcontainers.

You might be able to workaround it by downloading the container once manually?

try

docker run --rm -it --name HibernateTestingPGSQL \
    -e POSTGRES_USER=hreact -e POSTGRES_PASSWORD=hreact -e POSTGRES_DB=hreact \
    -p 5432:5432 postgres:13.0

Sanne on 4 Oct 2020

Changing to testcontainers 1.15.0-rc2 seems to have fixed it for me.

gavinking on 4 Oct 2020

👍3

Cool.
Alternatively: https://www.testcontainers.org/features/configuration/#disabling-ryuk

Sanne on 4 Oct 2020

👍1

Alternatively: https://www.testcontainers.org/features/configuration/#disabling-ryuk

Yep, that worked.

Is there a right way to set an environment variable from the gradle script?

gavinking on 4 Oct 2020

Apparently:

test {
    environment "TESTCONTAINERS_RYUK_DISABLED", "true"

gavinking on 4 Oct 2020

right, that should do it.

BTW please hold on pushing for some minutes :) release is running

Sanne on 4 Oct 2020

👍1

Release done.. I'll do the Quarkus updates now.

Sanne on 4 Oct 2020

Release done.. I'll do the Quarkus updates now.

Excellent!

gavinking on 4 Oct 2020

Is this still an issue?

DavideD on 13 Oct 2020

I believe the bug is still there, but the only people who are going to experience it are people who don't already have the thing fetched to their local machine.

gavinking on 13 Oct 2020

So I wonder if this was caused by the need to upgrade the test database when migrating from Postgres 12 to 13?

See, I have testcontainers.reuse.enable=true set, which perhaps reuses the database between test runs?

gavinking on 26 Oct 2020

I don't believe we strictly _need_ that upgrade to PostgreSQL 13 ?

AFAIR when I upgraded it I didn't need to make any related changes, so I'd expect everything to work fine on 12 as well - unless a regression was included in the meantime but that seems less likely than having some problems with docker/testcontainers.

Sanne on 26 Oct 2020

Maybe try to revert the configuration to 12 and see?

Sanne on 26 Oct 2020

I'm experiencing this again :-(

See, I have testcontainers.reuse.enable=true set, which perhaps reuses the database between test runs?

I set it to false, and it didn't fix the problem.

gavinking on 6 Nov 2020

environment "TESTCONTAINERS_RYUK_DISABLED", "true"

This fixed it again.

@Sanne Do you think I should push this change to the gradle script?

gavinking on 6 Nov 2020

yes, sure why not. It seems to be a new feature, and we've been happy so far without it.

Sanne on 6 Nov 2020

It's responsible for fetching images on behalf of Testcontainers - so not having a new container (such as postgresql 13) would trigger this.

It is actually responsible for terminating the containers after running the tests. If you kill the JVM, the shutdown hooks won't get executed and the containers will continue running, creating dangling containers.
The environment variable is there for CI systems where Ryuk cannot be used (e.g. BitBucket Pipelines) but they guarantee the cleanup themselves.

tl;dr: do not disable Ryuk ;)

bsideup on 7 Nov 2020

If you kill the JVM, the shutdown hooks won't get executed and the containers will continue running, creating dangling containers.

Well, TBH, I run with testcontainers.reuse.enable=true always, so I guess that's what happens anyway.

And, on balance, dangling containers is less worse than my tests just not running. :-)

But sure, if this is fixed in 1.15 (which just came out two days ago, I think) then that's great, and we'll just upgrade instead.

gavinking on 7 Nov 2020

Thanks for explaning that @bsideup !

Sanne on 7 Nov 2020

Was this page helpful?

0 / 5 - 0 ratings

Related issues

Enhancement - Integration with spring @Transactional annotation

yaakov-berkovitch · 16Comments

lazy collection fetching in Quarkus

markusdlugi · 30Comments

MS SQL: failure when inserting row if a column has a foreign key constraint

tsegismont · 20Comments

broken on Hibernate ORM master

gavinking · 16Comments

replicate HHH-14312

gavinking · 6Comments