Hibernate-reactive: lazy collection fetching in Quarkus

Created on 12 Mar 2021  ·  30Comments  ·  Source: hibernate/hibernate-reactive

When trying to fetch an association lazily with Mutiny in Quarkus, it doesn't work if the fetch is performed in a different session than the one used to persist the data. If the same session is used (so fetching right after persisting in the same session), it works.

Expected behavior
It works also in different sessions.

Actual behavior
An exception is thrown when trying to fetch the association:

2021-03-12 09:56:23,634 ERROR [org.jbo.res.rea.com.cor.AbstractResteasyReactiveContext] (vert.x-eventloop-thread-2) Request failed: org.hibernate.LazyInitializationException: Collection cannot be initialized: com.example.Author.books
    at org.hibernate.reactive.session.impl.ReactiveSessionImpl.initializeCollection(ReactiveSessionImpl.java:330)
    at org.hibernate.collection.internal.AbstractPersistentCollection$4.doWork(AbstractPersistentCollection.java:589)
    at org.hibernate.collection.internal.AbstractPersistentCollection.withTemporarySessionIfNeeded(AbstractPersistentCollection.java:264)
    at org.hibernate.collection.internal.AbstractPersistentCollection.initialize(AbstractPersistentCollection.java:585)
    at org.hibernate.collection.internal.AbstractPersistentCollection.read(AbstractPersistentCollection.java:149)
    at org.hibernate.collection.internal.AbstractPersistentCollection$1.doWork(AbstractPersistentCollection.java:178)
    at org.hibernate.collection.internal.AbstractPersistentCollection$1.doWork(AbstractPersistentCollection.java:163)
    at org.hibernate.collection.internal.AbstractPersistentCollection.withTemporarySessionIfNeeded(AbstractPersistentCollection.java:264)
    at org.hibernate.collection.internal.AbstractPersistentCollection.readSize(AbstractPersistentCollection.java:162)
    at org.hibernate.collection.internal.PersistentBag.size(PersistentBag.java:371)
    at org.hibernate.bytecode.enhance.spi.interceptor.LazyAttributeLoadingInterceptor.takeCollectionSizeSnapshot(LazyAttributeLoadingInterceptor.java:160)
    at org.hibernate.bytecode.enhance.spi.interceptor.LazyAttributeLoadingInterceptor.lambda$loadAttribute$0(LazyAttributeLoadingInterceptor.java:110)
    at org.hibernate.bytecode.enhance.spi.interceptor.EnhancementHelper.performWork(EnhancementHelper.java:130)
    at org.hibernate.bytecode.enhance.spi.interceptor.LazyAttributeLoadingInterceptor.loadAttribute(LazyAttributeLoadingInterceptor.java:76)
    at org.hibernate.bytecode.enhance.spi.interceptor.LazyAttributeLoadingInterceptor.fetchAttribute(LazyAttributeLoadingInterceptor.java:72)
    at org.hibernate.bytecode.enhance.spi.interceptor.LazyAttributeLoadingInterceptor.handleRead(LazyAttributeLoadingInterceptor.java:53)
    at org.hibernate.bytecode.enhance.spi.interceptor.AbstractInterceptor.readObject(AbstractInterceptor.java:153)
    at com.example.Author.$$_hibernate_read_books(Author.java)
    at com.example.Author.getBooks(Author.java:43)
    at com.example.TestResource.lambda$getBooks$2(TestResource.java:61)
    at io.smallrye.mutiny.operators.UniOnItemTransformToUni.invokeAndSubstitute(UniOnItemTransformToUni.java:31)
    at io.smallrye.mutiny.operators.UniOnItemTransformToUni$2.onItem(UniOnItemTransformToUni.java:74)
    at io.smallrye.mutiny.context.ContextPropagationUniInterceptor$1.lambda$onItem$1(ContextPropagationUniInterceptor.java:31)
    at io.smallrye.context.impl.wrappers.SlowContextualExecutor.execute(SlowContextualExecutor.java:19)
    at io.smallrye.mutiny.context.ContextPropagationUniInterceptor$1.onItem(ContextPropagationUniInterceptor.java:31)
    at io.smallrye.mutiny.operators.UniSerializedSubscriber.onItem(UniSerializedSubscriber.java:85)
    at io.smallrye.mutiny.context.ContextPropagationUniInterceptor$1.lambda$onItem$1(ContextPropagationUniInterceptor.java:31)
    at io.smallrye.context.impl.wrappers.SlowContextualExecutor.execute(SlowContextualExecutor.java:19)
    at io.smallrye.mutiny.context.ContextPropagationUniInterceptor$1.onItem(ContextPropagationUniInterceptor.java:31)
    at io.smallrye.mutiny.operators.UniDelegatingSubscriber.onItem(UniDelegatingSubscriber.java:24)
    at io.smallrye.mutiny.context.ContextPropagationUniInterceptor$1.lambda$onItem$1(ContextPropagationUniInterceptor.java:31)
    at io.smallrye.context.impl.wrappers.SlowContextualExecutor.execute(SlowContextualExecutor.java:19)
    at io.smallrye.mutiny.context.ContextPropagationUniInterceptor$1.onItem(ContextPropagationUniInterceptor.java:31)
    at io.smallrye.mutiny.operators.UniSerializedSubscriber.onItem(UniSerializedSubscriber.java:85)
    at io.smallrye.mutiny.context.ContextPropagationUniInterceptor$1.lambda$onItem$1(ContextPropagationUniInterceptor.java:31)
    at io.smallrye.context.impl.wrappers.SlowContextualExecutor.execute(SlowContextualExecutor.java:19)
    at io.smallrye.mutiny.context.ContextPropagationUniInterceptor$1.onItem(ContextPropagationUniInterceptor.java:31)
    at io.smallrye.mutiny.operators.uni.builders.UniCreateFromCompletionStage.lambda$forwardFromCompletionStage$1(UniCreateFromCompletionStage.java:30)
    at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859)
    at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837)
    at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
    at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2073)
    at com.ibm.asyncutil.iteration.AsyncTrampoline$TrampolineInternal.unroll(AsyncTrampoline.java:127)
    at com.ibm.asyncutil.iteration.AsyncTrampoline$TrampolineInternal.lambda$unroll$0(AsyncTrampoline.java:123)
    at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859)
    at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837)
    at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
    at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2073)
    at org.hibernate.reactive.pool.impl.Handlers.lambda$toCompletionStage$0(Handlers.java:26)
    at io.vertx.sqlclient.impl.SqlResultHandler.complete(SqlResultHandler.java:98)
    at io.vertx.sqlclient.impl.SqlResultHandler.handle(SqlResultHandler.java:87)
    at io.vertx.sqlclient.impl.SqlResultHandler.handle(SqlResultHandler.java:33)
    at io.vertx.sqlclient.impl.SocketConnectionBase.handleMessage(SocketConnectionBase.java:241)
    at io.vertx.sqlclient.impl.SocketConnectionBase.lambda$init$0(SocketConnectionBase.java:88)
    at io.vertx.core.net.impl.NetSocketImpl.lambda$new$2(NetSocketImpl.java:101)
    at io.vertx.core.streams.impl.InboundBuffer.handleEvent(InboundBuffer.java:237)
    at io.vertx.core.streams.impl.InboundBuffer.write(InboundBuffer.java:127)
    at io.vertx.core.net.impl.NetSocketImpl.handleMessage(NetSocketImpl.java:357)
    at io.vertx.core.impl.ContextImpl.executeTask(ContextImpl.java:366)
    at io.vertx.core.impl.EventLoopContext.execute(EventLoopContext.java:43)
    at io.vertx.core.impl.ContextImpl.executeFromIO(ContextImpl.java:229)
    at io.vertx.core.net.impl.VertxHandler.channelRead(VertxHandler.java:163)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
    at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)
    at io.vertx.pgclient.impl.codec.PgEncoder.lambda$write$0(PgEncoder.java:78)
    at io.vertx.pgclient.impl.codec.PgCommandCodec.handleReadyForQuery(PgCommandCodec.java:138)
    at io.vertx.pgclient.impl.codec.PgDecoder.decodeReadyForQuery(PgDecoder.java:226)
    at io.vertx.pgclient.impl.codec.PgDecoder.channelRead(PgDecoder.java:86)
    at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
    at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
    at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
    at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)

Reproducer

https://github.com/markusdlugi/hibernate-reactive-lazy-fetching

Steps to reproduce the behavior:

  1. Start database using infrastructure/docker-compose.yml.
  2. Start Quarkus application using mvn quarkus:dev
  3. Send request: POST http://localhost:8080/test/working.
    3.1. This works and returns a list of books.
  4. Send request: GET http://localhost:8080/test/failing/1 (or other authorId created via POST http://localhost:8080/test/failing).
    4.1 This fails with the above exception.
  • Quarkus Version: 1.12.2.Final
  • Hibernate Reactive Version: 1.0.0.Beta4 (also tested with 1.0.0.CR1 by overwriting the version in Maven, same behavior)
  • Hibernate Core Version: 5.4.28.Final (also tested with 5.4.29.Final by overwriting the version in Maven, same behavior)
bug
Source
picture markusdlugi

Most helpful comment

Just tested it and can confirm that lazy collection fetching now works in Quarkus. Thanks once again :smile:

picture markusdlugi on 18 Mar 2021
🎉2

All 30 comments

/cc @DavideD @gavinking

markusdlugi picture markusdlugi on 12 Mar 2021

Yeah, I think we decided this was the correct thing to do: in JPA there's no notion of reassociating an entity instance with a session. That's a bit different to the really old usage pattern of Hibernate where you could use saveOrUpdate() to directly reassociate a detached object. In JPA the model for dealing with detached objects is the merge() operation.

(And there's good reasons to not have this. Experience is that in practice users get themselves into a mess with reassociation.)

Since loading a detached collection really implies reattaching its parent, we simply don't support this.

However, IIRC, I do let you do this in a StatelessSession, since it's much more compatible with that programming model. Try using a stateless session and let me know if it works for you.

gavinking picture gavinking on 12 Mar 2021

Hi @gavinking, thanks for the quick reply.

I'm not quite sure whether I'm getting what you're saying. So what I implemented was exactly what is described in the documentation:

session.find(Author.class, authorId)
    .chain(author -> Mutiny.fetch(author.getBooks()));

This is working if the author and his books have been persisted with the same session (i.e., in the same HTTP request), but it's not working if I use another session (i.e., another request).

So you are implying that this is intended behavior, at least for a "normal" (stateful?) session? This seems to me like a really standard use case, because I will almost always try to get some data from the database some time after persisting it, so in a different session. This would mean that using a normal session, lazy associations are not supported. In that case, I would argue that it should be explicitly mentioned in the documentation that a StatelessSession needs to be used if lazy associations are to be fetched.

markusdlugi picture markusdlugi on 12 Mar 2021

I'm saying that in the new session you should use getReference() to obtain a session-bound reference to the entity that owns the collection. Then you can fetch the collection however you like.

This is the standard JPA model, it's even a bit easier to use in HR because I overloaded getReference().

gavinking picture gavinking on 12 Mar 2021

@gavinking do you mean something like this?

Author reference = session.getReference( Author.class, authorId );
return Mutiny.fetch( reference.getBooks() );
DavideD picture DavideD on 12 Mar 2021

Sure, but I made it even easier. You can directly pass the detached author to getReference().

gavinking picture gavinking on 12 Mar 2021

Okay, I tried those suggestions, but unfortunately they didn't work either. Maybe you can be so kind to point out if I'm doing something incorrectly here:

https://github.com/markusdlugi/hibernate-reactive-lazy-fetching/blob/f3458eee969d7c3597560aaf8a646b7aad2ee9b0/src/main/java/com/example/TestResource.java#L55-L79

The approach using getReference() fails with the following exception:

2021-03-12 14:17:51,381 ERROR [org.jbo.res.rea.ser.cor.ExceptionMapping] (vert.x-eventloop-thread-10) Request failed : java.lang.NullPointerException
    at org.hibernate.engine.internal.AbstractEntityEntry.overwriteLoadedStateCollectionValue(AbstractEntityEntry.java:336)
    at org.hibernate.persister.entity.AbstractEntityPersister.initializeLazyProperty(AbstractEntityPersister.java:1144)
    at org.hibernate.persister.entity.AbstractEntityPersister.initializeEnhancedEntityUsedAsProxy(AbstractEntityPersister.java:4497)
    at org.hibernate.bytecode.enhance.spi.interceptor.EnhancementAsProxyLazinessInterceptor.forceInitialize(EnhancementAsProxyLazinessInterceptor.java:221)
    at org.hibernate.bytecode.enhance.spi.interceptor.EnhancementAsProxyLazinessInterceptor.lambda$handleRead$0(EnhancementAsProxyLazinessInterceptor.java:133)
    at org.hibernate.bytecode.enhance.spi.interceptor.EnhancementHelper.performWork(EnhancementHelper.java:130)
    at org.hibernate.bytecode.enhance.spi.interceptor.EnhancementAsProxyLazinessInterceptor.handleRead(EnhancementAsProxyLazinessInterceptor.java:98)
    at org.hibernate.bytecode.enhance.spi.interceptor.AbstractInterceptor.readObject(AbstractInterceptor.java:153)
    at com.example.Author.$$_hibernate_read_books(Author.java)
    at com.example.Author.getBooks(Author.java:43)
    at com.example.TestResource.getBooksUsingReference(TestResource.java:69)
    ...

The other suggested solution using StatelessSession fails with:

2021-03-12 14:21:11,840 ERROR [org.jbo.res.rea.ser.cor.ExceptionMapping] (vert.x-eventloop-thread-5) Request failed : org.hibernate.LazyInitializationException: Unable to perform requested lazy initialization [com.example.Author.books] - no session and settings disallow loading outside the Session
    at org.hibernate.bytecode.enhance.spi.interceptor.EnhancementHelper.throwLazyInitializationException(EnhancementHelper.java:199)
    at org.hibernate.bytecode.enhance.spi.interceptor.EnhancementHelper.performWork(EnhancementHelper.java:89)
    at org.hibernate.bytecode.enhance.spi.interceptor.LazyAttributeLoadingInterceptor.loadAttribute(LazyAttributeLoadingInterceptor.java:76)
    at org.hibernate.bytecode.enhance.spi.interceptor.LazyAttributeLoadingInterceptor.fetchAttribute(LazyAttributeLoadingInterceptor.java:72)
    at org.hibernate.bytecode.enhance.spi.interceptor.LazyAttributeLoadingInterceptor.handleRead(LazyAttributeLoadingInterceptor.java:53)
    at org.hibernate.bytecode.enhance.spi.interceptor.AbstractInterceptor.readObject(AbstractInterceptor.java:153)
    at com.example.Author.$$_hibernate_read_books(Author.java)
    at com.example.Author.getBooks(Author.java:43)
    at com.example.TestResource.lambda$getBooksUsingStatelessSession$3(TestResource.java:78)
    ...
markusdlugi picture markusdlugi on 12 Mar 2021

I will have a look, it seems related to bytecode enhancements

DavideD picture DavideD on 12 Mar 2021

@DavideD, any update?

I tried to reproduce the issue without Quarkus by just using the session-example in this repo, but I was unable to do so. With just Hibernate Reactive and plain Mutiny, it works as expected. It also works using find(), no getReference() or StatelessSession necessary.

So this issue only seems to happen in Quarkus.

markusdlugi picture markusdlugi on 15 Mar 2021

I'm still looking into it.

The problem happens because of bytecode enhancements that are enabled by default in Quarkus.
I haven't still figure out which set of properties recreate the issue when using only Hibernate Reactive. But I will continue to work on it today

DavideD picture DavideD on 15 Mar 2021
👍1

I can now recreate the error in reactive. It requires two things:

  1. set hibernate.bytecode.allow_enhancement_as_proxy to true
  2. set SessionFactoryOptions#enableCollectionInDefaultFetchGroup to false. It should be set to true: https://github.com/hibernate/hibernate-reactive/blob/a3f4c61eef71baab8ca321da5af0bfb9052bde4b/hibernate-reactive-core/src/main/java/org/hibernate/reactive/provider/service/ReactiveSessionFactoryBuilderService.java#L27

@gavinking Am I correct in thinking that we should make it sure that in Quarkus enableCollectionInDefaultFetchGroup is set to false? I admit that at the moment I don't understand the whole issue fully (I've tried to follow the conversation on issue #374)

DavideD picture DavideD on 16 Mar 2021

Um. I think it only makes sense to have the collection in the default fetch group. I don't think it ever makes sense to exclude it.

gavinking picture gavinking on 16 Mar 2021

n Quarkus enableCollectionInDefaultFetchGroup is set to false

Sorry, yeah, I meant to write is set to true . Time for a break :-)

DavideD picture DavideD on 16 Mar 2021

Ah OK good :-) Take a break :-)

gavinking picture gavinking on 16 Mar 2021

@DavideD That's some great findings, thank you :)

Just wanted to point out, that according to the discussion in #374 and the related hibernate/hibernate-orm#3558, it seems like enableCollectionInDefaultFetchGroup was deliberately left to be false in Quarkus? At least that's what I'm gathering from this comment:

https://github.com/hibernate/hibernate-orm/pull/3558#issuecomment-695003875

Anyways, it does look like Quarkus currently doesn't set that property to true, because it is unconfigured in FastBootReactiveEntityManagerFactoryBuilder:

https://github.com/quarkusio/quarkus/blob/55cf15173ff7b2985a8422de050d1c9c708be57c/extensions/hibernate-reactive/runtime/src/main/java/io/quarkus/hibernate/reactive/runtime/boot/FastBootReactiveEntityManagerFactoryBuilder.java#L26-L32

And if I checked it correctly, the default in Hibernate ORM still seems to be false, which would explain why we see this behavior in Quarkus only.

markusdlugi picture markusdlugi on 17 Mar 2021

@Sanne needs to look at this one.

gavinking picture gavinking on 17 Mar 2021

If we only have to set it to true, it seems pretty easy to solve. I will create a fix and then check with @Sanne

DavideD picture DavideD on 17 Mar 2021

Well what I'm wondering is if it's set to false because that is what ORM needs, and if we need to be careful to not step on regular Hibernate?

gavinking picture gavinking on 17 Mar 2021

great find, thanks. Yes it's a little tricky because the two extensions ("regular" and "reactive") are highly coupled ATM.

Need to separate them properly, that was developed in too much rush.

Sanne picture Sanne on 17 Mar 2021

(OTOH, I'm not sure why ORM would need it; as I've argued before, I think it should be on by default.)

gavinking picture gavinking on 17 Mar 2021

hum ok. Let's do this for both ORMs (in Quarkus).

Sanne picture Sanne on 17 Mar 2021

@gavinking, I think that title is wrong, isn't it? I guess you meant to put enableCollectionInDefaultFetchGroup there instead. Bytecode enhancements are necessary for Quarkus anyway because of native, I assume :)

markusdlugi picture markusdlugi on 17 Mar 2021

Ah yeah, sure, I guess you're right.

you're saying that you see this bug with default settings in Quarkus?

gavinking picture gavinking on 17 Mar 2021

Yes, no special configuration is necessary in Quarkus to get this bug. @DavideD just had to do some special configuration to reproduce in plain HR (without Quarkus).

markusdlugi picture markusdlugi on 17 Mar 2021

Sure.

The thing is it's not actually a bug in HR. I deliberately added that "collection in default fetch group" thing to core specifically because we need it on all the time in HR.

So the bug is in the Quarkus extension which turns it off.

(Or, arguably, in core for doing the wrong thing by default.)

gavinking picture gavinking on 17 Mar 2021
👍2

I've sent a fix for Quarkus: https://github.com/quarkusio/quarkus/pull/15818
getReference doesn't seem to work on quarkus at the moment. I will create a separate issue for it.

DavideD picture DavideD on 17 Mar 2021
❤1 👍1

Sounds great, thank you! 👍

markusdlugi picture markusdlugi on 17 Mar 2021

Just tested it and can confirm that lazy collection fetching now works in Quarkus. Thanks once again :smile:

markusdlugi picture markusdlugi on 18 Mar 2021
🎉2

Thank you @markusdlugi

DavideD picture DavideD on 19 Mar 2021

Thanks for reporting it @markusdlugi

gavinking picture gavinking on 19 Mar 2021
Was this page helpful?
0 / 5 - 0 ratings

Related issues

bharward picture bharward  ·  13Comments
gavinking picture gavinking  ·  16Comments
akoufa picture akoufa  ·  30Comments
DavideD picture DavideD  ·  37Comments
blafond picture blafond  ·  7Comments