Hibernate-reactive: Example fails with ORM 5.4.24

Goops, you're right: @Sanne and I both missed this because of other spurious failures in the CI.

I hazard a guess that this is related to #443.

Is that issue related to DirtyChecking?
I think that problem is related to this: https://github.com/hibernate/hibernate-reactive/issues/447

Anyway, it happens after the update to ORM 5.4.24

Is that issue related to DirtyChecking?

Well, I figured that the bug is related to batch loading. I might be wrong though.

Is that issue related to DirtyChecking?

Well, I figured that the bug is related to batch loading. I might be wrong though.

Oh well, it looks like I'm wrong and the problem is indeed elsewhere.

I'm working on a test case. Hopefully I will be able to confirm this soon enough.

It makes sense though because only the example fails and it is the only place where we enable the bytecode enhancements

OK, great.

So, yes. The problem is the dirty checking of the collection when byte code enhancements are enabled. I don't know why it was working before. Probably, this bug in ORM was hiding the problem and now that it's fixed we see it.

Uff, what a PITA.

OTOH, why does collection dirty checking even _matter_ when we don't even have CollectionPersisters yet?

This is the issue:

The enhanced method $$_hibernate_hasDirtyAttributes has changed, from

    public boolean $$_hibernate_hasDirtyAttributes() {
        boolean var1 = false;
        var1 = this.$$_hibernate_tracker != null && !this.$$_hibernate_tracker.isEmpty();
        return var1;
    }

to

    public boolean $$_hibernate_hasDirtyAttributes() {
        boolean var1 = false;
        var1 = this.$$_hibernate_tracker != null && !this.$$_hibernate_tracker.isEmpty() || this.$$_hibernate_areCollectionFieldsDirty();
        return var1;
    }

    public boolean $$_hibernate_areCollectionFieldsDirty() {
        boolean var1 = false;
        if (!var1 && this.$$_hibernate_collectionTracker != null) {
            if (this.movies == null && this.$$_hibernate_collectionTracker.getSize("movies") != -1) {
                var1 = true;
            } else if (this.movies != null && this.$$_hibernate_collectionTracker.getSize("movies") != this.movies.size()) {
                var1 = true;
            }
        }

        return var1;
    }

the call to this.movies.size() causes a LazyInitializationException

movies is the name of the association inside the test I've created

@DavideD so do you think that what is going on here is just that we don't have transparent lazy init (i.e. it is missing a call to fetch()), or is it something deeper?

@DavideD I've commented on https://github.com/hibernate/hibernate-orm/pull/3645, because that call to size() doesn't look right to me. (But I could easily be missing something.)

@DavideD so do you think that what is going on here is just that we don't have transparent lazy init (i.e. it is missing a call to fetch()), or is it something deeper?

I think that's the issue. I'm trying to see if I can fetch the collection somewhere so that it is already fetched.
By the way, everything works if I set the association for EAGER fetching.

I'm trying to see if I can fetch the collection somewhere so that it is already fetched.

OK, but don't kill yourself on that _right now_ because it still feels wrong to me that we're fetching an un fetched association with mappedBy just to dirty check its elements. Let's see what @beikov and @Sanne have to say.

(OTOH, even if core isn't doing the right thing _in this particular case_, we still might need the fix your working on in other cases.)

For additional info, this is branch with the test case (still a work in progress): https://github.com/DavideD/hibernate-reactive/tree/447-dirtychecking

Test:
https://github.com/DavideD/hibernate-reactive/blob/447-dirtychecking/it-bytecode-enhancements/src/test/java/org/hibernate/reactive/it/bytecodeenhancements/BytecodeTest.java

Entities:
https://github.com/DavideD/hibernate-reactive/tree/447-dirtychecking/it-bytecode-enhancements/src/main/java/org/hibernate/reactive/it/bytecodenhancements

Thanks @gavinking

Why is the movies collection != null? I tried to reproduce this in core, but couldn't because such a collection should never be non-null if lazy.

Well I recall that there was a kinda long discussion somewhere. An issue I opened where I argued that this "double-laziness" or collections was at least arguably bad in Hibernate ORM, and a much worse problem for HR, where fetching isn't transparent. We need to let you obtain a reference to the collection to call fetch() on it.

OK found it. So could this be related to #374?

And https://github.com/hibernate/hibernate-orm/pull/3558

Is collectionsInDefaultFetchGroupEnabled always true in RX?

Hmm, if I set it to true, the collection is eagerly initialized.

Is collectionsInDefaultFetchGroupEnabled always true in RX?

Yup, it is.

Hmm, if I set it to true, the collection is eagerly initialized.

Ummmm ... really? That wasn't the intention!

collectionsInDefaultFetchGroupEnabled is supposed to just include the collection proxy in the default fetch group. It's not supposed to actually fetch the collection itself.

So either I totally screwed up, or something has changed since I wrote that code.

I'm with you that the current approach to enhancement is not ideal and that the field should be initialized to a persistent collection, but I think that the issue you are having in RX might be caused by something else. When I set the flag to true, I see the collection being initialized through this trace:

org.hibernate.collection.internal.AbstractPersistentCollection.initialize(AbstractPersistentCollection.java:585)
org.hibernate.collection.internal.AbstractPersistentCollection.read(AbstractPersistentCollection.java:149)
org.hibernate.collection.internal.AbstractPersistentCollection$1.doWork(AbstractPersistentCollection.java:178)
org.hibernate.collection.internal.AbstractPersistentCollection$1.doWork(AbstractPersistentCollection.java:163)
org.hibernate.collection.internal.AbstractPersistentCollection.withTemporarySessionIfNeeded(AbstractPersistentCollection.java:264)
org.hibernate.collection.internal.AbstractPersistentCollection.readSize(AbstractPersistentCollection.java:162)
org.hibernate.collection.internal.PersistentBag.size(PersistentBag.java:371)
org.hibernate.test.bytecode.enhancement.dirty.DirtyTrackingCollectionTest$StringsEntity.$$_hibernate_removeDirtyFields(DirtyTrackingCollectionTest.java)
org.hibernate.test.bytecode.enhancement.dirty.DirtyTrackingCollectionTest$StringsEntity.$$_hibernate_clearDirtyCollectionNames(DirtyTrackingCollectionTest.java)
org.hibernate.test.bytecode.enhancement.dirty.DirtyTrackingCollectionTest$StringsEntity.$$_hibernate_clearDirtyAttributes(DirtyTrackingCollectionTest.java)
org.hibernate.tuple.entity.PojoEntityTuplizer.afterInitialize(PojoEntityTuplizer.java:228)
org.hibernate.persister.entity.AbstractEntityPersister.afterInitialize(AbstractEntityPersister.java:5093)
org.hibernate.engine.internal.TwoPhaseLoad.afterInitialize(TwoPhaseLoad.java:405)
org.hibernate.loader.plan.exec.process.internal.AbstractRowReader.afterInitialize(AbstractRowReader.java:271)
org.hibernate.loader.plan.exec.process.internal.AbstractRowReader.finishUp(AbstractRowReader.java:214)
org.hibernate.loader.plan.exec.process.internal.ResultSetProcessorImpl.extractResults(ResultSetProcessorImpl.java:96)
org.hibernate.loader.plan.exec.internal.AbstractLoadPlanBasedLoader.executeLoad(AbstractLoadPlanBasedLoader.java:105)
org.hibernate.loader.entity.plan.AbstractLoadPlanBasedEntityLoader.load(AbstractLoadPlanBasedEntityLoader.java:285)
org.hibernate.persister.entity.AbstractEntityPersister.doLoad(AbstractEntityPersister.java:4441)
org.hibernate.persister.entity.AbstractEntityPersister.load(AbstractEntityPersister.java:4431)
org.hibernate.event.internal.DefaultLoadEventListener.loadFromDatasource(DefaultLoadEventListener.java:569)
org.hibernate.event.internal.DefaultLoadEventListener.doLoad(DefaultLoadEventListener.java:537)
org.hibernate.event.internal.DefaultLoadEventListener.load(DefaultLoadEventListener.java:208)
org.hibernate.event.internal.DefaultLoadEventListener.proxyOrLoad(DefaultLoadEventListener.java:332)
org.hibernate.event.internal.DefaultLoadEventListener.doOnLoad(DefaultLoadEventListener.java:108)
org.hibernate.event.internal.DefaultLoadEventListener.onLoad(DefaultLoadEventListener.java:74)
org.hibernate.event.service.internal.EventListenerGroupImpl.fireEventOnEachListener(EventListenerGroupImpl.java:121)
org.hibernate.internal.SessionImpl.fireLoadNoChecks(SessionImpl.java:1186)
org.hibernate.internal.SessionImpl.fireLoad(SessionImpl.java:1175)
org.hibernate.internal.SessionImpl.access$2100(SessionImpl.java:193)
org.hibernate.internal.SessionImpl$IdentifierLoadAccessImpl.doLoad(SessionImpl.java:2786)
org.hibernate.internal.SessionImpl$IdentifierLoadAccessImpl.lambda$load$1(SessionImpl.java:2767)
org.hibernate.internal.SessionImpl$IdentifierLoadAccessImpl.perform(SessionImpl.java:2723)
org.hibernate.internal.SessionImpl$IdentifierLoadAccessImpl.load(SessionImpl.java:2767)
org.hibernate.internal.SessionImpl.find(SessionImpl.java:3322)
org.hibernate.internal.SessionImpl.find(SessionImpl.java:3284)
org.hibernate.test.bytecode.enhancement.dirty.DirtyTrackingCollectionTest.lambda$test$1(DirtyTrackingCollectionTest.java:57)
org.hibernate.testing.transaction.TransactionUtil.doInJPA(TransactionUtil.java:235)
org.hibernate.testing.transaction.TransactionUtil.doInJPA(TransactionUtil.java:276)
org.hibernate.test.bytecode.enhancement.dirty.DirtyTrackingCollectionTest.test(DirtyTrackingCollectionTest.java:56)
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
java.lang.reflect.Method.invoke(Method.java:498)
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
org.hibernate.testing.junit4.ExtendedFrameworkMethod.invokeExplosively(ExtendedFrameworkMethod.java:45)
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
java.util.concurrent.FutureTask.run(FutureTask.java:266)
java.lang.Thread.run(Thread.java:748)

So $$_hibernate_removeDirtyFields() is calling size() on an uninitialized collection. _Should_ it be?

I'm with you that the current approach to enhancement is not ideal and that the field should be initialized to a persistent collection, but I think that the issue you are having in RX might be caused by something else. When I set the flag to true, I see the collection being initialized through this trace:

I'm probably missing something, but isn't this the same issue?
It's still calling size() but from a different method

You are right. Apparently the collections aren't registered as being lazy which causes this problem. I'm investigating now.

Here a PR that should fix these issues: https://github.com/hibernate/hibernate-orm/pull/3664

Thanks @beikov, the test seems to work using that branch

@beikov excellent, thanks

Thanks @beikov, the test seems to work using that branch

@DavideD can you retest now, @beikov just updated the PR, let's see if it still fixes the problem.

Here a PR that should fix these issues: hibernate/hibernate-orm#3664

Confirmed, we just need a release of hibernate-core.

Fixed by #463

Thanks all