Jdbi: Make @CreateSqlObject less confusing.

Thanks for reporting this. I suspect we will end up with a technically breaking change here, but I think the existing behavior is poor enough that we make a technically breaking change early in the 3.x release cycle. Please flag this issue if you depend on the existing behavior or disagree with potentially shipping a small breaking change here.

Just for context as to _why_ @CreateSqlObject doesn't play nice with on-demand:

• On-demand is a core-level abstraction, implemented using proxies. When you call a method on an on-demand proxy, a real SQL object is created and the method call is delegated to the real SQL object. The handle backing that real instance is closed after the delegate method call returns.
• @CreateSqlObject is implemented using handle.attach(sqlObjectType), with the original SQL Object's backing handle.

Thus, the backing handle for the created SQL Object is closed before the SQL Object can even be returned.

None of the above is set in stone--it's just how it's implemented now.

I'm not convinced we'll have to break compatibility to fix this.

Sorry, I don't really get it... What's the issue with onDemand and CreateSqlObject exactly? I've never had any problems with it, and something about @qualidafial's explanation is just making my mind go tilt...

A connection is acquired for the call to fooProxy.usecase. Foo.usecase calls the createSqlObject method bar, which returns a new Bar after attaching it to the current (usecase's) handle. usecase's handle is closed when usecase returns. As long as you're not doing anything asynchronous with the returned Bar, how can bar's handle expire too early? bar's lifecycle and use is limited to the body of Foo.usecase, as is the handle...

I can't speak to the internals of JDBI, however I can comment on the behavior we saw in production.

Basically, in our lower environments things were fine. But with the increased load in production, we continually would see SQL statements not execute against the proper database node. For example, a Read Only select would hit the Master node, and an Insert would hit a read replica. In addition, we would get connection closed errors where it was in the middle of an operation.

Essentially, it seemed @CreateSqlObject created a thread safety issue - where the read-only flag was being altered by competing requests.

To "solve" the issue - we removed all usage of @CreateSqlObject, and ended up with something like this:

• class FooDao
• class BarDao
• class CombinedDao extends FooDao, BarDao

And then we jdbi.onDemand(CombinedDao) and only use access via that.
Switching to that pattern, while not as pretty, got rid of all the aforementioned errors in production.

Just for info, I'm thinking of going back to JDBI 2 because all my code uses onDemand with abstract classes at the moment, and I can't work out how to restructure in a way I'll be happy with!

I tend to have service interfaces, with abstract implementations classes containing logic but no SQL, with methods annotated as transactional, using CreateSqlObject to provide access to DAO classes which are purely SQL. A simplified example:

Interface

public interface AccountService {
    void addAccount(Account account, User user);
}  

Implementation

public abstract class AccountServiceJdbi implements AccountService {

    @Override  
    @Transaction  
    public final void addAccount(@BindBean() Account account, User user) {
        long accountId =  accountDao().insertAccount(account);
        accountDao().linkAccountToOwner(accountId, user.getId());
    }

    @CreateSqlObject
    abstract AccountDao accountDao();
}

(You can imagine the Dao)

This gives a really nice separation of logic and data access, while allowing transactions over multiple DAO methods. Unit tests for the service are easy to write and understand by implementing the DAO interfaces.

I've tried to do similar in JDBI 3, but I think it means the implementation of my service class has to be an interface with default methods for the ones containing logic. Default methods can't be made final, so the code just doesn't feel as concise, and gives me less control over how my class can be used.

Is there any way I could structure my code in JDBI 3 to have final transactional methods?

You are correct that there is no way to define an interface method as final.

You _can_ however define your own SQL method annotations on par with @SqlQuery, @SqlUpdate, etc, and provide a static implementation for methods with that annotation.

You _can_ however define your own SQL method annotations on par with @SqlQuery, @SqlUpdate, etc, and provide a static implementation for methods with that annotation.

Thanks for the reply - I don't quite follow how that helps me. What is the recommended way to have queries in multiple DAOs, but execute them in the same transaction?

way to have queries in multiple DAOs, but execute them in the same transaction?

Personally I use something like

interface Dao1 {
  @SqlQuery("...")
  void query1();
}

interface Dao2 {
  @SqlQuery("...")
  void query2();
}

interface JdbiServiceImpl extends Service {
  @CreateSqlObject
  Dao1 dao1();
  @CreateSqlObject
  Dao2 dao2();

  @Transaction
  @Override
  void businessCase() {
    dao1().query1();
    dao2().query2();
  }
}

Service service = handle.attach(JdbiServiceImpl.class);
service.businessCase();

Works great for me this way. @CreateSqlObject is basically like spring's dependency injection by getter/setter. I put onDemand instances on my spring context as beans so it works exactly like regular services, callers don't need to know about jdbi or the implementation interface.

The anti-pattern-looking StockReductionCase thing you see is an example of nested jdbi "dependency injection" with CreateSqlObject. It has dependencies of its own inside it, just like the service implementation. It's basically a service on its own, I just call it Case instead of Service to avoid cyclic dependency by only ever putting Cases (needed in multiple places) inside Services (top-level non-reusable logic) and Queries into either.

I've heard mixed reports of success getting transactions to scope correctly with @CreateSqlObject, especially when combined with onDemand().

The surest way to have multiple SQL objects execute in the same transaction is run transactions via Handle.inTransaction() or Jdbi.inTransaction(). Inside the callback, any DAOs created via Handle.attach() will be part of the handle's transaction:

jdbi.useTransaction(handle -> {
  Dao1 dao1 = handle.attach(Dao1.class);
  Dao2 dao2 = handle.attach(Dao2.class);

  dao1.doStuff();
  dao2.doMoreStuff();
});

Thanks for the replies
@qualidafial this looks like the way I'll have to go, but it does make it hard to write unit tests for the service class. As it instantiates the Daos itself I can't substitute mock Daos for unit tests.

@TheRealMarnes thanks too - this looks similar to what I'd tried, I just don't like using default methods as they can be overridden.

@qualidafial I would like to confirm with you that we cannot use jdbi's @Transaction annotation on service methods that are invoking multiple dao methods?

If this assumption is correct, this behavior is very dangerous without being properly documented as a part of official documentation. The large number of developers have experience with Spring stack and using @Transactional annotation is very standard way to support transactions across multiple DAOs.

@svlada The @Transaction annotation _only_ works on methods of SQL objects. When you say "service methods" I get the impression you're using the annotation on some object outside of Jdbi's influence, e.g. an injected class.

An example from our test suite:

@Test
public void testInsertAndFind() {
    Foo foo = handle.attach(Foo.class);
    Something s = foo.insertAndFind(1, "Stephane");
    assertThat(s).isEqualTo(new Something(1, "Stephane"));
}

@Test
public void testTransactionPropagates() {
    Foo foo = dbRule.getJdbi().open().attach(Foo.class);

    assertThatExceptionOfType(Exception.class)
        .isThrownBy(() -> foo.insertAndFail(1, "Jeff"));

    Something n = foo.createBar().findById(1);
    assertThat(n).isNull();
}

public interface Foo {
    @CreateSqlObject
    Bar createBar();

    @SqlUpdate("insert into something (id, name) values (:id, :name)")
    int insert(@Bind("id") int id, @Bind("name") String name);

    @Transaction
    default Something insertAndFind(int id, String name) {
        insert(id, name);
        return createBar().findById(id);
    }

    @Transaction
    default Something insertAndFail(int id, String name) {
        insert(id, name);
        return createBar().explode();
    }
}

public interface Bar {
    @SqlQuery("select id, name from something where id = :id")
    Something findById(@Bind("id") int id);

    default Something explode() {
        throw new RuntimeException();
    }
}

I also want to clarify with respect to onDemand() + @CreateSqlObject:

• @CreateSqlObject DAOs may only be used from inside methods of the DAO that created them--as in the example above.
• Calling fooDao.createBar().findById() will throw an exception stating that the connection is closed.

Sorry to comment on this again. Is there anything in your road map that will help with this?

I still really miss being able to use CreateSqlObject in abstract classes. Having to do everything in interfaces still feels restrictive; often I would want to make a "service" type method transactional, but this would mean my Service classes had to also be interfaces using default methods and I start losing the clarity of separation of concerns, concrete final methods etc.

@tamslinn as far as your problem with interfaces instead of abstract classes is concerned, I think I can safely say we won't be coming back from that decision in the near future. Jdbi3 uses jdk Proxies, which only supports interfaces.

I don't think a contributed separate module built on cglib or some such that offers this feature is entirely out of the question, but it's not a concern right now.

If the sqlobject axioms suit you that badly, you could always refactor your Services to be regular classes not enhanced by jdbi but rather having a Jdbi instance injected to use the fluent API instead.

As for the original subject of the behavior between our transaction support, onDemand, etc, that's becoming more and more of a focus to work on, though I don't think we have really concrete plans yet. But we're getting to it.

Thanks very much for the update. Totally makes sense re the proxies.
The specific limitation for me is not being able to start/end transactions outside of the the JDBI classes.
(As mentioned by @svlada it's quite common in other frameworks to use @Transactional at a higher level)
I really like everything else about JDBI though, so I'll keep at it, maybe I'll come up with a design pattern I'm happy with for my service classes :)

Sorry to comment on this again. Is there anything in your road map that will help with this?

No problem at all. This is clearly a source of confusion, and something we'd like to fix -- we just really want to make sure to get it right this time, since we didn't last time :)

(As mentioned by @svlada it's quite common in other frameworks to use @Transactional at a higher level)

Yeah, but as far as I understand, that's all done via AOP hooks in your DI framework (like Spring). Since JDBI doesn't "wrap" all your service objects, we do not have any opportunity to provide AOP-like solutions for objects that we do not create ourselves. Even the old cglib implementation would only notice @Transactional on the daos themselves, it would never have worked on a separate service class.

If it would be helpful, we could consider adding e.g. Spring and/or Guice AOP bindings to allow transactions at a higher level. This would not be part of core but e.g. part of the spring extensions. This isn't likely to be something the core developers jump on, but a contribution would be seriously considered for inclusion. Maybe that solves your problem in a "cleaner" way?

We decided to switch from cglib to Proxy mostly for compatibility reasons -- Proxy is a supported jdk api, whereas cglib (or more particularly asm) tends to break with every major release (it broke on 8, 9, 11, ...) which turns into a huge maintenance headache.

Maybe https://github.com/jdbi/jdbi/pull/1252 is relevant?

Hi,

Thanks for the info @stevenschlansker. I'm not actually using Spring with JDBI for my current project, was just mentioned for comparison. I totally understand re the AOP stuff, just trying to find a way round it that means I can split the code how I want to - the abstract classes used to work for me, but it makes complete sense to move away from cglib

I'm thinking now I'll just construct my service classes when I need them, that way I can construct them inside a transaction.

         try (Handle handle = jdbi.open()) {
                handle.useTransaction(h -> {
                    AccountService accountService = new AccountServiceImpl(h.attach(AccountDao.class));                    
                    accountService.addAccount(a, u);
                });
          }

I think this will mean I can unit test the service with a Mock Dao, keep service methods final and not rely on default methods in interfaces for transaction management, and the added overhead of service class instantiation shouldn't have significant impact.

Just a quick reply to the code example in your last comment: you could skip the try block and just call jdbi.useTransaction() directly. This method allocates a temporary handle which is automatically closed when your callback returns.

Hi everyone, there's a PR #1579 -- as of jdbi 3.10.0, CreateSqlObject and onDemand should (finally) play nicely together.