Jdbi: Builtin mapper for ZonedDateTime discards time zone information

You are correct, but this is all that JDBC supports at this time. We have to convert all java.time types to java.util or java.sql types which lack time zone / offset data.

Out of curiosity, are you using Postgres? Because their TIMESTAMP WITH TIME ZONE behaves differently than you'd expect.

are you using Postgres? Because their TIMESTAMP WITH TIME ZONE behaves differently than you'd expect.

i am aware. (Big disappointment for otherwise very standards-compliant database btw.)

You are correct, but this is all that JDBC supports at this time.

I was expecting JDBI to pull H2's org.h2.api.TimestampWithTimeZone without needing me to do this manually, _somehow_.
As I don't know of any "portable" (working across different drivers) way to pull TIMESTAMP W/TZ value, I don't have any true solution.
However, if you don't pull all information, you should not pretend you do -- i.e. IMO there should be no builtin mapper to ZonedDateTime. Otherwise you're doing just like Postgres -- not what I'd expect.

I think it's still convenient if a user wants to use ZonedDateTime and doesn't care that it's is returned in the app server's timezone instead of the database's timezone.

By the way, Dropwizard uses a different version of the mapper for JDBI2, where a user can explicitly set the database's timezone: https://github.com/dropwizard/dropwizard/blob/master/dropwizard-jdbi/src/main/java/io/dropwizard/jdbi/args/ZonedDateTimeMapper.java

If you could provide example code for how to get/set ZonedDateTime with H2, I'd be happy to add that to the plugin.

I think it's still convenient if a user wants to use ZonedDateTime and doesn't care that it's is returned in the app server's timezone instead of the database's timezone.

@arteam , If user is aware of making this choice -- i agree.

However, AFAIR https://github.com/jdbi/jdbi/blob/master/core/src/main/java/org/jdbi/v3/core/mapper/BuiltInMapperFactory.java#L182 won't even work on H2 for TS w TZ, as rs.getTimestamp(..) is not supported for that type.

If you could provide example code for how to get/set ZonedDateTime with H2, I'd be happy to add that to the plugin.

I don't know of any portable way -- the only way I know is (org.h2.api.TimestampWithTimeZone) rs.getObject(..) and for e.g. Oracle it would be similar, but with Oracle-specific class.

Generally, I'd prefer to be _forced_ by the API to use correct transformation (timestamp to LocalDateTime) and add "zone-ing" explicitly.

You seem to disagree, which means, there is no point in keeping this issue open.

One thing we do in our stack, we actually assert that Java's timezone == Postgres's timezone, in which case it all works just right. Should we maybe emit a warning if there is a mismatch? Would that be an OK compromise between enabling features people expect, and being TZ-correct?

Not sure why you closed this so quickly, I felt there was more to discuss.

Even if we can't resolve this generically for all vendors, we can definitely support this on a per-vendor basis.

The mappers and argument factories that ship with Jdbi can be overridden simply by registering another mapper/argument. Last registered (per type) wins.

If you look at the source for PostgresPlugin, you'll see that it registers JavaTimeMapperFactory which provides alternative mappers using ResultSet.getObject(Class) for LocalDate, LocalTime, LocalDateTime and OffsetDateTime, which are directly supported by the Postgres JDBC driver.

If H2 has a way to store ZonedDateTime without eliding the zone, we can modify H2DatabasePlugin to use a custom mapper for that.

I'll reopen this for now, there is good discussion to be had here, and @findepi we do care a lot about doing things correctly! We are also just pragmatic and people expect to actually be able to use the time types...

Do I understand correctly from this convo that the jdbi team would be interested in having e.g. a jdbi3-hsqldb subproject/artifact/... that provides arguments and mappers specifically for hsqldb (which supports using java.time objects directly through setObject/getObject without conversion)?

Absolutely.

To be honest I agree with @findepi where he said that jdbi shouldn't provide mappers for things that actually cannot be accurately mapped, like timezones/offsets that aren't persisted... The idea of a "compromise" here sounds very dangerous to me - I've always considered full data integrity to be one of the highest priorities in any project. Users shouldn't need to double-check that libraries are working correctly.

I'd rather find out with a runtime exception during a test run that jdbi doesn't provide a mapper for a given type and then write one myself, than find out who-knows-when that a mapper provided by jdbi has been sneakily altering my data.

I have a large unit test that writes and then reads values of many types, to verify implementation correctness/data integrity in both the db itself (hsqldb) and jdbi, so I could write jdbi mappers/arguments to handle any issues. I've had to write several now because hsqldb currently has basically no support for java.time despite the spec saying so (huge bug). I hadn't thought of testing java.time objects with zones/offsets other than my system one yet, so I just did, and sure enough, they all come back with the offset/zone reset to system values, but due to jdbi's mapping to Timestamp, not because of hsqldb bugs.

I suggest those flawed mappers be removed from jdbi3 and moved into vendor-specific artifacts (jdbi3-hsqldb and others) where their implementation can be made fully correct.

I might be able to contribute jdbi-hsqldb myself since I've written all the necessary mappers and arguments for my own project already anyway.

I agree that probably would have been the wiser path, however at this point v3 is released and the cat is out of the bag.

Removing these mappers or argument factories would be a breaking change to any existing users who depend on them.

If someone depends on them already (aware of their flaws or not), I tihnk it would be simple enough for those users to patch them back in: the conversions are pretty much one-liners. Keeping them in on the other hand will lead to ever more and more people like me who assume it to work because it's provided (honestly, who knows all the technical specs like jdbc's non-support of timezones by heart?) and may or may not find out later on that their data was being mishandled...
Basically I think the potential future damage by allowing them to stay is way greater than the damage caused now to the few who have updated by removing them.

@TheRealMarnes i think removing a feature (even once we agree it should not exist) is not so simple. People tend to assume semantic versioning (regardless whether the project follows it or not). I don't know the project rules, but usually you would at most deprecate it in v3 (here deprecation is tricky, because there is no API method that can be marked deprecated, documentation and logging is what's left) and remove the feature in v4, expecting people to read release notes / migration guide.

I agree something like that is never simple and the solutions will always make someone unhappy, but I tend to think the end justifies the means and upsetting a flawed system to replace it by a better one is in the long term always worth the upset. And I don't think expecting people to read release notes / migration guide is unreasonable... With maintenance and minor updates, sure, but not with a major update. Jdbi3 itself is unusable coming from jdbi2 without reading the new guide too.

I think I could get behind a solution to disable those mappers / arguments, provided reinstating them was a one-liner--maybe move those mappers to a plugin?

Like a non-vendor-specific jdbi3-core-experimental?
Edit: my bad, you mean jdbi.installPlugin(new ExperimentalMappersPlugin())

That'd be the idea, although these aren't really 'experimental' as such.

TimezoneInsensitiveJavaTimeMappers then :P

It's not just the mappers, it's also the argument factories.

How about JavaTimeYoloTZPlugin :wink:

Even if we can't resolve this generically for all vendors, we can definitely support this on a per-vendor basis.

@qualidafial i learned recently that there seems to be an emerging de-facto standard for retrieving date/time values in post-Java8 world.

First, a little bit more background:

• i created this issue about JDBI's TIMESTAMP WITH TIME ZONE conversion to ZonedDateTime, which discards time zone information
• there is another issue, with all JDBC API. That is how TIMESTAMP, DATE, TIME objects are supposed to be retrieved, i.e. using java.sql.{Timestamp,Date,Time} classes. The classes all extend from java.util.Date. Anyone remember how Calendar was introduced and java.util.Date's methods dealing with year/month/day/hour/minute/second where deprecated? For a good reason so (and I don't mean Calendar solved all the problems).
  
  
  
  • you cannot retrieve java.sql.Timestamp from a database, if your _JVM time zone_ didn't observe such a local time. So an app running in US can store a timestamp value of 2017-03-26 02:30:00 (without zone!) in some shared database, but an app running in Europe (with eg Europe/Berlin as JVM zone) won't be able to retrieve this timestamp, because there was DST in Europe at that time.
  
  
  • oh, same applies to java.sql.Date, because this is almost identical class. java.sql.Date represents date just as if it was java.sql.Timestamp at midnight, but sometimes there is no midnight. There are zones with DST changes at midnight (either current or in the past).
  
  
  • would you think java.sql.Time is free of this problem, because it represents retrieved time as a timestamp on 1970-01-01? Unfortunately no, since there are zones that were nice enough to have policy change on that date (e.g. America/Bahia_Banderas, America/Hermosillo). If JVM is one if these zones, a program using JDBC won't be able to retrieve certain TIME values.
  
  

A non-solution would be to retrieve using JDBC old API and convert them to new classes. But this is generally wrong:

• ~resultSet.getTimestamp(col).toLocalDateTime()~
• ~resultSet.getDate(col).toLocalDate()~
• ~resultSet.getTime(col).toLocalTime()~

This compiles and works nice, except this doesn't solve any of the problems described above (even retrieving date doesn't work, if JVM zone is Pacific/Apia and date retrieved is 2011-12-30;)

Now, to the emerging solution. ResultSet has this less-popular API allowing you to ask JDBC driver to convert type for you:

• resultSet.getObject(col, LocalDateTime.class) // for TIMESTAMP
• resultSet.getObject(col, LocalDate.class) // for DATE
• resultSet.getObject(col, LocalTime.class) // for TIME
• resultSet.getObject(col, ZonedDateTime.class) // for TIMESTAMP WITH TIME ZONE

This appears to be working with JDBC drivers for PostgreSQL, MySQL, Oracle and H2. And, magic, circumvents the problems of "unrepresentability in JVM zone" described above.
Even though this conversion is not yet implemented in e.g. SQL Server driver, I think this could be made _the_ default mapping for Java Time classes that JDBI ships.

Note:
JDBC 4.2 spec doesn't mandate that a driver must implement this. However, the spec requires that PreparedStatement.setObject and RowSet.setObject accept instances of the Java Time classes and treat them correctly, so drivers need to support these classes explicitly anyway.

That's cool! My main concern is that changing to this approach could manifest as a regression for users using esoteric drivers. But maybe with appropriate messaging on the release notes we could make this small technically breaking change in the name of doing things The Right Way...

@stevenschlansker you are right. I think JDBI could provide

• optional set of mappings that a user can plug in to restore original behavior
• some appropriate commentary in the release notes (or even a major version bump)

Let's keep it to a minor version, we just released 3, and this is only a breaking change if you hold it wrong... ;)

Just to play back what it sounds like I'm hearing:

• Move the existing built-in java.time mappers into a separate plugin (I like YoloTimePlugin)
• Add new built-in mappers in their place that use ResultSet.getObject(column, type) instead of converting from java.sql types.
• Document the shit out of this breaking change in the release notes: explain the impact of this change and which database vendors will be impacted, and what users need to do to mitigate problems

I can get behind this.

We still haven't answered the question about binding these date/time objects though. Anything we change about mapping results _out of_ the database needs to be mirrored in the way we bind date/time arguments _into_ SQL statements.

Anything we change about mapping results out of the database needs to be mirrored in the way we bind date/time arguments into SQL statements.
@qualidafial , very good question. I haven't been testing this. This is explicitly covered by JDBC 4.2 spec, so _should just work™_.

This is explicitly covered by JDBC 4.2 spec, so _should just work™_.

This presumes that JDBC driver implementors follow the spec perfectly. I've yet to find a JDBC driver implementation without bugs.

I get the feeling it would be too cumbersome to provide plugins with mappers for every database and even its different versions that have different support for datatypes. Maybe a more blank slate approach could work: you start with a Jdbi instance with no mappers preinstalled, and the core artifact provides a bunch of different mappers that you can manually pick to install, some having different logic to handle the same datatypes - get/setObject, get/setX, conversion to other type, etc. You'd need to figure out for yourself which mappers are best for your database (version) and application logic, and use unit tests to keep it in check.

That's basically what I'm already doing now anyway in my project. There are some mappers preinstalled in Jdbi that don't work with my db and I don't need them for now, so I've actually overridden them with argument/mapper factories that throw an UnsupportedOperationException, and others I've overridden with better implementations (see below).

This would be a solution for those who like to get intimate with their db to make sure everything absolutely works, and would change nothing for people who want stuff to Just Werk™ (insofar as it actually does):

• add a method to Jdbi instances to clear out all their mappers/arguments (clean slate)
• expose the current built-in mapper/arg impls for us to manually install
• keep adding alternative mapper impls to the core artifact when demand for them rises and let people install them on their own, generifying whatever possible like below

By doing this I've managed to find... way too many points in the hsqldb driver where they deviate from their own specs (reported and fixed in the source, still waiting for them to finally fricking publish an updated artifact), and worked around those imperfections by finding the working mapping logic myself.

// I want strings handled differently
@Component
public class CleanStringArgumentFactory extends AbstractArgumentFactory<String> {
    protected CleanStringArgumentFactory() {
        super(Types.VARCHAR);
    }

    @Override
    protected Argument build(String value, ConfigRegistry config) {
        return (position, statement, ctx) -> statement.setString(position, StringUtils.trimToNull(value));
    }
}

// no need for jdbi's built-in logic
@Component
public class SetObjectArgumentFactory implements ArgumentFactory {
    private static final Set<Type> SUPPORTED = ImmutableSet.of(UUID.class, LocalDateTime.class);

    @Override
    public Optional<Argument> build(Type type, Object value, ConfigRegistry config) {
        return SUPPORTED.contains(type)
            ? Optional.of(new LoggableArgument(value, (position, statement, context) -> statement.setObject(position, value)))
            : Optional.empty();
    }
}

// these built-ins don't work and I don't need them anyway
@Component
public class UnsupportedArgumentFactory implements ArgumentFactory {
    private static final Set<Type> UNSUPPORTED = ImmutableSet.of(ZonedDateTime.class, OffsetDateTime.class, OffsetTime.class);

    @Override
    public Optional<Argument> build(Type type, Object value, ConfigRegistry config) {
        if (UNSUPPORTED.contains(type)) {
            throw new UnsupportedOperationException(type.getTypeName() + " is not supported at the moment");
        } else {
            return Optional.empty();
        }
    }
}

// voodoo
@Inject
    public JdbiConfiguration(
        @Named("dataSource") DataSource dataSource,
        Set<ArgumentFactory> arguments,
        Set<ColumnMapper> mappers,
        Set<ColumnMapperFactory> mapperFactories
    ) {
        jdbi = Jdbi.create(dataSource);
        jdbi.clearMappings();

        jdbi.installPlugin(new SqlObjectPlugin());

        arguments.forEach(jdbi::registerArgument);
        mapperFactories.forEach(jdbi::registerColumnMapper);
        mappers.forEach(jdbi::registerColumnMapper);

        // ...

        // Jdbi instance fine-tuned for my exact database and demands, without surprises and at my own responsibility

Howbouda? You could for example provide a SetObjectArgumentFactory that needs to be initialized with a set of classes it should activate on, and all sorts of variations of mapping logic for lots of datatypes, with some configurable behavior where applicable. Everybody happy. It takes some work from the end user, but I feel good knowing jdbi does exactly what I want it to do and what my database supports doing.

I get the feeling it would be too cumbersome to provide plugins with mappers for every database and even its different versions that have different support for datatypes.

I'd love to have plugins for lots of database vendors. Come one, come all: if your database needs/wants some customizations to adapt Jdbi to the idiosyncrasies of the database vendor or its JDBC driver, we want to hear about it. Or better yet, we want pull requests! :)

Bonus points if your database is supported directly by TravisCI so we can actually run tests against it: https://docs.travis-ci.com/user/database-setup/

As to your "blank slate" suggestion, I have only minor objections:

• I would probably put the clearMappings() method directly on RowMappers, ColumnMappers, Arguments, and JdbiCollectors config classes instead of on Jdbi, and I would name them clear().
• I'm a bit hesitant to make the built-in mappers, arguments, and collectors public. Some of them have wonky names or designs that we get away with because they're internal. I would probably want to refactor them some before making them part of the API.
• I'm questioning whether the blank slate is really necessary. All of Jdbi's registry classes use the last-registered-wins approach, so it is always possible to override the out-of-the-box behavior. In that sense the blank slate feels a little overkill.

@jdbi/contributors anybody else care to weigh in?

I included the blank slate idea to avoid accidentally using a default mapper in case you forgot to register one of your choice. I'd rather have an NotImplementedException than a default impl. Last-wins is nice and all, but there's no way to know which ones you need to overrule in case you want none.

It's not like it hurts to have. If people don't want that approach, they don't need to call the clear methods and nothing changes for them. It's just for those who prefer a no-defaultsies way of working.

I agree that we should not just make existing mappers public. If we choose to provide a clean slate option, we should bundle the mappers and binders into reasonable Plugin chunks, PrimitivesPlugin, BoxedPrimitivesPlugin, CollectionsPlugin, whatever.

Then, we can add a new factory for the "base" jdbi with no configuration at all, and modify the existing factories to use this and add the WholeJdbiEnchiladaPlugin.

I'm somewhat against 'clear` unless we can show cases where it's difficult to build it additively -- feels like a code smell to me.

I'm somewhat against 'clear` unless we can show cases where it's difficult to build it additively -- feels like a code smell to me.

Well if you add a separate no-preconfig jdbi factory, then it won't be difficult to build it additively so we won't need any clear() indeed.

Right, I think we're in agreement here, I was just expressing a preference adding a new "clean" factory method over a clear() style.

So, does this solve everyone's issues then? Personally I think the plan sounds pretty sexy.

• make Jdbi no-defaults (throws exceptions on pretty much everything)
• make all the existing mappers and binders (and tx handlers etc) pluggable (PrimitivesPlugin, BoxedPrimitivesPlugin, etc), as plugin bundles and as individual classes
• provide some alternative implementations (UTCJavaTimePlugin vs LocalJavaTimePlugin vs ConvertJavaTimeToUtilDatePlugin, GetObjectMapper.forClasses(Class... cs), SetObjectArgumentFactory.forClasses(Class... cs), NullsToDefaultsPrimitivesPlugin vs NullThrowsExceptionPrimitivesPlugin) and expose them the same way (in plugins and individually), so people can compose their own total desired mapping logic, taking only exactly what they want. Potentially a lot of mapper configuration necessary here, like different strategies for handling illegal nulls or missing timezones, and other possible disparities
• maybe make db-specific bundle plugins, so users with popular databases+versions can start from a no-config Jdbi and one-line install known good mapping schemes for their db (HsqlDb2_4_0Plugin), again with configuration options for different strategies
• and make it all backward compatible by just making the current factory methods perform the preconfiguration we have right now (BuiltInGenericPreconfigurationPlugin), alongside the no-config factory methods.

make Jdbi no-defaults

I don't have very strong opinion, but this all sounds very complex from user perspective. Users will need to understand various mappings possible for the types they want to use and choose an appropriate one. This is hard, especially when temporals are in play -- it really isn't trivial to understand different options possible.

Users will need to understand various mappings possible for the types they want to use and choose an appropriate one

No, only if they want to. The existing factory methods will keep outputting Jdbi instances with the exact same preinstalled mappers that they do now. We'll just additionally get DIY config factory methods, and people who opt for those will need to figure out the intimacies of mapping. There'll be a midway too, where you get a DIY instance and just install the YourDbAndVersionPlugin and roll with whatever it does. My lengthy explanation was mostly jdbi internals and advanced usage, not the baseline.

TLDR: You'll still be able to do Jdbi.create(dataSource) and you won't need to give any additional hoots about it that you don't already give. Someone else will just choose to Jdbi.createBlank(dataSource).install(x).install(y).install(z)...

I'm on board, except I don't think we should deprecate the existing factories, I believe most users will continue to use it as the "out of the box" behavior is pretty good.

Fair enough

@TheRealMarnes This whole conversation started out dealing with JSR-310 data types that aren't supported directly by JDBC.

I think there's consensus about removing the errant java.time mappers and arguments, and putting them into a plugin instead of built in.

Do you still want to pursue that?

Just the java.time ones? A JavaTimePlugin?

Yes, a plugin just for java.time types, but named to communicate that the arguments and mappers are naive, lossy implementations as @findepi described.

SimplisticJavaTimePlugin?

Waiting with this until my other PRs are taken care of... It's a bit too much at the same time.