Azure-sdk-for-java: [BUG] azure-cosmos 3.4.0, 3.5.0, 3.6.0 QueryPlan doesn't support DISTINCT

Created on 19 Nov 2019 · 19Comments · Source: Azure/azure-sdk-for-java

Describe the bug
We're attempting to upgrade to use version 3.4.0 via the Spring Cosmos 2.2.1.M1 version and ran into what seems to be a bug. Specifically we can no longer execute "distinct" queries.

Exception or Stack Trace

We receive the following from the query plan retriever.

com.azure.data.cosmos.CosmosClientException: Gateway Failed to Retrieve Query Plan: Query contains 1 or more unsupported features. Upgrade your SDK to a version that does support the requested features:
Query contained Distinct, which the calling client does not support.
ActivityId: f569ddf6-e3e0-445a-9546-6931f17a4033, Microsoft.Azure.Documents.Common/2.7.0, StatusCode: BadRequest

It seems the client is not including the "distinct" query feature in the query plan header. I'm not sure if this intentional. If so, how do we execute "distinct" queries?

https://github.com/Azure/azure-sdk-for-java/blob/c4360f67b4445d2da3f79e48eb641a70ea9864a5/sdk/cosmos/microsoft-azure-cosmos/src/main/java/com/azure/data/cosmos/internal/query/QueryPlanRetriever.java#L23

Code Snippet

Here is the code that used to work but is failing with 3.4.0.

val query = SqlQuerySpec(
    "SELECT DISTINCT r.deviceId FROM root r WHERE $expression",
    parameterCollection
)
val options = FeedOptions().apply { enableCrossPartitionQuery(true) }
return cosmosClient
    .getDatabase(cosmosDBConfig.database)
    .getContainer(DocumentCollectionNames.sensor)
    .queryItems(query, options)
    .collectResults()
    .map {
        objectMapper.readValue<JsonNode>(it.toJson()).get("deviceId").toString()
    }

Expected behavior

3.4.0 Should continue to support "Distinct" queries just like 3.3.x

Setup (please complete the following information):

OS: macOS
IDE : IntelliJ
Version of the Library used : 3.4.0

Cosmos Service Attention v3-item v4-item customer-reported

Source

allenhumphreys

👍1

Most helpful comment

Any news about this issue? Also the version 3.7.0 is affected by this bug

apescione on 24 Mar 2020

👍6

All 19 comments

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @kushagarathapar @srinathnarayanan @southpolesteve @shurd

msftbot[bot] on 20 Nov 2019

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @kushagarathapar @srinathnarayanan @southpolesteve @shurd

msftbot[bot] on 20 Nov 2019

@kushagraThapar

@maggiepint FYI Kushagra's GitHub handle with msftbot needs updating

southpolesteve on 20 Nov 2019

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @kushagraThapar @srinathnarayanan @southpolesteve @shurd

msftbot[bot] on 21 Nov 2019

Looks like the GitHub handle is fixed!

Sorry to test our bots on your issue @allenhumphreys .

maggiepint on 21 Nov 2019

Its a known gap. Its on our backlog to address it.

kirankumarkolli on 21 Nov 2019

@kirankumarkolli Is there anyway to perform distinct queries with the 3.4.0 version of the library? I'm not seeing anyway to opt out of query plans

allenhumphreys on 21 Nov 2019

@allenhumphreys DISTINCT support demands some SDK side implementation, which is a known gap and in our backlog for JAN.

kirankumarkolli on 21 Nov 2019

@kirankumarkolli Thank you for confirming. I would like to provide a bit of feedback on your versioning. When backwards compatibility isn't guaranteed, a new major version number should be employed per semantic versioning to convey that the version no longer works. This investigation has used a lot of our time trying to figure out how to use the latest version of the library.

Thank you.

allenhumphreys on 21 Nov 2019

Thanks for working with Microsoft on GitHub! Tell us how you feel about your experience using the reactions on this comment.

msftbot[bot] on 21 Nov 2019

😕1 👎1

@allenhumphreys thanks a-lot for the feedback. Apologies for the confusion.

/cc: @kushagraThapar

kirankumarkolli on 21 Nov 2019

I'm reopening this as January is coming to an end and I haven't seen any thing regarding DISTINCT support being re-added. 2 versions have come out since this was originally opened. @kirankumarkolli

allenhumphreys on 23 Jan 2020

👍1

I also need this. I don't understand why the official CosmosDB Java library doesn't support DISTINCT when it's clearly part of the CosmosDB SQL dialect.

timwhit on 23 Jan 2020

Any news about this issue? Also the version 3.7.0 is affected by this bug

apescione on 24 Mar 2020

👍6

DISTINCT support is being added to V4 at the moment, and its actively in development.

mbhaskar on 24 Mar 2020

I don't understand how this was available in previous versions, it was _broken_ by accident and it's taking this long to get it added back in?

timwhit on 24 Mar 2020

Earlier it was only partial availability in java sdks, it might probably work only queries scoped to single partition that could complete in single continuation. The current implementation would make it available across partitions.

mbhaskar on 25 Mar 2020

Why will be available in V4 and not in an hotfix 3.x.y?

baldassarreC on 25 Mar 2020

@allenhumphreys - since DISTINCT is now supported in V4 and I think you guys have already updated the SDK to v4, can we close this issue ?

@baldassarreC - V3 SDK is no longer in development actively. New features are being currently built on V4 SDK as it is much better in performance and stability as compared to V3 SDK. We encourage all customers to upgrade to V4 SDK as soon as they can.

Meanwhile, you can read more about V4 SDK here: https://devblogs.microsoft.com/cosmosdb/java-sdk-v4-ga/