Azure-sdk-for-java: [Bug] infinite iteration over blobList

Hi, @asm0dey. This does sound like rather unexpected behavior. I can assure you that the listing results are not intended to loop this way! Are you at all able to capture any network traces so we can see if the PagedIterable is issuing repeated requests or simply emitting duplicate cached values? And do you only have those three blobs in your container? Or are there more expected results you aren't seeing?

P.S. Sounds like a neat plugin :)

I absolutely can, but I don't know why request and response bodies aren't being logged.

But once again — I can't reproduce this behavior in clean environment so I don't thnk this is real bug, just something with classpath/classloader/whatever.

P.S. And yes, you'll definitely like this plugin, it's https://plugins.jetbrains.com/plugin/12494-big-data-tools
Hope to see Azure implemented in next EAP release after 6th

That'd be great. Thanks! I don't think request and response bodies are typically logged because they can by quite large.

If you suspect some sort of classpath error, we could start with mvn dependency:tree and see what's going on? There may be some conflicts causing something strange.

Wow that'd be great! Thanks for working on that project, and if we can be supportive in any way, please let us know. We love to see other folks in the community building tools around Storage and Azure in general!

It's not that easy because we don't have traditional build system like maven or gradle, we'e building our project with IDEA itself (and on CI server too).

But please, find UML diagram of dependencies here.

I really can't find anything conflicting here, but I'm not that experienced in reactor and can't even imagine what could make it work the wrong way.

Talking about logging: there is option to log headers and bodies, but body can obly be logged if size of body is known from headers, but according to logs there is no header.

Here is small snip of log:

2020-03-25 09:43:08,627 [57159248]   INFO - rvice.listBlobHierarchySegment - --> GET https://pavelfinkelshteynjb.blob.core.windows.net/testcontainer1?prefix=&delimiter=/&marker=REDACTED&restype=REDACTED&comp=REDACTED
Date:Wed, 25 Mar 2020 06:43:08 GMT
Authorization:REDACTED
Content-Length:0
x-ms-version:REDACTED
x-ms-client-request-id:c4016194-f020-45fc-b254-11a04fd65d52
User-Agent:azsdk-java-azure-storage-blob/12.5.0 (11.0.6; Linux 5.5.8-1-MANJARO)
(empty body)
--> END GET

2020-03-25 09:43:08,639 [57159260]   INFO - rvice.listBlobHierarchySegment - <-- 200 https://pavelfinkelshteynjb.blob.core.windows.net/testcontainer1?prefix=&delimiter=/&marker=REDACTED&restype=REDACTED&comp=REDACTED (68 ms, unknown-length body)
Date:Wed, 25 Mar 2020 06:43:07 GMT
Server:Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0
Transfer-Encoding:chunked
x-ms-version:REDACTED
Content-Type:application/xml
x-ms-request-id:REDACTED
x-ms-client-request-id:7836d3ae-cf10-4432-a511-479dbf4ec09d
(body content not logged)
<-- END HTTP 

In class com.azure.core.http.policy.HttpLoggingPolicy we can find method logResponse, and there

                String contentTypeHeader = response.getHeaderValue("Content-Type");
                long contentLength = this.getContentLength(logger, response.getHeaders());
                if (this.shouldBodyBeLogged(contentTypeHeader, contentLength)) {
// snip

                } else {
                    responseLogMessage.append("(body content not logged)").append(System.lineSeparator()).append("<-- END HTTP");
                    return this.logAndReturn(logger, responseLogMessage, response);

and in method getContentLength

    private long getContentLength(ClientLogger logger, HttpHeaders headers) {
        long contentLength = 0L;
        String contentLengthString = headers.getValue("Content-Length");
        if (CoreUtils.isNullOrEmpty(contentLengthString)) {
            return contentLength;
// snip

So this API doesn't support logging of bodies.

@asm0dey Sorry for the delay. Can you update your HttpLogOptions on your client builders to add "marker" as an allowedQueryParameter? It looks like it's being redacted from your logs, and I suspect that perhaps something is triggering a list call with the same marker over and over again. We coincidentally saw some similar behavior in some development we're doing over here, and I think it might be the same cause.

I'm not seeing anything in the UML diagram that stands out, but I've also never looked at one before.

Absolutely. Please find full log attached:
idea.log
Marker is empty all the way

grep marker= idea.log | more tells us that there is constantly repeating request GET https://pavelfinkelshteynjb.blob.core.windows.net/testcontainer1?prefix=&delimiter=/&marker=&restype=REDACTED&comp=REDACTED is going

Also we can see that there is some enormous number of requests: 673. It was like 2-4 requests from client side

@rickle-msft any ideas on this?

@rickle-msft just a friendly reminder

Hi, @asm0dey. I'm so sorry about the delay! My emails from GitHub were getting filtered incorrectly for a little while, and when I noticed and fixed it, I must have also missed picking up this issue. Apologies again and thank you for the reminder.

Yea it's interesting that all the requests have that empty marker= query parameter. The only way I can reproduce that specific behavior is by explicitly passing an empty string as the continuation token, and even that doesn't loop infinitely.

@anuchandy Do you have any sense of if, as suggested, some sort of dependency problem could be causing this? It looks like he's using a shaded 12.5.0 jar, and I'm not too familiar with jar shading and how that might affect interactions with the classpath and classloader.

Actually, I've even tried to unshade it and it changes literally nothing
in this behavior. Also I'm not changing any package names or smth like
that.

I've even reached one of reactor project developers, but his only idea
was that iterator is being recreated every call.
On 20/04/29 10:53AM, Rick Ley wrote:

Hi, @asm0dey. I'm so sorry about the delay! My emails from GitHub were getting filtered incorrectly for a little while, and when I noticed and fixed it, I must have also missed picking up this issue. Apologies again and thank you for the reminder.

Yea it's interesting that all the requests have that empty marker= query parameter. The only way I can reproduce that specific behavior is by explicitly passing an empty string as the continuation token, and even that doesn't loop infinitely.

@anuchandy Do you have any sense of if, as suggested, some sort of dependency problem could be causing this? It looks like he's using a shaded 12.5.0 jar, and I'm not too familiar with jar shading and how that might affect interactions with the classpath and classloader.

--
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub:
https://github.com/Azure/azure-sdk-for-java/issues/9465#issuecomment-621366540

--
Regards,
Pasha

Big Data Tools @ JetBrains

@asm0dey

could you please run the following code in your setup and share what you see as output. Yes, it's going to run infinitely, just break after you see Processing Page:start 3-4 times.

BlobContainerAsyncClient blobContainerClient = new BlobContainerClientBuilder()
        .endpoint("<your-storage-account-url>")
        .sasToken("<your-sasToken>")
        .containerName("mycontainer")
        .buildAsyncClient();

PagedFlux<BlobItem> flux = blobContainerClient.listBlobsByHierarchy(path);
flux.byPage()
        .doOnNext(new Consumer<PagedResponse<BlobItem>>() {
            @Override
            public void accept(PagedResponse<BlobItem> response) {
                System.out.println("Processing Page:start");
                List<BlobItem> items = response.getValue();
                for (BlobItem item : items) {
                    System.out.println(item.getName());
                }
                if (response.getContinuationToken() == null) {
                    System.out.println("ContinuationToken: null");
                } else if (response.getContinuationToken().length() == 0) {
                    System.out.println("ContinuationToken: empty");
                } else {
                    System.out.println("ContinuationToken: non-empty:" + response.getContinuationToken());
                }
                System.out.println("Processing Page:end");
            }
        }).blockLast();

btw, please redact the item name if you wish, I'm interested in the null | empty | non-empty attributes of the continuationToken.

Sorry for delay, I believe it's difference it TZs.
Here it is

Processing Page:start
.travis.yml
build.sh
.git/
KateTest12/
KateTest1234/
KateTest12345/
KateTestRenamed/
Test1/
demo/
plugins/
ContinuationToken: empty
Processing Page:end
Processing Page:start
.travis.yml
build.sh
.git/
KateTest12/
KateTest1234/
KateTest12345/
KateTestRenamed/
Test1/
demo/
plugins/
ContinuationToken: empty
Processing Page:end
Processing Page:start
.travis.yml
build.sh
.git/
KateTest12/
KateTest1234/
KateTest12345/
KateTestRenamed/
Test1/
demo/
plugins/
ContinuationToken: empty
Processing Page:end

I'm writing in kotlin, so my code looks slightly different:

    BlobServiceClientBuilder()
      .endpoint(endpoint)
      .apply { authProvider(this) }
      .httpClient(NettyAsyncHttpClientBuilder().build())
      .buildAsyncClient()
      .getBlobContainerAsyncClient(container)
      .listBlobsByHierarchy("")
      .byPage()
      .doOnNext { response ->
        println("Processing Page:start")
        val items = response.value
        for (item in items) {
          println(item.name)
        }
        when {
          response.continuationToken == null -> println("ContinuationToken: null")
          response.continuationToken.isEmpty() -> println("ContinuationToken: empty")
          else -> println("ContinuationToken: non-empty:" + response.continuationToken)
        }
        println("Processing Page:end")
      }
      .blockLast()

And also I've already complained that http client isn't being autodetected because of classloader issues, so I have to create it manually.

I hope it changes nothing.

@asm0dey thank you for sharing this!

@rickle-msft - from the output, it looks like the marker (aka continuationToken) for the last page is empty "" and not null. The exit criteria for the "page fetching loop" is null continuation token.
I think that explains why this is getting into the infinite loop.

If I run the above code locally, the last page always has the marker property value as null so not reproducible in my storage account. The output looks like below,

Processing Page:start
foo.jpeg
ContinuationToken: null. <------  [In Paul's case this is an empty string ""]
Processing Page:end

if we manually set (in debug mode) that to empty string "" then I can see that it getting into an infinite loop.

Any chance storage service over the wire returns marker property as an empty string, indicating that it is the last page?

@anuchandy I think the service always returns the empty xml element <NextMarker /> when there isn't a next page. The rest docs state "Note that the Prefix, Marker, MaxResults, and Delimiter elements are present only if they were specified on the request URI. The NextMarker element has a value only if the list results are not complete." I believe that implies NexMarker will always be present but may not always have a value.

Is there any chance the different environment is somehow causing the deserializer to give empty string instead of null? We could also make the check for null or empty instead of just null?

Any ideas on minimal test to reproduce substitution of "" with null in our environment?

@rickle-msft thanks for the reference. Let me reach out to few people and get back, but I can see that the previous runtime indeed treat the empty string as an exit condition ref.

while (nextPageLink != null && nextPageLink != "") {

not completely sure but this is probably an indication that when next token is of type String then some azure services might be using empty string to indicate completion.

I'm seeing the same problem with the Iterable returned by DataLakeServiceClient.listFileSystems method. When I used an asynch client and printed markers, as in the suggestion above, I see empty strings.

@rodburgett I suppose it could help if you post information on your environment. Possibly we could find common things in our environments.

@asm0dey I think the only common element in our environments is the blob storage jar, and even there we are using different versions. I'm working on Window 10, using Gradle for execution. I first ran into this problem with version 12.3.0 of the azure-storage-blob jar. I recently upgraded to 12.6.0 and continue to see the problem.

@asm0dey, @rodburgett - within the team we discussed treating "" as well the signal to end the paging, given it is the case with many azure-services. Opened a draft for it. (this issue results in identifying another missing functionality in the core level)

While the infinite-loop problem you're running into will disappears as a side effect of this change, it is interesting to understand why your environment treats empty XML node as "" instead of null.

can you share the version of Jackson that gets loaded in the application class-path?

I'm using Jackson jars v2.10.2.

@anuchandy here is my full classpath if it'll help:

accessors-smart-1.2.jar
aircompressor-0.15.jar
annotations-17.0.0.jar
asm-5.0.4.jar
avro-1.8.0.jar
avro-1.8.2.jar
aws-java-sdk-bundle-1.11.375.jar
azure-core-1.4.0.jar
azure-core-http-netty-1.5.0.jar
azure-storage-blob-12.6.0.jar
azure-storage-common-12.6.0.jar
bigdatatools.jar
commons-beanutils-1.9.3.jar
commons-beanutils-1.9.4.jar
commons-cli-1.2.jar
commons-codec-1.11.jar
commons-codec-1.5.jar
commons-collections-3.2.2.jar
commons-collections4-4.4.jar
commons-compress-1.4.1.jar
commons-compress-1.8.1.jar
commons-configuration2-2.1.1.jar
commons-io-2.5.jar
commons-lang-2.6.jar
commons-lang3-3.7.jar
commons-lang3-3.9.jar
commons-logging-1.1.3.jar
commons-logging-1.2.jar
commons-math3-3.1.1.jar
commons-net-3.6.jar
commons-pool-1.5.4.jar
commons-text-1.4.jar
commons-text-1.8.jar
curator-client-2.12.0.jar
curator-framework-2.12.0.jar
curator-recipes-2.12.0.jar
diff-match-patch-1.2.jar
dnsjava-2.1.7.jar
fastutil-6.5.7.jar
google-cloud-storage-1.105.2.jar
guava-11.0.2.jar
hadoop-annotations-3.2.0.jar
hadoop-auth-3.2.0.jar
hadoop-aws-3.2.0.jar
hadoop-client-3.2.0.jar
hadoop-common-3.2.0.jar
hadoop-distcp-3.2.1.jar
hadoop-hdfs-client-3.2.0.jar
hadoop-mapreduce-client-common-3.2.0.jar
hadoop-mapreduce-client-core-3.2.0.jar
hadoop-mapreduce-client-jobclient-3.2.0.jar
hadoop-yarn-api-3.2.0.jar
hadoop-yarn-client-3.2.0.jar
hadoop-yarn-common-3.2.0.jar
hive-storage-api-2.6.0.jar
htrace-core4-4.1.0-incubating.jar
jackson-annotations-2.10.1.jar
jackson-core-2.10.1.jar
jackson-core-asl-1.9.11.jar
jackson-core-asl-1.9.13.jar
jackson-databind-2.10.1.jar
jackson-dataformat-xml-2.10.1.jar
jackson-datatype-jsr310-2.10.1.jar
jackson-mapper-asl-1.9.11.jar
jackson-mapper-asl-1.9.13.jar
jackson-module-jaxb-annotations-2.10.1.jar
jakarta.activation-api-1.2.1.jar
jakarta.xml.bind-api-2.3.2.jar
javax.servlet-api-3.1.0.jar
jaxb-api-2.2.11.jar
jcip-annotations-1.0-1.jar
jersey-client-1.19.jar
jersey-core-1.19.jar
jersey-servlet-1.19.jar
json-smart-2.3.jar
jsoup-1.12.1.jar
jsp-api-2.1.jar
jsr305-3.0.0.jar
jsr311-api-1.1.1.jar
kerb-admin-1.0.1.jar
kerb-client-1.0.1.jar
kerb-common-1.0.1.jar
kerb-core-1.0.1.jar
kerb-crypto-1.0.1.jar
kerb-identity-1.0.1.jar
kerb-server-1.0.1.jar
kerb-simplekdc-1.0.1.jar
kerb-util-1.0.1.jar
kerby-asn1-1.0.1.jar
kerby-config-1.0.1.jar
kerby-pkix-1.0.1.jar
kerby-util-1.0.1.jar
kerby-xdr-1.0.1.jar
log4j-1.2.17.jar
moshi-1.7.0.jar
moshi-adapters-1.7.0.jar
moshi-kotlin-1.7.0.jar
netty-buffer-4.1.45.Final.jar
netty-codec-4.1.45.Final.jar
netty-codec-http-4.1.45.Final.jar
netty-codec-http2-4.1.45.Final.jar
netty-codec-socks-4.1.45.Final.jar
netty-common-4.1.45.Final.jar
netty-handler-4.1.45.Final.jar
netty-handler-proxy-4.1.45.Final.jar
netty-resolver-4.1.45.Final.jar
netty-tcnative-boringssl-static-2.0.27.Final.jar
netty-transport-4.1.45.Final.jar
netty-transport-native-epoll-4.1.45.Final-linux-x86_64.jar
netty-transport-native-unix-common-4.1.45.Final.jar
nimbus-jose-jwt-4.41.1.jar
nv-websocket-client-2.9.jar
okhttp-2.7.5.jar
okhttp-3.14.7.jar
okio-1.17.2.jar
opencsv-5.1.jar
orc-core-1.6.2.jar
orc-shims-1.6.2.jar
paranamer-2.7.jar
parquet-avro-1.9.0.jar
parquet-column-1.9.0.jar
parquet-common-1.9.0.jar
parquet-encoding-1.9.0.jar
parquet-format-2.3.1.jar
parquet-hadoop-1.9.0.jar
parquet-jackson-1.9.0.jar
protobuf-java-2.5.0.jar
re2j-1.1.jar
reactive-streams-1.0.3.jar
reactor-core-3.3.3.RELEASE.jar
reactor-netty-0.9.5.RELEASE.jar
snappy-java-1.1.1.3.jar
snappy-java-1.1.1.6.jar
snappy-java-1.1.7.3.jar
stax2-api-3.1.4.jar
stax2-api-4.2.jar
token-provider-1.0.1.jar
woodstox-core-5.0.3.jar
woodstox-core-6.0.2.jar
xz-1.0.jar
xz-1.5.jar

I can see number of duplicate dependencies. Do you think it can break something?

export.zip

@rodburgett @anuchandy Here I attach minimal plugin that reproduces infinite iteration. It can be built by gradle and there is run configuration to run bundled idea

@asm0dey - I was able to build the project, how can I run it, I see below error indicating "no plugin specified", nothing list under "use classpath of the module" option, am I using the right config here, if not could you provide detailed steps?

The easiest way is to just call gradle's runIde task. If you'll run it from idea — wull.be able to debug it

@asm0dey thanks for that hint, I was able to load the plugin.

I spent some time debugging the plugin code, the issue is nothing specific to storage SDK.

Within the IntelliJ sandbox, Jackson is unable to locate the XML parser. Jackson uses ServiceLoader to load the XML parser factory. Outside the plugin environment, it is able to load WstxInputFactory [com.fasterxml.woodstox:woodstox-core] but not in the sandbox, hence Jackson switches to a default implementation Stax2ReaderAdapter which has no way to detect empty node. @asm0dey I guess you will need to see how to make plugin ClassLoader to aware of this dependency that is loaded on demand.

Here is the updated plugin project that has only Jackson dependency export-2.zip if that helps.

Btw in the above-updated plugin code, the following Gradle dependency doesn't have to be explicitly specified

group: 'com.fasterxml.woodstox', name: 'woodstox-core', version: '6.0.3'

this is a transitive dependency of "jackson-dataformat-xml" so it can be removed.

I came across this post if it helps: https://intellij-support.jetbrains.com/hc/en-us/community/posts/206761975-How-to-load-classes-in-a-plugin-with-ServiceLoader-

If we follow the instruction listed in the above IntelliJ plugin thread, then Jackson is able to use the ServiceLoader:

Thread.currentThread().setContextClassLoader(this.getClass().getClassLoader());

not sure this is the optimal solution for plugin though.

Wow! Very impressive research, thank you very much for this! And it actually works. I couldn't even think that it's about classloader too.

@anuchandy is there any hope that you'll be able to add an overloaded method which will accept another classloader as argument? This answer on SO gives insight on why this may be useful.

I have a simple Java/Gradle repro case, no IDE required. After adding that woodstox-core dependency to build.gradle the first 'ContinuationToken' returned during iteration is now null. And my iteration loop ends as expected.

Running outside of an IDE it looks like a dependency problem, not a class loader problem.

Thanks @anuchandy

@rodburgett happy to hear you're unblocked.

@asm0dey - we've some work-in-progress to enable users to configure JacksonAdapter. See these modules 1, 2. Hopefully, libraries allow the user to provide a configured adapter soon, we are piloting this with some selected SDKs (e.g., cosmos).

Given the plugin brings its conflicting dependencies, another option to explore is shading.

I feel like the plugin framework should provide a way to handle the standard java service-provider loading facility instead of developers working around it. If we look at the OSGi plugin/bundler world, it gives a service-loader mediator scheme to handle the same case, this enables loading to work on 3rd party libraries without app developers modifying it.

I spend some time to look into how IntelliJ plugin works underneath and opened an issue in IntelliJ plugin repro: https://intellij-support.jetbrains.com/hc/en-us/community/posts/360008239419-How-to-make-the-plugin-CCTL-aware-of-the-Service-Provider-type-that-java-utils-ServiceLoader-looks-for

@anuchandy I believe it will be too hard to introduce Classloader which will behave correctly in IDEA cause there are many plugins and many threads, but ContextClassLoader is being set up per thread.

And yes, both OSGi and JBoss Modules provide a way to work with service loaders correctly, but ways they do that are not too simple. Do we really want more complexity in our IDE? :)

The main reason I'm worried about setting CCL in the az libraries is, this may be a workaround for one case, but given its a library and will be used in many environments, setting CCL without understanding what goes underneath the env is risky.

Even if we want to take the route of passing classloaders 3rd party libs, not all 3rd party libs may be supporting it. Jackson XMLMapper is an example.

yea OSGi mediator is not simple, but I think the framework provided functionality is always safe and it ensure nothing breaks. So I thought it should be solved similarly in InteillJ plugin fx, hence the issue.

I guess once we have the option to configure the JackonAdpater (like I mentioned in the previous comment), users should be able to provide the factory, so there is no CCL workaround.

Or alternatively, shading is another possible option to consider, here is a related thread https://github.com/Azure/azure-sdk-for-java/issues/11104, but in this case, there was also a conflict issue.

I believe you can even create real issue at youtrack.jetbrains.com.

Shading can be nice alternative, but we should decide who should be responsible for shading — library or plugin. And anyways shading should be very refined, because for example shading of reactor is not recommended by reactor team.

Thank you for the correct link to report the issue, opened one here: https://youtrack.jetbrains.com/issue/IDEA-241229

@asm0dey @anuchandy What is the status of this issue? Do we need to keep it open for tracking or are we satisfied with opening the issue with jetbrains?

@rickle-msft thank you very much for asking and tracking! I'm absolutely satisfied with current state.

Sounds good. In that case I will close it and we can reopen it if we need further discussion here. Thanks for all your work!

I am facing same issue in my custom plugin, what is the workaround? @anuchandy

@tooptoop4 In my case it was classloader issue, did you check?

I am facing the same problem with listBlobs(). Is there a solution other than changing classloader?

    <dependency>
      <groupId>com.azure</groupId>
      <artifactId>azure-storage-file-datalake</artifactId>
      <version>12.1.1</version>
    </dependency>
    <dependency>
        <groupId>com.fasterxml.jackson.dataformat</groupId>
        <artifactId>jackson-dataformat-xml</artifactId>
        <exclusions>
            <exclusion>
                <groupId>*</groupId>
                <artifactId>*</artifactId>
            </exclusion>
        </exclusions>
    </dependency>
    <dependency>
        <groupId>com.fasterxml.woodstox</groupId>
        <artifactId>woodstox-core</artifactId>
        <version>6.2.1</version>
    </dependency>

@nkanala my workaround was to add each blob name to a collection then break the loop if it finds a blob already in the collection

@tooptoop4 @nkanala Thank you for your willingness to collaborate on this. One heads up I'll give you around that is if you're doing a hierarchical listing, maybe add an extra test case around the scenario where one blob name is a prefix for another blob name. e.g. case 1: foo and foo/bar. Case 2: foo/ and foo/bar. I believe in the former case the returned items will be foo and foo/ and in the latter case only foo/ will be returned, but I thought it better to mention in case your app has other logic to transform these results before comparing.

Just FYI, I experience this issue when upgrading from jackson-databind 2.11.4 (Dec 2020) to 2.12.0 (Nov 2020) c.f. https://mvnrepository.com/artifact/com.fasterxml.jackson.core/jackson-databind

So my solution in this case is to stay at 2.11.4. :)

I experienced the same issue, writing a plugin for graylog integration with Azure EventHubs. Adding Thread.currentThread().setContextClassLoader(this.getClass().getClassLoader()); like @anuchandy points out before creating the client solved the problem.

Here is the code instantiating the BlobContainerAsyncClient:

//Instantiating the BlobContainerAsyncClient
119               BlobContainerAsyncClient blobContainerAsyncClient =
120                   new BlobContainerClientBuilder()
121                       .connectionString(STORAGE_CONNECTION_STRING)
122                       .containerName(STORAGE_CONTAINER_NAME)
123                       .httpClient(httpClient)
124                       .httpLogOptions(new HttpLogOptions().setLogLevel(HttpLogDetailLevel.BODY_AND_HEADERS))
125                       .buildAsyncClient();
// Instantiating the event processor
129               BlobCheckpointStore blobCheckpointStore = new BlobCheckpointStore(blobContainerAsyncClient);
130               EventProcessorClientBuilder eventProcessorClientBuilder =
131                   new EventProcessorClientBuilder()
132                       .connectionString(EH_CONNECTION_STRING)
133                       .consumerGroup(EH_CONSUMER_GROUP)
134                       .processEvent(eventContext -> {
135                         LOG.info(
136                             "Processing event from partition {} with sequence {} %n",
137                             eventContext.getPartitionContext().getPartitionId(),
138                             eventContext.getEventData().getBodyAsString());
139                         if (eventContext.getEventData().getSequenceNumber() % 10 == 0) {
140                         }
141                       })
142                       .processError(errorContext -> {
143                         LOG.error(
144                             "Error occurred in partition processor for partition {}, {}.%n",
145                             errorContext.getPartitionContext().getPartitionId(), errorContext.getThrowable());
146                       })
147                       .checkpointStore(blobCheckpointStore);

The output would indefinitely be:

graylog_1 2021-01-22 23:25:11,690 INFO : com.azure.storage.blob.implementation.ContainersImpl$ContainersService.listBlobFlatSegment - --> GET https://shgraylog.blob.core.windows.net/shgraylog?prefix=REDACTED&marker=REDACTED&include=REDACTED&restype=REDACTED&comp=REDACTEDDate:Fri, 22 Jan 2021 23:25:11 GMTAuthorization:REDACTEDContent-Length:0x-ms-version:REDACTEDx-ms-client-request-id:e94e4822-d575-4172-9967-5f040598af7cUser-Agent:azsdk-java-azure-storage-blob/12.9.0 (1.8.0_265; Linux; 4.19.121-linuxkit)(empty body)--> END GET
graylog_1 2021-01-22 23:25:11,692 INFO : com.azure.storage.blob.implementation.ContainersImpl$ContainersService.listBlobFlatSegment - --> GET https://shgraylog.blob.core.windows.net/shgraylog?prefix=REDACTED&marker=REDACTED&include=REDACTED&restype=REDACTED&comp=REDACTEDDate:Fri, 22 Jan 2021 23:25:11 GMTAuthorization:REDACTEDContent-Length:0x-ms-version:REDACTEDx-ms-client-request-id:3309ac6e-33ef-46f5-b64b-60b1b0a98cd8User-Agent:azsdk-java-azure-storage-blob/12.9.0 (1.8.0_265; Linux; 4.19.121-linuxkit)(empty body)--> END GET
graylog_1 2021-01-22 23:25:11,700 INFO : com.azure.storage.blob.implementation.ContainersImpl$ContainersService.listBlobFlatSegment - <-- 200 https://shgraylog.blob.core.windows.net/shgraylog?prefix=REDACTED&marker=REDACTED&include=REDACTED&restype=REDACTED&comp=REDACTED (19 ms, unknown-length body)Date:Fri, 22 Jan 2021 23:25:11 GMTServer:Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0Transfer-Encoding:chunkedx-ms-version:REDACTEDContent-Type:application/xmlx-ms-request-id:REDACTEDx-ms-client-request-id:fa182abe-03bc-48cd-b981-de7246bb1ee9(body content not logged)<-- END HTTP
graylog_1 2021-01-22 23:25:11,703 INFO : com.azure.storage.blob.implementation.ContainersImpl$ContainersService.listBlobFlatSegment - --> GET https://shgraylog.blob.core.windows.net/shgraylog?prefix=REDACTED&marker=REDACTED&include=REDACTED&restype=REDACTED&comp=REDACTEDDate:Fri, 22 Jan 2021 23:25:11 GMTAuthorization:REDACTEDContent-Length:0x-ms-version:REDACTEDx-ms-client-request-id:88b6b377-092a-4eee-9fdd-6d00a4c07e22User-Agent:azsdk-java-azure-storage-blob/12.9.0 (1.8.0_265; Linux; 4.19.121-linuxkit)(empty body)--> END GET
graylog_1 2021-01-22 23:25:11,713 INFO : com.azure.storage.blob.implementation.ContainersImpl$ContainersService.listBlobFlatSegment - <-- 200 https://shgraylog.blob.core.windows.net/shgraylog?prefix=REDACTED&marker=REDACTED&include=REDACTED&restype=REDACTED&comp=REDACTED (20 ms, unknown-length body)Date:Fri, 22 Jan 2021 23:25:11 GMTServer:Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0Transfer-Encoding:chunkedx-ms-version:REDACTEDContent-Type:application/xmlx-ms-request-id:REDACTEDx-ms-client-request-id:3309ac6e-33ef-46f5-b64b-60b1b0a98cd8(body content not logged)<-- END HTTP

Sorry for the verbose comment, hope it helps someone who faces the same issue.

Just FYI, I experience this issue when upgrading from jackson-databind 2.11.4 (Dec 2020) to 2.12.0 (Nov 2020) c.f.

I've experienced this same issue upgrading to 2.12.0, the problem seems to be that the default behaviour in that version has changed (https://github.com/FasterXML/jackson-dataformat-xml/issues/411) and for an empty block, it now returns an empty string instead of null. I don't think there's a way to inject a custom SerializerAdapter. The fix should be fairly trivial to implement in JacksonAdapter#JacksonAdapter and configure the parser to behave as expected.