Azure-sdk-for-java: JSR203 NIO FileSystemProvider?

Hey, Ryan!

We aren't adhering to NIO2. Instead we're following Reactive Streams through RxJava. We actually just released a preview of the new version of the SDK which offers this. You can find it here: https://github.com/Azure/azure-storage-java/tree/New-Storage-SDK-V10-Preview

Good to know, thanks… that's not exclusive with supporting an NIO interface though, right? In fact, presumably a thin NIO facade over what's already there would be straightforward.

Can reinterpret this issue as a request for that 😄

Possibly not. I think this will have to be a discussion for our team sometime after GA of the new version. I'll add an item to the backlog to ensure a discussion on the best way to support this scenario.

+1 for having JSR203 NIO FileSystemProvider for Azure storage. That would allow to plug Azure cloud storage in Java applications as a plain file system.

A JSR203 NIO FileSystemProvider will be extremely useful, and would allow to use the Azure storage alongside local storage in a transparent way. We rely on NIO interfaces also on other cloud storage backends and the advantages in terms of usability are great.

This would be useful for genomics scenarios as well, where some JVM-based workflow engines abstract cloud vs local file access using NIO.

This would be useful for genomics scenarios as well, where some JVM-based workflow engines abstract cloud vs local file access using NIO.

I think all people in this thread are exactly referring to this scenario.

Hi, guys.

Quick update. Since this issue has continuously been referenced by customers, I want to work on at least prioritizing the discussion surrounding this to work towards a more conclusive answer on our path forward.

We GA'd the library a couple months ago, which was the first benchmark we wanted to hit before looking closer at this. There are a few features we didn't get into GA that we still feel are rather critical that we want to be sure to complete soon. Many of these have been completed by now, but those (and a few bug reports we've prioritized) have been the bulk of development since August. The most notable remaining critical item is uploadFromNonReplayableFlowable, which I should get to start working on next week or possibly the week after. Once that is finished (ideally speaking, that would be maybe a week or two of work), there would be potentially more room for discussion, depending on when the service ends up coming out with new features and how critical those are to support. So that's sort of the landscape of dev time.

I noticed @sodic mentioned that he was taking a look at implementing this himself. Would he or anyone else here be willing to discuss the possibility of a pull request? We should discuss design and management and whether this repo is the correct place for such a feature before committing to anything, but if you're willing to contribute, we might be able to work something out.

Would anyone interested in this feature be willing to hop on a call to discuss the details of the scenario and the work involved in supporting this story? I'm not super familiar with JSR203 and it would certainly help jumpstart the conversation if I could get a primer.

In the meanitime, is it strongly required that JSR203 be used in these scenarios? Just brainstorming here, but Azure Storage does have a few other options that work with filesystems in this sort of transparent manner. Fuse and SMB are solutions for mounting a Linux or Windows filesystem respectively. Without having more details, it sounds like those might be possible options. They don't interact with the Java client in any special way, but they both transparently back a file system on azure storage, so any file you write or operate on locally is reflected as a blob in Storage.

Thanks, guys, for your continued interest.

Hi Rick, thanks for your detailed reply. I'll provide my feedback.

Regarding the possibility to use alternative adapters, such Fuse, that's something I've investigated in the past, but none was satisfying. For example Fuse requires the installation of separate OS libraries, which need root permissions, and the compilation of the fuse client. Operations that are overkill for average users. Also the client requires separate configuration and does not integrated smoothly with the target application.

I would surely be interested in this effort, I've implemented a couple of JSR203 file system drivers. It's not a particular friendly API but neither so complicated. The most challenging part is to have a detailed knowledge of the target API and storage system ie. Azure blob storage, that I guess you have.

If you are interested to collaborate on this I would happy to provide some advice and to contribute my implementation for Google storage. It can be used as a reference implementation for the Azure version (it's groovy code, but converting to Java it would be trivial). Let me know what do you think about that.

@pditommaso. Thanks for the feedback! I've blocked off some time next sprint (starting next week) to do some more investigation and possibly design. Since you're interested in this effort, is there a channel where we could set up a call or an IM chat to discuss some details? Perhaps mid-week would be best so I can do some reading beforehand.

Nice. I've created this gitter channel. We may organise also a call something around Wed or Thu.

@rickle-msft was there any outcomes from the discussion?

Yes, but nothing final. This feature is seen by our management as sort of a convenience thing, so it's been prioritized lower than work needed for supporting service features and GAing some other tools. Our goal is to post a sort of design doc here once we have one that people can comment on, but I don't have a very solid timeline for that at the moment, unfortunately.

+1 for having JSR203 NIO FileSystemProvider for Azure storage.

This would be very useful to have, any updates?

We got up to the point of pulling the initial work item into our sprint and then ended up having to defer it becuase several crucial stability issues arose in v11 at the same time and eventually led to this effort to rework things for v12 (now in preview). I completely recognize that this is probably our most requested feature for this sdk, so once we our confident in the stability of our baseline sdk, this will be a high priority for us (which I realize I've been saying for a year, but I promise I'm not just stalling).

Any updates on this @rickle-msft ? Our customer needs this urgently

Unfortunately not. Since the last update, we have been working on GAing v12, which is now out. Our management is currently doing some planning for the next semester starting in the new year, and I submitted this request to be considered for inclusion in that work. I don't have any word yet on whether or not it was accepted.

Thanks for getting back to me, @rickle-msft ! I appreciate it.
BTW, here is some relevant info from AWS and GCS that you might find relevant.

Thank you for sharing that! I am sure it will speed up our development to have these references when we get started on this project. I know this has been eagerly anticipated for a while, so I am looking forward to getting started on it whenever I get a green light

To all those still watching this thread, I will begin investigating this again sometime next week and will start on an implementation thereafter. I will continue to post updates of progress here.

Quick update: This package is in master and is actively moving towards preview. You can see the progress here. The latest PR is here and includes support for creating directories.

There is also a design doc posted in the root of that project if anyone has an interest in reading that to understand the approach we are taking. As development continues, javadocs on the public methods will also be an important source for understanding more details of the design.

Hi! Is there any updates on this feature?

@SHarrySeldon Yes! We are hoping to preview in the next month or two. You can see the current state of the project here. The next planned items are adding support for InputStream and OutputStream.

If you have any scenarios that are particularly interesting to you or are interested in installing and trying out what's available, we would be happy to have your input.

Hello, everyone!

I am very happy to announce that we have officially released the first preview of Azure Storage Blob NIO. The package info is

<groupId>com.azure</groupId>
  <artifactId>azure-storage-blob-nio</artifactId>
  <version>12.0.0-beta.1</version>

and you should be able to import it into your projects now.

Thank you all for your patience, and please let us know if you have any feedback; we look forward to hearing from you! I am going to close this particular issue as we have released now, but please do feel free to open new issues or feature requests as you have them.

@SHarrySeldon @ryan-williams @pditommaso @fstrozzi @stewartadam @landro @nalinigans

Nice! Is beta.2 expected to be avail/public? Can only see beta.1.

Sorry. Beta.1 is the version we just shipped. My comment was wrong (I accidentally pulled from the pom after we did our version increment).

Ok, no pb.