Mongoose: Ability to specify which ES6 promises library mongoose uses

Really looking forward to being able to use Promise.all(), to do something after all database work has been done.

:+1:

One thing that wasn't very clear on https://github.com/LearnBoost/mongoose/issues/1699 is what implementation will be the default.

Well given that it's not a backwards-breaking release, mpromise will have to be the default, but you'll be able to override.

+1

:+1:

Is there a branch where this is being hacked on? I can not stand the error handling flow with mpromise and I'm about to crack open the source to see what this would take.

mpromise:

  query.exec()
    .then(function(ou) {
      if(!ou) {
        return next(new errors.http.NotFound('The specified OU was either not found, or your credentials lack the required permissions to view it.'));
      }

      res.send(ou);
    }, next)
    .end(next);

Have to handle the rejection and also put end there. Without end, exceptions (for example, I had NotFound class misspelled) are silently swallowed and express just stalls.

bluebird:

  query.exec()
    .then(function(ou) {
      if(!ou) {
        return next(new errors.http.NotFound('The specified OU was either not found, or your credentials lack the required permissions to view it.'));
      }

      res.send(ou);
    })
    .catch(next);

Doing promisifyAll(require('mongoose')) seems to actually be still be working with Mongoose 4 so far. Would regression tests that cover this be too targeted?

Not at the moment. Most of the work will take place in the kareem module as per #2754 and vkarpov15/kareem#2, because that will allow us to kill two birds with one stone and remove the really messy point cut nonsense that was written to make hooks and promises work together. Feel free to take a crack at it though, I'm open to PRs.

But why should we keep support for other promises library? The ES6 Promises spec is now rock-solid and here to stay. Can't we just use pure ES6 Promises, with a polyfill loaded when they are not available in older versions of node?

If so, I can have a try at it.

The point is that you'll be able to use whatever ES6-compatible promises library you like. Plenty of people are still heavily invested in bluebird, when, q, rsvp, etc. etc. and each of those libraries has it's own particular quirks that a generic polyfill won't capture.

I'm open to alternative suggestions - I don't particularly like or use promises, this feature is motivated by the fact that there's a mountain of issues involving people asking for "support bluebird feature X in mpromise" or "native rsvp.js support" and letting people bring their own promises library to the party is the easiest way to close these issues.

I see what you mean. I'm a big user and supporter of the Promises. I think it should be considered to enforce the standard.
A+ promises has been chosen as the go-to for the ES6. I suggested the polyfill to ensure support for the ES6 incompatible node versions (eg. https://github.com/jakearchibald/es6-promise).

It should be in the hands of the other promises library to be compatible and mixable with the ES6 promises.

EDIT:
And there would be no breaking in the current API or am I missing something?

The Promises/A+ spec is very different from the ES6 promises spec, which is in turn different from the promises libraries I listed in the previous comments. While it would be good for me if the myriad promises libraries all consolidated into ES6, I doubt that's ever going to happen, because the beauty of open source is that some people who love promises are going to want additional features and will write their own promises libraries.

No breaking change for the current API, what I'm thinking is a way to say mongoose.set('Promise', require('bluebird')); or something like that, so it would be an opt-in and mpromise would be the default.

Oh right, sorry for that mistake.
I've taken a look at the current implementation, and the other promises library.

I think I can make something like this work:

mongoose.set('Promise', Promise);

mongoose.set('Promise', require('bluebird'));

mongoose.set('Promise', require('q').defer());

mongoose.set('Promise', require('when').defer());

// and so on...

So basically, you should expose to Mongoose your promise of choice object that has the resolve and reject methods.

Would this be what you had in mind? If so, I will work on a pull request.

EDIT:
It feels extremely silly to write mongoose.set('Promise', Promise);. I think ES6 should be the default, with the possibility to use your library of choice (and mpromise whenever ES6 Promises are not available).

Sure, I'd appreciate your help. The tricky bit would be 1) getting it to work with hooks - see vkarpov15/kareem#2, and 2) making it backwards compatible with mpromise.

Also, re: Q, we'll use require('q').Promise since that's the Q syntax that's closest to the ES6 spec

Thanks for your input, will work on it. Do you prefer a competed pull request or an ongoing PR?

Completed is better, but I'll invariably have some suggestions. Let me know if you get stuck

+1 for this feature. i would like to use with bluebird

Sorry about being drastic - but an alternative approach would be to deprecate promises support altogether. That would shut up anyone using promises asking for the support of other libraries.
People using powerful promise libraries like bluebird can keep using it anyway since you expose a callback API and bluebird can trivially wrap that at a 0 cost - in fact chances are that since you don't know or use promises (like you said) chances are it'll be slower an more error prone to support them manually anyway.

@benjamingr 100% disagree. Now with iojs and NodeJs merging into Node 3.0, there will be support for ES6 Promises and Generators.

Dropping the support would be a huge step backwards.

That is, Promise.promisifyAll(require("mongoose")) creates a fast (faster than any likely manual attempts) wrapper for Mongoose that is standards complaint and surrounds all the API without you having to do anything about it. In fact, you can do Promise.promisifyAll on the exports object yourself and expose bluebird promises and dual promises method (save - saveAsync) for free and then say you expose a bluebird promise interface without having to do any actual work.

So, while I use bluebird promises myself - I think this whole thing can be fixed by a section in the docs rather than complicating your code by coding to two interfaces.

@albertorestifo except nothing will actually be dropped. I'm very well aware of what promises are (over 1500 points and 500 answers in stack overflow :P) and I'm even responsible for some parts of how they work in io.js (like https://github.com/nodejs/io.js/issues/256).

That doesn't change the fact I believe that since @vkarpov15 doesn't use promises he shouldn't have to support them in his library _especially since that brings no benefit over using promises with it anyway_. You can use promises with mongoose as easily even if it doesn't provide them - mongoose implementing support internally is more to maintain, likely slower, and more error prone. @vkarpov15 can continue working on the callback API and promise users can wrap Mongoose with promises with an easy one liner anyway.

@benjamingr that is supposing one uses an external Promise library. I'm sticking to the one in the specs, and you know very well, it has no wrapper.

I do see your point. Supporting both creates a huge mess. I personally despise callbacks, so I would drop those instead. Maybe there should be two separate repositories, one using generators and promises, the other using callbacks. Both would maintain the same structure and the API as close as possible.

@benjamingr that is supposing one uses an external Promise library. I'm sticking to the one in the specs, and you know very well, it has no wrapper.

If you choose to use a slow, harder to debug and less feature rich implementation that is of course your choice to make :P but how is that related to the wrapper? It's perfectly easy to write a wrapper similar to bluebird's promisifyAll using native promises*.

If you want a promise - enabled version of Mongoose as a separate package here's how you could do it:

• Step 1, open your favorite editor or just an editor you're kind of OK with.
• Step 2, type module.exports = require("bluebird").promisifyAll(require("mongoose"))
• Step 3, create appropriate package.json file, publish to npm
• Step 4, tens of thousands of downloads.

Now, I know what you're thinking "this isn't using native promises", well you can still delete every method except then and catch from the promise prototype and all and race from Promise and end up with the same API - or you can just tell people you're exporting native promises - they won't know since it's just two Promises/A+ implementations, I promise ;)

(*writing it fast is harder in the user level because there is no fast way to create promises right now which is why io.js will likely export a promisify function itself eventually - that said by taking a promise constructor you're forcing it to be slow anyway).

That is definitely a decent alternative. I would like to keep promises in-house though, because, like it or not, that's how users are going to be using mongoose, so we might as well have solid test coverage for it, so we can point and say "this is how you use promise library X with mongoose". The downside when you disconnect things into separate modules is that it's tricky to say "ok this version of mongoose-promises only works with mongoose 3.8, this one works with mongoose >= 4.1" and it's difficult for mongoose to avoid breaking an overarching promises wrapper.

That is definitely a decent alternative. I would like to keep promises in-house though, because, like it or not, that's how users are going to be using mongoose, so we might as well have solid test coverage for it,

Why would you want/need test coverage for it? There is no point in testing promises themselves in your code - the libraries have tests already - this is like testing the http module when you're using it.

and it's difficult for mongoose to avoid breaking an overarching promises wrapper.

Bluebird does something really simple - it finds prototypes and then adds methods with the Async suffix to them - this is really simple and it works well in practice - it's a one liner with most libraries including Mongoose and it did not break even once for me in the past year.

I'm not sure why you'd want to maintain a lot of stitching code that is potentially error prone, manually having to support two APIs with their edge cases sounds like a lot of work and you can break things as you go - you can "borrow" bluebird's promisifyAll code and adapt it to work with other promise libraries (it's open source after all) but I certainly wouldn't do it manually.

"ok this version of mongoose-promises only works with mongoose 3.8, this one works with mongoose >= 4.1" and it's difficult for mongoose to avoid breaking an overarching promises wrapper.

Well, would you mind giving me an example of a breaking change that'd have to happen if promisification is automatic?

1) I'd like to test things the way users use them.

2) I'd love to be lazy and avoid it, but as I understand it most other promises libraries don't have a promisifyAll equivalent. I assume this is why "support Xpromise feature Y" is the most popular mongoose feature request. Furthermore, we don't have any intention of rewriting Bluebird's promisification, just make mongoose functions return a promise natively.

3) Depends on the implementation of promisification and how you're using it :)

1) I'd like to test things the way users use them.

What do you mean by testing the way users use them? Can you show me an analogy for callbacks?

2) I'd love to be lazy and avoid it, but as I understand it most other promises libraries don't have a promisifyAll equivalent. I assume this is why "support Xpromise feature Y" is the most popular mongoose feature request. Furthermore, we don't have any intention of rewriting Bluebird's promisification, just make mongoose functions return a promise natively.

You don't have to rewrite, you can take it - it's not generic since a generic implementation would be slower.

It's also not laziness not to manually implement a widely available feature. Are NodeJS lazy for not putting express in core? Are TC39 lazy for not putting underscore in core? By sticking to a convention (callbacks) you are giving users the ability to use whichever concurrency primitive they want.

3) Depends on the implementation of promisification and how you're using it :)

Well, bluebird's promisification, or Q's or When's - they vary in implementation but they all do the same thing - so just pick one you like. I'm just wondering how it'd break.

There is one thing I'm missing in this discussion here:

If mongoose returns a standard promise (by standard meaning the ES6 native implementation), should't it be compatible with any ES6-compatible Promise library? I can do Promise.all([model.query().exec(), ...]) just fine, as well as the bluebird, q, when equivalent.

So, why not return callbacks and standard promises (like it's doing now, but "getting rid" of mpromise) and let the user use it's favourite promise library? Or am I missing something here?

@albertorestifo well, native promises are slow at the moment and it'll take time, potentially years before they reach parity with userland libraries - so mainly that.

@benjamingr you make some good points. The primary point of mongoose's promises is to enable you to use yield with mongoose async operations without any other library, which is why we're keeping promises for the foreseeable future. IMO that's something that should really be part of mongoose core going forward.

Do Q or When have promisification capability?

Do Q or When have promisification capability?

Yes, virtually every widespread promise library I know offer some sort of promisification:

Here's when: https://github.com/cujojs/when/blob/master/docs/api.md#nodeliftall
Here's Q: https://github.com/kriskowal/q/wiki/API-Reference#qnfbindnodefunc-args

Native promises don't have it yet, but it's something that's being worked on - once a fast path for creating promises (that is - not the promise constructor) exists then NodeJS will likely support it in core for native promises (since it can't be done _fast_ in userland).

The primary point of mongoose's promises is to enable you to use yield with mongoose async operations without any other library

You need a library in order to use yield in a meaningful way with promises anyway. If you can write the 9 LoC that pump a generator as an async function yourself you can definitely write promisify - and if you're like most users you use a library for that anyway.

I definitely see the want/need to allow promises in Mongoose, they're the way forward and how the language does concurrency now but I honestly think that doing it manually method-by-method will be painful. It might be beneficial to just demonstrate how it's done using libraries in a "using with promises" or "using with generators" section in the documentation.

The thing is that we already have done it method-by-method, we just need to modify the internal promise wrapper. Either way, we can't quite remove promises from the core until 5.0. I'm amenable to the idea of deprecating it, @benjamingr does make some good arguments that I'm gonna have to consider closely, but I think the next step would be #2688 anyway.

You obviously have more experience with Mongoose and more importantly it's users - so I understand that choice. Thanks a lot for hearing me out.

I typed "its" but my iPhone felt like it should autocorrect it to "it's" and GitHub felt that editing comments is not an interesting use case. Sorry for double-comment-spam :)

You can edit comments on the website if you click the pencil in the top right of your comment.

I'm always open to a good hearty debate, especially one that teaches me something new :beers: I may ping you later to discuss more :)

The future belongs to native promises, it seems. Mongoose is an industry standard for mongo ops, it's totally valid for it to rely on standard promises as well.

Standard promises will inevitably mature and become more widespread than any framework. A humble +1 for using them.

@iliakan perhaps in the future. I wouldn't expect native promises to become the "standard" until mid-2016 at the earliest, the various fragmented promises libraries are too deeply entrenched and have too many subtle quirks. Either way, mongoose can't switch to 'native by default' without a massive backwards breaking change.

@vkarpov15 sure I understand.

What could be beneficial meanwhile - a simple page which outlines major incompatibilities between mpromise and native promises. At least things you shouldn't try with mpromise ;)

Do I get it right that there's no catch right now, and that's all the limitations?

No idea, haven't really dug in to the ES6 API. mpromise implements Promises/A+ and nothing else, which means no .catch(), no long stack traces, etc. etc. Basically, anything that isn't .then() is outside the scope of mpromise's implementation.

Generally speaking, something that implements Promises/A+ also implements ES6 promises from a high-level perspective, but the reverse is not true. Promises/A+ is very specific about the low-level implementation details, for instance, bluebird does not obey Promises/A+ exactly, and I'm not sure about other common promises libraries but I'm sure they don't obey the spec in their own unique ways. That's what's going to make this particularly tricky.

You mean, can now use bluebird and native promises?

Yup require('mongoose').Promise = global.Promise will make mongoose use native promises. You should be able to use any ES6 promise constructor though, but right now we only test with native, bluebird, and Q

@vkarpov15 This is great! Thank you so much!

@vkarpov15 Thanks a lot! Great job!

Yeah definitely. That's really cool! :)

@vkarpov15 This is really nice