mocha 4 doesn't exit unlike mocha 3

Created on 3 Oct 2017  ·  61Comments  ·  Source: mochajs/mocha

Prerequisites

  • [x ] Checked that your issue isn't already filed by cross referencing issues with the common mistake label
  • [x] Checked next-gen ES issues and syntax problems by using the same environment and/or transpiler configuration without Mocha to ensure it isn't just a feature that actually isn't supported in the environment in question or a bug in your code.
  • [ x] 'Smoke tested' the code to be tested by running it outside the real test suite to get a better sense of whether the problem is in the code under test, your usage of Mocha, or Mocha itself
  • [ x] Ensured that there is no discrepancy between the locally and globally installed versions of Mocha. You can find them with:
    node node_modules/.bin/mocha --version(Local) and mocha --version(Global). We recommend avoiding the use of globally installed Mocha.

Description

I have been running specific set of tests for few years now and have been always upgrading to the latest mocha and everything was ok.
With mocha 4 suddenly all tests are passing but it doesn't end as if the --no-exit is automatically added although I never added it.

Steps to Reproduce

Expected behavior:
After all tests end, the process should stop even if there are timeoutes or sockets that prevent the process from existing.

Actual behavior:
Mocha 4 process waits forever like mocha 3 with --no-exit flag

Reproduces how often:
With our tests always. I have 700 tests so it is hard to pinpoint which one casuses the issue or if maybe it is in our codebase.

Versions

mocha 4.0.0 fails. before that everything works good.

faq question

Most helpful comment

Other than providing a quick fix (use --exit), I do agree they left the core issue of finding the faulty tests to the user. I'm struggling with that myself now, but when upgrading major versions, I make sure to read the release notes and not blindly upgrade

All 61 comments

I am seeing the exact same problem. Tests will pass and then mocha just hangs. See my Travis CI:
https://travis-ci.org/mkrufky/node-dvbtee/builds/282593109

I also noticed this same problem on video-dev/hls.js - mocha just hangs after passing tests:
https://travis-ci.org/video-dev/hls.js/builds/282590422

Thanks. tok bad the mocha website is not updated with this breaking change. The cli args there do not mention it.

Other than providing a quick fix (use --exit), I do agree they left the core issue of finding the faulty tests to the user. I'm struggling with that myself now, but when upgrading major versions, I make sure to read the release notes and not blindly upgrade

The release notes suggest using why-is-node-running except it doesn't work due to https://github.com/mafintosh/why-is-node-running/issues/7

Hit by this too. I understand the reasoning behind the change to the default, and thank you for incrementing the major version, but given that why-is-node-running is abandoned and broken, this may not be the most user-friendly change.

Hi all,

First off, I want to apologize for the rough upgrade path on this point -- we definitely agree that Mocha ought to tell the user what's keeping the tests running (and then it wouldn't even need to keep the process open; it could just be a reason to return failure after they're done), but haven't found an entirely satisfactory way to do so yet. Version 4 didn't get the amount of time we wanted to put into it, as it was prompted by our CI failing due to changes in PhantomJS 1.x's installer (package-lock.json probably would have prevented this if we'd had it set up beforehand, but we still can't get it working). why-is-node-running was the one tool we found to help, but we don't think we can integrate it (between the requirement of --expose-internals and lack of a good way to programmatically get its output); I have found that it does work if you follow a couple steps:

  • run Mocha with --expose-internals (node --expose-internals node_modules/mocha/bin/_mocha)
  • make your first test file (e.g. listed in mocha.opts) contain (or at least begin with) after(require('why-is-node-running'))

...so it's better than nothing (although I'll see if I can update the release notes to describe this in more detail), but if anyone knows a better option please let us know!

Also sorry about missing the site documentation -- we'll get it updated as soon as possible. (Once #2987 is done we can even make it an automatic part of our version/publish scripting!)

@ScottFreeCode

borisov@glossy:~/test/mocha $ node --version
v6.11.3

borisov@glossy:~/test/mocha $ ./node_modules/.bin/mocha --version
4.0.0

borisov@glossy:~/test/mocha $ ./node_modules/.bin/mocha --expose-internals test.spec.js 
  error: unknown option `--expose-internals'

Edit:

This works:

node --expose-internals ./node_modules/.bin/mocha test.spec.js

Right, sorry I wasn't clear about that -- --expose-internals is a Node option that can be used by running node --expose-internals ./node_modules/mocha/bin/_mocha instead of ./node_modules/.bin/mocha. That's also something we can fix, since Mocha passes certain flags on to Node and we can add --expose-internals to those flags.

Created mochajs/mochajs.github.io#81 and #3045 to track updating the documentation and Mocha's Node flags, respectively. Will keep this issue open at least till the documentation is updated.

@ScottFreeCode you may want to mention that --expose-internals only works for node <= 6. Hopefully people can downgrade to node 6 temporarily so they can find what timer needs to be cancelled or unref'ed, or sockets needing to be unref'ed. You may also want to point people to after and afterEach hooks to do cleanup.

(is there a global "after" hook that mocha calls when all tests are complete?)

In any event, I appreciate the help, and thanks for Mocha!

you may want to mention that --expose-internals only works for node <= 6.

Are you sure? I tested with Node 8.6.0. Albeit not on all OSes, so I guess I should fire up every box I have and triple-check...

Are you sure? I tested with Node 8.6.0. Albeit not on all OSes, so I guess I should fire up every box I have and triple-check...

This "works" in that it triggers the output from the module, but it doesn't actually give me much information, regardless of the version of Node.js.

I'm going to add some updates to the site, but please see my forthcoming comments on #3045.

This "works" in that it triggers the output from the module, but it doesn't actually give me much information, regardless of the version of Node.js.

It gives a useful stacktrace for setTimeout and setImmediate, but no real info at all for other things such as a listening server (as I just recently found out while trying to understand how it's doing what it's doing). The "async-dump" example in the gist has a real advantage in terms of working for everything (and not requiring --expose-internals), even though it's only useable on newer versions of Node.

I suppose my general advice would be "if you're pulling your hair out, just add --exit and relax". Then don't use it for your next project :wink:

Sorry to comment on the closed PR again, but there might a user-friendly solution here:

One could log a warning about the changed behavior after the runner has finished for 3s or so, but the process didn't exit.
Would just need to add a setTimeout() after the runner finished and call timeout.unref() to make sure this timeout doesn't prevent the process from exiting. If the timeout gets executed it's time for a warning 😉

I’ve considered that but I’m wary that it will cause other problems with reporters or other integrations... I can’t prove it won’t.

If you're still having problems and why-is-node-running doesn't work right, check out wtfnode.

That package worked much better for me.

I can see how having existing tests suddenly stop exiting can be disconcerting (to say the least). I do see a mention about the new behavior in the release notes.

I discovered the change accidentally myself with new tests that forced me to ensure I called redis.quit() in an after() function and I am definitely pleased with the new behavior, which seems correct and appropriate to me.

I didn't need to use [this gist] this time, but as already mentioned by @boneskull, it looks like it could be really useful when it's not clear which async tasks haven't completed and are holding up the process from exiting.

I must admit that I don't quite understand the issues in the referenced tickets that lay behind this new behavioural change.

AFAICT, it has something to do with unhandled Promise rejections. But pardon me, if you don't resolve a Promise given to a Mocha test, then Mocha will neither succeed nor continue (if --bail is set) from that test, right? So it must be some poorly implemented nested Promise which doesn't have a rejection handler, but the containing code must be resolving anyway.

If this is the case, I don't see how it's Mocha's job to detect those issues. And if implementing some magic to detect those issues, which may make current (correctly-written) tests hang indefinitely, then that magic should be an opt-in behaviour, not an opt-out one — i.e. --no-exit rather than --exit.

I'm looking at all of my tests using the official node-mongodb-native driver hanging because that driver pools connections and don't close them all on .close(), seemingly be design. My tests are behaving properly, but a dependency is not (or is it? I don't know necessarily for a fact that it's bad behaviour to have lingering sockets or file descriptors until a script is exited elsewhere — do you?). So what am I testing here? My code, or someone else's? I thought I was testing my own, but apparently not.

Sure, I can add the --exit flag, but the next guy might not, and either way, it just seems wrong that I can write perfectly fine tests with perfectly fine code being tested, and still have some random dependency cause my tests to hang indefinitely.

If I add a process.exit() in a final after() hook, my tests won't hang, but then it'll break horribly with --watch (possibly other features), not only preventing me from watching for changes, but ejecting me into a cursorless shell.

It may very well be that I'm the clueless one here (certainly there's a lot I'm not getting, and a lot of people have been involved, so I would tend to think so :) ), I just feel like this whole thing is not quite... right...?

Cheers :)

EDIT: Given this behavior, is there any way for me to check inside a Mocha test if it's being run with --watch so that I can make sure not to call process.exit() and break things?

@DanielSmedegaardBuus TL;DR on the solution: put --exit in the mocha.opts file. (This is typically the solution for anything that needs to always be set when Mocha is run, as a more general tip.) More detailed clarification of what this change is about: below.

2640 is about promise rejections, isn't intended to do anything but make them consistent with synchronous exceptions thrown from non-promise asynchronous test code, and hasn't been implemented yet.

This is about tests setting up resources, listeners or ongoing work and not cleaning it up, regardless of whether promises are used in their implementation and/or in the test. For example:

I have a test for an API based on websockets, which includes opening and closing connections. This API was buggy and was not closing the connections right, but passed my tests since the tests didn't really have a way to assert that the underlying connection was treated correctly. (If I had strictly tested only my own code, I could have verified it's using the dependency in what I mistakenly thought was the right way; I was testing with the real websocket precisely to catch issues with usage correctness in the first place however -- integration tests to complement the unit tests.) I caught that error when I switched to the --no-exit behavior (of Mocha 3; now the default in Mocha 4) and discovered that if I ran those tests Node would not close because the websocket connection is still listening.

(It's true it's possible that the error lies in a dependency rather than in your own code, but even then, there can be value in knowing that before you ship with a given version of that dependency -- as alluded-to above, this is more applicable for integration tests in the first place than for unit tests, but ideally a project has both of those.)

This isn't always necessary, of course -- in some cases, a resource of this nature might be meant to be left alive till the process is killed, or might keep Node alive even though no other cleanup of the resource really matters, or might be part of a resource pool that reuses individual members and doesn't need to clean up the pool as a whole -- in such cases --exit would be correct. The reason for the change of the default behavior was to raise the visibility of such issues: before, you would probably never know to try --no-exit and check for these kinds of errors, now you'll run into them by default and can --exit if you determine that the behavior is correct (or at least not worth worrying about).

There is, of course, something of an issue with the "fix" that just hanging when this happens isn't very informative. We've still got to figure out if we can integrate some way of actually programmatically checking for things still running (so a proper warning or error can be given) that will work on all supported versions of Node without interfering with anything that might still be cleaning up or closing when Mocha reaches the end.

I'm sorry, but this was very frustrating. I was just trying to find out why mocha won't exit. https://boneskull.com/mocha-v4-nears-release/#mochawontforceexit links to why-is-node-running, so now I have gone, tried to use that module, seen a cryptic message "Error: Cannot find module 'internal/linkedlist' ... (stack trace)", and after following the rabbit hole trying to find out why --expose-internals doesn't work, I wind up at https://github.com/mochajs/mocha/issues/3045 to find a supposed solution, but why-is-node-running tells me that there is 1 known handle keeping it open, and a few unknown handles, but doesn't list anything.

This might not be the right place to bring this up, but mocha's docs shouldn't be recommending solutions that require this much googling and issue diving to get to work. I was trying taking things out and adding them in, with very inconsistent behavior. For the sake of future users and my sanity:

package.json:

"scripts": {
    "test": "mocha --exit"
}

That's the best I can do.

@jeffvandyke I understand your frustration and went through the same issues--but if you've got any assertions behind promises, you may appreciate this change.

I had a number of specs that had flaky async behavior. Depending on the run order of my specs, uncaught promise rejection errors would be logged to the console and then be scrolled out of sight by hundreds of other tests that passed, or the node instance would shut down before the rejection happened.

After removing the --exit from my mocha.opts, the rejected promises would reliably log errors, and highlight those bugs.

I know, lots of shared pain :) Mocha's documented behavior actually sounds correct to me, but even though my 5 tests seem simple enough, just using node-fetch to test an api, I'm still unsure why mocha doesn't exit, even though they all pass successfully (yes, they are able to fail too). I could spend more time trying to figure out why why-is-node-running isn't working, but I'm tired of monkeying about other people's code, so I think I'll give myself a break for now.

I'll probably keep playing with it, and if I can get something reproducible, I might open a new issue, no promises though.

would recommend the new node inspector’s debugger... it has good async traces.

Yay! Turns out wtfnode is much better at shedding light on this. I found out that when using node-fetch, I have to call result.text() or .json() or the connection stays open. Running wtfnode ./node_modules/.bin/_mocha and pressing Ctrl-C revealed the open connections.

I think it would have been easier if the docs recommended this package rather than why-is-node-running.

Another suggestion: I really like @andywer's idea of having some kind of warning if mocha dosn't exit. It would be sweet if mocha could even use wtfnode somehow to list active handles keeping node alive after some time has passed.

Whatever the right thing should be, I think it's garbage that I had to look this far to figure out why mocha wouldn't exit. Anyway, I've given my suggestions :)

Hi guys,
I think this feature might still be not working properly, this is the most basic test I could think of and mocha still fails to exit by itself. --no-exit is an excellent default option and I am all for it, but either I'm failing at understanding what I am doing wrong or something is intrinsically wrong with mocha and it's preventing to close even most simple of tests.

describe('describe', function() { it('it', function(done) { done(); }); });

summary: --exit does work, --no-exit never closes any test.

In my case I'm using a Koa application, and testing with Mocha 4 + Supertest.

I had to just close server together with done call following release notes "To avoid false positives and encourage better testing practices".

Before:

request(app.listen())
  .post('/')
  .send(requestSrc)
  .expect({ f: {} }, done)

After:

const server = app.listen()

request(server)
  .post('/')
  .send(requestSrc)
  .expect({ f: {} }, () => {
    server.close()
    done()
  })

Hope it helps somenoe.

I'm doing some bootstrapping for mocha before every test. Basically starting a little HTTP server in the same process.

Is there an event I can hook into to shut it down after Mocha stops running? I just want to shut it down, but just at that time.

I love the idea behind this feature btw.. it's a good test for seeing if you have dangling events.

@evert Doesn't it work closing server in after?

Is there a global 'after everything' that runs after mocha is done? I may have missed that! Couldn't find it in the docs.

Yes, there is global after hook.

Thank you! I missed that, sorry :/ didn't know the terms to ctrl-f for.

wtfnode didn't really work for me. All I could see was that some PID started by mocha was hanging.

On the other hand, @boneskull gist worked: https://github.com/mochajs/mocha/issues/3044#issuecomment-351299745. Thank you!

wtfnode needs to be run against _mocha, not mocha.

Thanks for the --exit. It fixed it for me.

enter this script in package.json:
"scripts": {
"test": "mocha --exit"
}

I tried various remedies mentioned in this thread:

  • why-is-node-running produces no output
  • wtfnode produces output but not enough to determine where the File descriptors/Sockets/Servers/Timers are created in the code
  • async_hooks debug method errors
  • adding –exit to mocha cmd line flag results in exit

So follow @ProfJigsaw and use –exit, that’s what I am doing. Would be great if there was an improvement by mocha though, at least a way to figure out where issues are and how to resolve them.

That's what a debugger's for, IMO. How would you debug your own scripts if they never exited?

https://github.com/GoogleChromeLabs/ndb is a cool project.

Off-topic: I love this ticket, just for the information on troubleshooting tools it keeps on giving! :)

@borisovg Imagine how great this ticket would be if it actually provided a solution! :)
@boneskull Is there some "list File descriptors/Sockets/Servers/Timer" debugger feature that I am not aware of?

@mjgs it did for me - wtfnode worked but only when I used it with the _mocha binary, not the mocha one:

$ wtfnode ./node_modules/.bin/_mocha my-shitty-code.spec.js 


  tests
    ✓ some test


  1 passing (5ms)

^C[WTF Node?] open handles:
- File descriptors: (note: stdio always exists)
  - fd 1 (tty) (stdio)
  - fd 2 (tty) (stdio)
- Servers:
  - :::8080 (HTTP)
    - Listeners:
      - request: (anonymous) @ /home/borisov/test/my-shitty-code.js:4
- Intervals:
  - (5000 ~ 5 s) (anonymous) @ /home/borisov/test/my-shitty-code.js:8

@borisovg Thanks for your really super example. In my case wtfnode shows there is a mongo connection open, but all mongo connections in my my-shitty-code.js have been closed, so the problem is from somewhere else, and wtfnode doesn't give enough info as to where it is from.

If wtfnode indicates that a mongo db connection is open, then it's open. Why would you assume it's wrong?

@evert I think you might have misinterpreted what I wrote

Minimal examples are great for illustration, and it's actually useful to see similar output to what you are seeing, but in real code that has been written over many years these connections aren't so easy to find. It's good to know that there are mongo connections still open, but to actually find them is non-trivial. I eventually found some open connections, but they were much much deeper in the code than the surface level where all connections had been verifiably closed, wtfnode didn't help in identifying where they were only that they existed by listing the mongoose module path. I still have some to track down, based only on port number, but I am hopeful.

I ran into this very issue writing a Serverless function that communicated with Firebase. Turns out their admin SDK maintains an open handle indefinitely. Because of this change (and the subsequent suggestion to use wtfnode, I was able to identify that fact and save myself a ton of headache (and costs) down the road.

In my opinion, it would be very helpful to include some sort of "if it hangs for X long and doesn't have any output throw some text to stdout" logic. I'm unsure how feasible that is or how much bandwidth is available to produce such an improvement, but I think that might help alleviate some initial frustration of "wtf is going on with my mocha!"

var timers = sinon.useFakeTimers({
    now: new Date().getTime(),
    shouldAdvanceTime: true,
});

If I forget timers.restore(); the process hangs forever.

Based on the documentation @pgilad sent here, to me there is a cleaner solution. As the documentaion says:

To avoid false positives and encourage better testing practices, Mocha will no longer automatically kill itself via process.exit() when it thinks it should be done running.

A cleaner solution then it would be to create a global after function (an after outside any describe function), I would recommend in a separate file like exit-mocha.js or exit-mocha if you will. The callback sent to after you can force a gracefully node process exit which is going to exit without any error. The file can be sent to mocha cli as if it was another test file (it could simulate an --exit flag)

exit-mocha.js or exit-mocha

after('Exit mocha gracefully after finishing all tests execution'. function () {
  // Exit node process
  process.exit();
});

Then you can execute mocha tests like:

mocha exit-mocha test/**/*.spec.js

or

mocha exit-mocha.js test/**/*.spec.js

It is important that if you are using wildcards for the name of the tests files like I did with test/**/*.spec.js that the name of the exit-mocha file does NOT match the wildcard pattern, otherwise, it will not be possible for you to use it as a "flag"

@vctr90 That's a great solution, although you have a period where you should have a comma in your example. Also the whole thing can be code golf-ed down to just:

after('Exit mocha gracefully after finishing all tests execution', process.exit);

Is there any chance a dev could chime in on why adding this to Mocha proper would hurt someone (because it would clearly help a lot of people)?

@machineghost I like the new behavior for 2 reasons:

  1. Almost no other library will exit after it's 'done its work'. For example, closing a Node TCP server will not automatically trigger an exit. In that way, it's consistent with other libraries.
  2. If node doesn't exit, it means that there's events still waiting to be resolved. It's probably better to try and improve your code so things get cleaned up after each test. It could also suggest that there's memory leaks.

So when I run into this, this is an incentive for me to try and clean up my code so it doesn't happen.

Your perspective on this might be 'i don't care about memory leaks', or 'this is not worth fixing in my application'. If you're in this category the easiest is to make a mocha.opts and add --exit.

It just seems to me that this is about making more noise, not signal: in the significant majority of cases no one's app is going to get better because Mocha hung at the end.

If Mocha is going to default, it seems like it should default to ending when tests (and all after/afterEach calls) have finished. Not doing so just to tell people "hey your artificial testing environment is artificial" doesn't benefit the majority (or even a decent minority) of users.

If people really want to debug unclosed connections then it seems that should be the case you provide an option for. But all the rest of the time, does it really benefit the library's users to confuse everyone else with (what amounts to saying) "you're in a testing environment and one of a billion possible things is open"?

To put it another way, if you're going to tell 99% of the people that run into this "just use --exit" then maybe --exit shouldn't be a special option you have to provide ... maybe the default behavior should be to serve 99% of the user cases (while of course still giving users the _option_ of --tell-me-if-I-have-unclosed-stuff-in-my-testing-environment-and-that-is-actually-what-i-am-trying-to-find-out)?

I mean, if you made that the option, realistically how often do you think people would even pass it?

P.S. I just came across this: https://stackoverflow.com/questions/54999115/where-to-destroy-knex-connection. If you look at the second answer, this behavior in Mocha directly caused at least one user (who isn't me, and I swear I don't know them: I just found this by luck after making my last post) to add incorrect non-test code (this "feature" made them incorrectly think needed to call knex.destroy() in their app).

Summary version:

Nice Person: You probably don't usually need to explicitly call knex.destroy() – this is implied by the documentation itself saying (emphasis mine):

Nice Person: quotes docs

Confused Person: If I don't call knex.destroy() my script will hang. Then what does it mean we don't need to explicitly call knex.destroy()

Nice Person: Ah, you only mentioned elsewhere that this is for Mocha. For regular server uses you don't need to tear down the connection pool – for Mocha, you might want to look into a global teardown hook, see futurestud.io/tutorials/…

But to be clear, this user had a working environment, and because Mocha (essentially) told them their perfectly good code was wrong, they lost who knows how much time trying to solve the bug they created by destroying their connections. And that's just this one unlucky person who happened to do it in public, and who I happened to see.

So, I guess what I'm trying to convey is, this isn't just about what's philosophically correct, nor is it just that some noise is obscuring useful signals ... this behavior is actually causing harm, and making programmers waste time tilting at windmills.

I'm sorry, I mistook your tired point for an honest question. I regret answering. Maybe the developers can lock this thread.

"mocha --reporter mocha-allure-reporter ./tests/controllers --exit" worked for me. Indeed the --exit is a very good workaround. I use the version 5.2.0 in my project.

Is there a way to get the equivalent effect of using the --exit flag when using mocha "programmatically?"

I do not see a documented option for exit/noexit, nor a general way to pass in a flag string while using the NodeJS API.

Following the pattern of other options, I tried:

const mocha = new Mocha({
    exit: true,
});

but was not able to get the desired effect.

Cursory inspection of https://github.com/mochajs/mocha/blob/master/lib/mocha.js seems to show that this option needs to be added to not only the documentation, but also to the source.

Other than providing a quick fix (use --exit), I do agree they left the core issue of finding the faulty tests to the user. I'm struggling with that myself now, but when upgrading major versions, I make sure to read the release notes and not blindly upgrade

How exactly do you pass it ? This is my package.json script

"scripts": { "test": "istanbul cover node_modules/mocha/bin/_mocha --exit test/Testcases/ " } and it's not working

Other than providing a quick fix (use --exit), I do agree they left the core issue of finding the faulty tests to the user. I'm struggling with that myself now, but when upgrading major versions, I make sure to read the release notes and not blindly upgrade

How exactly do you pass it ? This is my package.json script

"scripts": { "test": "istanbul cover node_modules/mocha/bin/_mocha --exit test/Testcases/ " } and it's not working

Had the same issue and I think I am not the only one. I just had to move --exit at the end.
This did not work:
istanbul cover ./node_modules/mocha/bin/_mocha --exit -- test/.test.js
This did work me:
istanbul cover ./node_modules/mocha/bin/_mocha -- test/
.test.js --exit

exit does nothing when running Mocha programmatically, fwiw. the process is force-exited by the CLI wrapper, not Mocha-the-library.

Was this page helpful?
0 / 5 - 0 ratings