Mongoose: Callbacks Stop Happening

Are you connecting to a standalone, replica set, or sharded cluster? Also, can you confirm you're using SSL?

This has been a huge problem for me the past few days. Every application running ended up with a slower and slower event loop, causing increasing latency throughout the entire system. Restarting the Node applications resolved the issue each time, with it coming back after either minutes or hours of operation.

Reverting to mongoose 4.5.10 resolved the issue.

• Connecting to Replica Set
• Using SSL

Similar issue here with 4.6.0, I'm not sure if I'm connecting to a replica set or not, but definitely using SSL.

FWIW, 4.6.0 was dropping connections constantly. 4.5.10 is good.

@djanowski can you provide any more info, like what your connection code looks like and what version of mongodb you have?

@vkarpov15 I was using MongoDB 3.2.6.

Connection options:

auth: {
  authMechanism: 'SCRAM-SHA-1',
},
mongos: {
  ssl: true,
  sslValidate: true,
  sslCA: [cert],
},

Single mongos or multiple mongos?

Multiple mongos. Using a Compose Mongo deployment.

Anything else I can do to help out troubleshooting this? This is blocking us from moving to newer versions...

Hard for me to make any real hypothesis without seeing your code. If you're connecting with mongoose.connect I'd check the value of mongoose.connection.readyState when this issue starts happening. Also, I'd enable mongoose debug mode, mongoose.set('debug', true), which will print debug messages to the console for every mongoose operation. Whether those messages are printing or not would definitely help debug.

@vkarpov15 This is the code I'm using to connect: https://gist.github.com/sommestad/c0c6a7fa4feaadf84ecbb59cc3432c90
It's part of a private module, hence the Gist.

_(The reason for the connection wrapping module is that the connection is occasionally reused in other private NPM modules, which hasn't worked well with mongoose historically.)_

I had logging of the connection state enabled when these problems occurred, but the connection itself did not seem to change (no events emitted at least). So, it did not seem like the client was disconnected.

It is not connecting with a replicaSet in the URI (the target database is a Compose Deployment). As mentioned, SSL is used. Also, traffic is routed through an AWS NAT Gateway. AWS NAT has a idle timeout of 5 minutes, but these problems happened even during high traffic.

some problem for me , do you solved it ?

@youth7 nope no ideas. Are you also connecting to a compose deployment? Please provide additional info

Anything further I can do to try and find the cause of this? Any theories that can be investigated?

@sommestad nothing concrete. Do you have any logs from compose?

I tried the mongoose 4.7.2 but experienced the same issue after 48 hours. At this point, I'm hamstrung on 4.4.11 which has issues of its own.

I installed 4.7.2 on our demo server hoping to collect logs, but it's been running for two weeks without an issue. Maybe the volume of data/connections/etc provokes the issue?

Once it starts happening, the code to reproduce it is very simple. Basically any query's callback just disappears:

db.Something.find({}, callback); //this callback is never fired

As I mentioned, when this is happening, mongoose.connection.readyState === CONNECTED

Weird. I meant mongodb logs - can you dump the mongodb server logs from when that error happened?

I scanned the DB logs when this failure happened and could not see anything out of the ordinary. There was no issues related to number of connections, disconnects, problems with connecting or similar. I also reached out to Compose who couldn't see anything out of the order either.

@bendytree Can you throw some volume at that demo server, maybe using an HTTP load testing tool?

@bendytree Also, does it happen on all release lines of Node (4.x, 6.x and 7.x)?

@vkarpov15 Can we get 4.7.7 out the door so I can test with the latest MongoDB driver? 4.7.6 is pinned to a version that seems deprecated.

@vkarpov15 66d559b19a86c70e30a8f083d03eb22566571b7e Thanks! I'll give it a try.

Same here, a SSL replica here, after few minutes / hours, the requests just stop to callback() (I've added some more verbose logging) and no errors. I've upgrade to 4.7.7 last night but still have the issue.

EDIT: I tried to downgrade to 4.7.0 but it was worst (having the issue in less than 5 minutes). I'm trying 4.5.10.

@gierschv Node 6.9.x?

@djanowski no, still on 4.6/4.7.x LTS.

@djanowski I've only tried it on Node 4.4.2

Keeping a list of probable duplicates of this issue:

• #4638
• #4690
• #4901

FYI, 4.5.10 seems to work well here, no issues for the last 5 hours. I also had a high CPU usage like #4690 on all the machines that seems also to be fixed by the downgrade.

No new developments here? Anything more we can do to help find the bug?

@sommestad do you still have this problem with the latest mongoose?

I haven't dared do an upgrade yet, since it basically broke our entire system last time (the changes appear after "some" time so it's hard to verify without having it long running). If you have reason to believe it's fixed, we could try on a more isolated part of the system?

Edit: ping @varunjayaraman

On my side, we switched to Node 6.x LTS and upgraded to the last mongoose version a few weeks ago, no issue since.

We upgraded to Node 6.10.2, Mongoose 4.9.9 about two months ago. Still seeing the same issue.

Our connection string has the format mongodb://username:[email protected]:1234,db2.com:2345/db-name?ssl=true.

console.log("fetching user...");
userCollection.findOne({_id: userId}, function(err, user){
  console.log("done"); // this never happens
  ...
});

Once the callbacks stop, they never start again. If we restart the process then it connects fine.

If there's any data we can collect to help troubleshoot, then we'd love to help out.

I'm seeing the same issue and can reproduce it with the code below (sorry for the fast mock-up / code style).

To simulate the connection timeout, I lowered the poolSize to 1 and trigger requests to 2 endpoints (normal / slow) which are called in this order:

1. normal endpoint (which returns results immediately)
2. slow endpoint (which will start a slow query)
3. normal endpoint (which will wait for an available connection since it's in use by 2)
4. normal endpoint (which will trigger after a minute, when 2 and 3 resulted in a MongoError / connection timeout)

Reproduce using:

// Require modules
const mongoose = require('mongoose');
const express = require('express');
const request = require('request');
const parseJson = require('body-parser').json();

// Use native Promises in Mongoose
mongoose.Promise = Promise;

// Create the test schema
const testSchema = new mongoose.Schema({
  name: {
    type: String,
    trim: true,
    default: 'test'
  }
});

// Create the mongoose model
mongoose.model('Test', testSchema, 'test');

// Create some test data
mongoose.model('Test').create([
  {name: 'test1'},
  {name: 'test2'},
  {name: 'test3'},
  {name: 'test4'},
  {name: 'test5'}
]);

// Create the Express based app
const app = express();

// Create the express router
const router = express.Router();

// Create a normal route
router.route('/normal').get(parseJson, function (req, res, next) {
  mongoose.model('Test').find({}).then(
    function (data) {
      res.status(200).json(data);
    }
  ).catch(
    function (err) {
      res.status(400).json(err);
    }
  );
});

// Create up a slow route
router.route('/slow').get(parseJson, function (req, res, next) {
  mongoose.model('Test').find({$where: 'sleep(10000) || true'}).then(
    function (data) {
      res.status(200).json(data);
    }
  ).catch(
    function (err) {
      res.status(400).json(err);
    }
  );
});

// Add middleware to console log every request
var requestNumber = 1;
app.use(function (req, res, next) {
  console.log('Request ' + requestNumber, req.method, req.url);
  requestNumber++;
  next();
});

// Use the router
app.use('/', router);

// Listen for requests
app.listen(4000, function () {
  console.log('Server listening on port 4000');
});

// Catch any uncaught exceptions
process.on('uncaughtException', function (err) {
  console.log('Uncaught exception', err);
});

// Catch any unhandled rejections
process.on('unhandledRejection', function (reason) {
  console.error('Unhandled Rejection', reason);
});

// Database connection options
const connectionOpts = {
  // Use the new connection logic
  useMongoClient: true,

  // Do not auto reconnect (the Node code will auto reconnect)
  autoReconnect: false,

  // Use a poolsize of 1 to simulate a timeout using multiple request to slow and normal endpoints
  poolSize: 1
};

// Connection method with retry
const connectWithRetry = function () {
  // Check if we still need to connect
  if (mongoose.connection.readyState !== mongoose.Connection.STATES.connected) {
    // Connect
    return mongoose.connect('mongodb://localhost:27017/test', connectionOpts).catch(
      function (err) {
        console.log('Can not connect to mongo', err);
      }
    );
  }
};

// Helper function to execute a request to the normal endpoint
const doRequestNormal = function () {
  request({
    url: 'http://localhost:4000/normal',
    json: true,
    timeout: 60000
  }, function (error, response, body) {
    if (error) {
      console.log('Error on normal request', error, response);
    }
    console.log('Performed normal request, body:', body);
  });
};

// Helper function to execute a request to the normal endpoint
const doRequestSlow = function () {
  request({
    url: 'http://localhost:4000/slow',
    json: true,
    timeout: 60000
  }, function (error, response, body) {
    if (error) {
      console.log('Error on slow request', error, response);
    }
    console.log('Performed slow request', body);
  });
};

// Helper function to simulate requests
const doRequests = function () {
  doRequestNormal();
  doRequestSlow();
  doRequestNormal();

  setTimeout(function () {
    console.log('Do normal request after 1 minute');
    doRequestNormal();
  }, 60000);
};

// Connecting event
mongoose.connection.on('connecting', function () {
  console.log('Connecting to database...');
});

// Connected event
mongoose.connection.on('connected', function () {
  console.log('Database is connected, start requesting...');
  doRequests();
});

// Timeout event
mongoose.connection.on('timeout', function (err) {
  console.log('Database timeout error', err);
});

// Error event
mongoose.connection.on('error', function (err) {
  console.log('Database connection error', err);
});

// Disconnected event
mongoose.connection.on('disconnected', function () {
  console.error('Database got disconnected, reconnecting in 5 sec...');

  // Reconnect in 5 seconds
  setTimeout(connectWithRetry, 5000);
});

// Connect to the database with retry
connectWithRetry();

// Log the connection state every 10 seconds
setInterval(function () {
  console.log('Mongoose connection readystate is', mongoose.connection.readyState);
}, 10000);

Console output is as follows:

Connecting to database...
Server listening on port 4000
Database is connected, start requesting...
Request 1 GET /normal
Request 2 GET /slow
Request 3 GET /normal
Performed normal request, body: [ { _id: '597f8c6f41bf2e119594ba1a', __v: 0, name: 'test1' },
  { _id: '597f8c6f41bf2e119594ba1b', __v: 0, name: 'test2' },
  { _id: '597f8c6f41bf2e119594ba1c', __v: 0, name: 'test3' },
  { _id: '597f8c6f41bf2e119594ba1d', __v: 0, name: 'test4' },
  { _id: '597f8c6f41bf2e119594ba1e', __v: 0, name: 'test5' } ]
Mongoose connection readystate is 1
Mongoose connection readystate is 1
Mongoose connection readystate is 1
Performed slow request { name: 'MongoError',
  message: 'connection 0 to localhost:27017 timed out' }
Performed normal request, body: { name: 'MongoError',
  message: 'connection 0 to localhost:27017 timed out' }
Mongoose connection readystate is 1
Mongoose connection readystate is 1
Mongoose connection readystate is 1
Do normal request after 1 minute
Request 4 GET /normal
Mongoose connection readystate is 1
Mongoose connection readystate is 1
Mongoose connection readystate is 1
Mongoose connection readystate is 1
Mongoose connection readystate is 1
Mongoose connection readystate is 1
Error on normal request { Error: ESOCKETTIMEDOUT
    at ClientRequest.<anonymous> (/Users/adriaanmeuris/Documents/Projecten/Mongoose timeout/node_modules/request/request.js:819:19)
    at ClientRequest.g (events.js:291:16)
    at emitNone (events.js:86:13)
    at ClientRequest.emit (events.js:185:7)
    at Socket.emitTimeout (_http_client.js:620:10)
    at Socket.g (events.js:291:16)
    at emitNone (events.js:86:13)
    at Socket.emit (events.js:185:7)
    at Socket._onTimeout (net.js:339:8)
    at ontimeout (timers.js:365:14)
    at tryOnTimeout (timers.js:237:5)
    at Timer.listOnTimeout (timers.js:207:5) code: 'ESOCKETTIMEDOUT', connect: false } undefined
Performed normal request, body: undefined
Mongoose connection readystate is 1
Mongoose connection readystate is 1

I monitor the readyState of the database every 10 seconds, which indicates the connection is still active. No Mongoose error / timeout event triggers, and the callback of request 4 is never called (it times out after 60s using the request package). You can also test by visiting http://localhost:4000/slow and next http://localhost:4000/normal.

I'm using auto reconnection logic based on https://team.goodeggs.com/reconnecting-to-mongodb-when-mongoose-connect-fails-at-startup-83ca8496ca02, but this is never triggered as mongo does not disconnect.

One some occasions, I'm not able to repro when I comment the code that creates some test data (so basically querying an empty collection)

Possible related issues:

• #4789
• #4660

Any help would me greatly appreciated.

So I did some more digging an re-wrote the script above to work with node-mongodb-native without using mongoose. Results are very different now:

Connection settings and requests are the same:

1. normal endpoint (which returns results immediately)
2. slow endpoint (which will start a slow query)
3. normal endpoint (which will wait for an available connection since it's in use by 2)
4. normal endpoint (which will trigger after a minute)

Code:

// Require modules
const MongoClient = require('mongodb').MongoClient;
const Server = require('mongodb').Server;
const express = require('express');
const request = require('request');
const parseJson = require('body-parser').json();

// The db reference
var db;

// Function to insert test data
function insertTestData() {
  return new Promise(function (resolve, reject) {
    // Get the test collection
    var test = db.collection('test');

    // Insert some documents
    test.insertMany([
      {name: 'test1'},
      {name: 'test2'},
      {name: 'test3'},
      {name: 'test4'},
      {name: 'test5'}
    ], function (err, result) {
      if (err) {
        reject(err);
      } else {
        resolve();
      }
    });
  });
}

// Create the Express based app
const app = express();

// Create the express router
const router = express.Router();

// Create a normal route
router.route('/normal').get(parseJson, function (req, res, next) {
  // Get the documents collection
  var collection = db.collection('test');

  // Find some documents
  collection.find({}).toArray(function (err, data) {
    if (err) {
      res.status(400).json(err);
    } else {
      res.status(200).json(data);
    }
  });
});

// Create up a slow route
router.route('/slow').get(parseJson, function (req, res, next) {
  // Get the documents collection
  var collection = db.collection('test');

  // Find some documents
  collection.find({$where: 'sleep(10000) || true'}).toArray(function (err, data) {
    if (err) {
      res.status(400).json(err);
    } else {
      res.status(200).json(data);
    }
  });
});

// Add middleware to console log every request
var requestNumber = 1;
app.use(function (req, res, next) {
  console.log('Request ' + requestNumber, req.method, req.url);
  requestNumber++;
  next();
});

// Use the router
app.use('/', router);

// Listen for requests
app.listen(4000, function () {
  console.log('Server listening on port 4000');
});

// Catch any uncaught exceptions
process.on('uncaughtException', function (err) {
  console.log('Uncaught exception', err);
});

// Catch any unhandled rejections
process.on('unhandledRejection', function (reason) {
  console.error('Unhandled Rejection', reason);
});

// Database connection options
const connectionOpts = {
  // Do not auto reconnect (the Node code will auto reconnect)
  autoReconnect: false,

  // Use a poolsize of 1 to simulate a timeout using multiple request to slow and normal endpoints
  poolSize: 1
};

// Connection method with retry
const connectWithRetry = function () {

  return MongoClient.connect('mongodb://localhost:27017/test', connectionOpts).then(
    function (database) {
      db = database;
      setupEvents();
      console.log('Connected to mongo');
    }
  ).catch(
    function (err) {
      console.log('Can not connect to mongo', err);
      return err;
    }
  );
};

// Helper function to execute a request to the normal endpoint
const doRequestNormal = function () {
  request({
    url: 'http://localhost:4000/normal',
    json: true,
    timeout: 60000
  }, function (error, response, body) {
    if (error) {
      console.log('Error on normal request', error, response);
    }
    console.log('Performed normal request, body:', body);
  });
};

// Helper function to execute a request to the normal endpoint
const doRequestSlow = function () {
  request({
    url: 'http://localhost:4000/slow',
    json: true,
    timeout: 60000
  }, function (error, response, body) {
    if (error) {
      console.log('Error on slow request', error, response);
    }
    console.log('Performed slow request', body);
  });
};

// Helper function to simulate requests
const doRequests = function () {
  doRequestNormal();
  doRequestSlow();
  doRequestNormal();

  setTimeout(function () {
    console.log('Do normal request after 1 minute');
    doRequestNormal();
  }, 60000);
};

// Helper function to setup mongo events
function setupEvents() {
// Connecting event
  db.on('connecting', function () {
    console.log('Connecting to database...');
  });

// Connected event
  db.on('connected', function () {
    console.log('Database is connected, start requesting...');
  });

// Timeout event
  db.on('timeout', function (err) {
    console.log('Database timeout error', err);
  });

// Error event
  db.on('error', function (err) {
    console.log('Database connection error', err);
  });

// Disconnected event
  db.on('close', function () {
    console.error('Database got disconnected, reconnecting in 5 sec...');

    // Reconnect in 5 seconds
    setTimeout(connectWithRetry, 5000);
  });
}

// Connect to the database with retry
connectWithRetry().then(
  function () {
    return insertTestData();
  }
).then(
  function () {
    doRequests();
  }
);

Console output is as follows:

Server listening on port 4000
Connected to mongo
Request 1 GET /normal
Request 2 GET /slow
Request 3 GET /normal
Performed normal request, body: [ { _id: '598207c16caf9224cf3b8897', name: 'test1' },
  { _id: '598207c16caf9224cf3b8898', name: 'test2' },
  { _id: '598207c16caf9224cf3b8899', name: 'test3' },
  { _id: '598207c16caf9224cf3b889a', name: 'test4' },
  { _id: '598207c16caf9224cf3b889b', name: 'test5' } ]
Performed slow request [ { _id: '598207c16caf9224cf3b8897', name: 'test1' },
  { _id: '598207c16caf9224cf3b8898', name: 'test2' },
  { _id: '598207c16caf9224cf3b8899', name: 'test3' },
  { _id: '598207c16caf9224cf3b889a', name: 'test4' },
  { _id: '598207c16caf9224cf3b889b', name: 'test5' } ]
Performed normal request, body: [ { _id: '598207c16caf9224cf3b8897', name: 'test1' },
  { _id: '598207c16caf9224cf3b8898', name: 'test2' },
  { _id: '598207c16caf9224cf3b8899', name: 'test3' },
  { _id: '598207c16caf9224cf3b889a', name: 'test4' },
  { _id: '598207c16caf9224cf3b889b', name: 'test5' } ]
Do normal request after 1 minute
Request 4 GET /normal
Performed normal request, body: [ { _id: '598207c16caf9224cf3b8897', name: 'test1' },
  { _id: '598207c16caf9224cf3b8898', name: 'test2' },
  { _id: '598207c16caf9224cf3b8899', name: 'test3' },
  { _id: '598207c16caf9224cf3b889a', name: 'test4' },
  { _id: '598207c16caf9224cf3b889b', name: 'test5' } ]

No Mongo error / timeout events trigger like the previous script, but request 2 and 3 resolve correctly, and the callback of request 4 is triggered as well - in contrast to the script which is using Mongoose.

I'm trying to rule out whether this is a Mongoose vs. a Mongo issue - so any help regarding comparing the results of these scripts would be appreciated. Maybe @vkarpov15 can chime in on this one? Thanks!

@adriaanmeuris that's strange, here's the output I get from running your first script:

$ node gh-4513.js 
Connecting to database...
Server listening on port 4000
Database is connected, start requesting...
Request 1 GET /normal
Request 2 GET /slow
Request 3 GET /normal
Performed normal request, body: [ { _id: '5998e4a79bb81d37ac4b1a26', __v: 0, name: 'test1' },
  { _id: '5998e4a79bb81d37ac4b1a27', __v: 0, name: 'test2' },
  { _id: '5998e4a79bb81d37ac4b1a28', __v: 0, name: 'test3' },
  { _id: '5998e4a79bb81d37ac4b1a29', __v: 0, name: 'test4' },
  { _id: '5998e4a79bb81d37ac4b1a2a', __v: 0, name: 'test5' } ]
Mongoose connection readystate is 1
Mongoose connection readystate is 1
Mongoose connection readystate is 1
Mongoose connection readystate is 1
Mongoose connection readystate is 1
Performed slow request [ { _id: '5998e4a79bb81d37ac4b1a26', __v: 0, name: 'test1' },
  { _id: '5998e4a79bb81d37ac4b1a27', __v: 0, name: 'test2' },
  { _id: '5998e4a79bb81d37ac4b1a28', __v: 0, name: 'test3' },
  { _id: '5998e4a79bb81d37ac4b1a29', __v: 0, name: 'test4' },
  { _id: '5998e4a79bb81d37ac4b1a2a', __v: 0, name: 'test5' } ]
Performed normal request, body: [ { _id: '5998e4a79bb81d37ac4b1a26', __v: 0, name: 'test1' },
  { _id: '5998e4a79bb81d37ac4b1a27', __v: 0, name: 'test2' },
  { _id: '5998e4a79bb81d37ac4b1a28', __v: 0, name: 'test3' },
  { _id: '5998e4a79bb81d37ac4b1a29', __v: 0, name: 'test4' },
  { _id: '5998e4a79bb81d37ac4b1a2a', __v: 0, name: 'test5' } ]
Mongoose connection readystate is 1
Do normal request after 1 minute
Request 4 GET /normal
Performed normal request, body: [ { _id: '5998e4a79bb81d37ac4b1a26', __v: 0, name: 'test1' },
  { _id: '5998e4a79bb81d37ac4b1a27', __v: 0, name: 'test2' },
  { _id: '5998e4a79bb81d37ac4b1a28', __v: 0, name: 'test3' },
  { _id: '5998e4a79bb81d37ac4b1a29', __v: 0, name: 'test4' },
  { _id: '5998e4a79bb81d37ac4b1a2a', __v: 0, name: 'test5' } ]
Mongoose connection readystate is 1
Mongoose connection readystate is 1
^C
$ 

Mongoose stays connected. Can you clarify your versions of mongodb, node, and mongoose?

thanks for testing! Indeed Mongoose always stays connected, but callbacks stop happening (you can see the request coming into Express, but they'll timeout)

I was testing with:

• node 6.8.1
• mongodb 2.2.30
• mongoose 4.11.5

I've updated to:

• node 6.8.1
• mongodb 2.2.31
• mongoose 4.11.7

I can still reproduce the issue locally with the first script I posted. It only occurs when a Mongo connection times out, so you might want to change the query that simulates this:

mongoose.model('Test').find({$where: 'sleep(10000) || true'})

(I'm not sure if this will trigger a timeout on every machine so a change to this query might be necessary to reproduce)

We're no longer seeing these issues after updating to mongoose 4.10.4.
Seems a bit odd, since others seem to experience issues, but have had many services running for a couple of weeks without any signs of the problem coming back. 🤔

However, we did see issues in the same way as @adriaanmeuris previously, so could be that we just haven't experienced any timeouts .

Running Node 8.

We are still seeing this issue periodically.

I couldn't reproduce it with @adriaanmeuris's script, but here is a modified version that reproduces it every time for me:

bendytree/mongoose-issue-4513

Thanks @bendytree, with your script I can reproduce as well:

7) Database connecting...
49) Database connected...
49) Dropping...
57) Creating...
156) FastQuery-A: starting...
166) FastQuery-A: success: 1
1158) SlowQuery: starting...
2158) FastQuery-B: starting...
362165) SlowQuery: failed: MongoError: connection 0 to localhost:27017 timed out
362167) FastQuery-B: failed: MongoError: connection 0 to localhost:27017 timed out
605159) FastQuery-C: starting...
1210149) Giving Up...

It's essentially the same script as mine but with a longer timeout duration. I guess simulating a timeout locally takes more/less time depending on your setup, since not everyone could repro with my script (which uses a 10s sleep).

@vkarpov15 can you try this updated script. I've tested & reproduced with mongoose 4.8.11.

I managed to repro with @bendytree , this is a case of socketTimeoutMS too low. The

362591) SlowQuery: failed: MongoError: connection 0 to localhost:27017 timed out
362591) FastQuery-B: failed: MongoError: connection 0 to localhost:27017 timed out

messages are indicative of the mongodb driver's socket timing out. The mongodb driver times out after 30 seconds by default, but your OS also has a socket timeout that you need to be aware of, so it's generally a good idea to avoid queries that take more than 15 seconds. However, you can configure mongodb's socket timeout using the socketTimeoutMS option:

mongoose.connect(DB_URL, {
  useMongoClient: true,
  autoReconnect: false,
  poolSize: 1,
  socketTimeoutMS: 0
}).catch((err)=>{ log('Connect failed', err); });

With that option set to 0, your script almost succeeds on my machine, the 10 minute query still dies but callbacks continue

$ node gh-4513.js 
9) Database connecting...
43) Database connected...
43) Dropping...
53) Creating...
177) FastQuery-A: starting...
189) FastQuery-A: success: 1
1174) SlowQuery: starting...
2174) FastQuery-B: starting...
601181) SlowQuery: failed: MongoError: Interrupted by the host
601182) FastQuery-B: success: 1
605173) FastQuery-C: starting...
605174) FastQuery-C: success: 1

But the subsequent query succeeds.

Either way, mongoose should emit a timeout event in this case, so we'll fix that issue.

I think you're confusing connectTimeoutMS (with a default of 30000ms) with socketTimeoutMS (with a default of 360000ms).

It's advised to set socketTimeoutMS to two to three times the length of the slowest operation which runs through the driver, setting socketTimeoutMS to 0 actually means apply the operating system default socket timeout value (more info at http://mongodb.github.io/node-mongodb-native/2.2/reference/faq/).

So this question remains: why do these callbacks stop happening when this limit (fixed or default OS timeout) is hit?

@vkarpov15 with [email protected], I now see the timeout event, thanks for that. However you still see this callback issue (using @bendytree's original script, for faster testing you can set LONG_QUERY_DURATION_IN_MS to 2000 and socketTimeoutMS to 1000)

I've did some more testing and found out this can be reproduced without Mongoose as well. When reading http://mongodb.github.io/node-mongodb-native/2.2/reference/faq/, I found out that: Closing the socket forces a reconnect of the driver’s connection pool and introduces latency to any other operations which are queued up

Even though mongoose.connection.readyState is always 1, I tried setting autoReconnect to true, an re-ran the script:

8) Database connecting...
47) Database connected...
48) Dropping...
73) Creating...
123) FastQuery-A: starting...
129) FastQuery-A: success: 1
1122) SlowQuery: starting...
2126) FastQuery-B: starting...
3130) Database timeout...
3133) SlowQuery: failed: MongoError: connection 0 to localhost:27017 timed out
3133) FastQuery-B: failed: MongoError: connection 0 to localhost:27017 timed out
6063) Database timeout...
7122) FastQuery-C: starting...
7125) FastQuery-C: success: 1
8129) Database timeout...
11070) Database timeout...
13073) Database timeout...

So it looks like callbacks are working with autoReconnect set to true 👍

Upon further reading this article: http://mongodb.github.io/node-mongodb-native/2.0/tutorials/connection_failures/, I realized that the reason why the callbacks don't happen after the timeout, is because the operation is buffered while waiting for server to reconnect.

So i tried setting autoReconnect back to false+ set bufferMaxEntries to 0, this is the output:

8) Database connecting...
51) Database connected...
51) Dropping...
58) Creating...
112) FastQuery-A: starting...
116) FastQuery-A: success: 1
1115) SlowQuery: starting...
2115) FastQuery-B: starting...
3123) Database timeout...
3123) SlowQuery: failed: MongoError: connection 0 to localhost:27017 timed out
3123) FastQuery-B: failed: MongoError: connection 0 to localhost:27017 timed out
7115) FastQuery-C: starting...
7117) FastQuery-C: failed: MongoError: no connection available for operation and number of stored operation > 0

Which explains why the callback did not trigger in the first version of @bendytree's script.

@vkarpov15 This now leaves us with 2 questions:

• why are there no connection events emitted (I would expect to see a disconnected event)
• the timeout event event seems to keep emitting

here's the updated script (based on @bendytree's script, tested with [email protected] and [email protected]):

const LONG_QUERY_DURATION_IN_MS = 2000;
const DB_URL = 'mongodb://localhost:27017/test_mongoose_callbacks';

const mongoose = require('mongoose');
mongoose.Promise = Promise;

var startTime = new Date().getTime();
var log = (msg) => {
  var seconds = Math.round(new Date().getTime() - startTime);
  console.log(seconds+") "+msg);
};

var Model = mongoose.model('Test', new mongoose.Schema({
  name: { type: String, trim: true, default: 'test' }
}), 'test');

mongoose.connection.on('connecting',   () => { log('Database connecting...'); });
mongoose.connection.on('timeout',      () => { log('Database timeout...'); });
mongoose.connection.on('error',        () => { log('Database error...'); });
mongoose.connection.on('disconnected', () => { log('Database disconnected...'); });

const doRequest = function (title, slow) {
  var logRequest = (msg) => { log(title+": "+msg); };

  logRequest("starting...");
  var filter = slow ? {$where: 'sleep('+LONG_QUERY_DURATION_IN_MS+') || true'} : {};
  Model.count(filter).exec((err, count) => {
    logRequest(err ? "failed: "+err : "success: "+count);
  });
};

mongoose.connection.on('connected', () => {
  log("Database connected...");
 log("Dropping...");
 mongoose.connection.db.dropDatabase(function(){
   log("Creating...");
   Model.create([ {name: 'test1'} ], function(){
      setTimeout(() => { doRequest("FastQuery-A", false); }, 0);
      setTimeout(() => { doRequest("SlowQuery", true);  }, 1000);
      setTimeout(() => { doRequest("FastQuery-B", false); }, 2000);
      setTimeout(() => { doRequest("FastQuery-C", false); }, LONG_QUERY_DURATION_IN_MS+5000);
      setTimeout(() => { log("Giving Up..."); }, LONG_QUERY_DURATION_IN_MS+30000);
    });
  });
});

mongoose.connect(DB_URL, {
  useMongoClient: true,
  autoReconnect: true,
  //bufferMaxEntries: 0, // Uncomment to disable buffering operations 
  poolSize: 1,
  socketTimeoutMS: 1000
}).catch((err)=>{ log('Connect failed', err); });

setInterval(()=>{
  if (mongoose.connection.readyState === 1) return;
  log('Mongoose.readystate = ', mongoose.connection.readyState);
}, 1000);

Yeah you're right we need to do better about this. The timeout event is necessary, but not really sufficient to handle this case.

The reason why 'disconnected' is not emitted is that 'timeout' means one socket timed out, not the entire pool. Mongoose would need to know how many sockets in the pool are still alive when the timeout event occurred, and I'd need to investigate more to figure out how we do that.

Will investigate more why the timeout event keeps re-emitting. Thanks for your patience :+1:

I tracked this down to a couple bugs in the mongodb driver: https://github.com/mongodb-js/mongodb-core/pull/215 . Going to have to wait on that to get merged and deployed before we can patch it in to mongoose.

@vkarpov15 in my own testing i am experiencing mongoose.connection.readyState never changes. is this the same issue?

@r3wt that issue should be fixed in 4.11.13, which version are you using?

@vkarpov15 I was using 4.11.11 at the time of my post. its disheartening that such obvious bad behavior can sneak into production for such a notable library. perhaps some better testing procedures are needed. (I am not trying to be disrespectful, i know how hard maintaining open source projects is. sometimes it sucks)

@vkarpov15 i am developing a mongoose plugin. it is necessary for the plugin to block until the connection is ready, otherwise it deadlocks waiting for connection(for reasons). the code i am using to await the connection to become ready is the following:

function ready(){
    console.log('ready()');
    return new Promise((resolve,reject)=>{
        var _int = setInterval(()=>{
            if(mongoose.connection.readyState == 1 || forceReady){
                clearInterval(_int);
                console.log('mongoose connected ready().resolve()')
                resolve();
            }
        },200);
    })
}

i also tried listening via mongoose.connnection.on('connected') but alas that didn't work either. on a blank installation it worked fine. but in conjunction with other mongoose plugins and stuff listening for the events, it is where the problem occurs. connection.readyState remains 0 forever and the connected event is never emitted.

it seems that require('mongoose') !== require('mongoose') in my opinion. i hope this make sense.

ie:

1. user script require('mongoose')
2. plugin a require('mongoose')
3. plugin b require('mongoose')

the user script and the plugin receive the info about the connection. but plugin b never does. does it make sense?

@r3wt indeed, it is definitely disheartening. We've since added tests that actually stop/start a mongodb server and assert mongoose does the right thing to prevent this from happening in the future. We're sorry for the trouble.

As for blocking until the connection is ready, mongoose.connection.on('connected') should work. What other plugins are you using? Try running npm list | grep "mongoose", your assertion that require('mongoose') !== require('mongoose') makes me think that you have multiple versions of mongoose somewhere.

Unfortunately gonna have to push this one back again because mongodb driver is held up from releasing. Sorry about that and thanks for your patience :+1: 🌴

@vkarpov15 Just wanted to say thanks so much for your work on this fix. We've been using it in production for a few months now with 0 issues.

@bendytree thanks for your kind words and your patience in getting this issue repro-ed. The mongodb driver team's been swamped working on releasing 3.0 in time for the mongodb 3.6 release, so they haven't been able to put out a 2.x release with this fix yet, I'll ping them again. Happy holidays!