Sentry-javascript: @sentry/node memory leaks on node 10.13.0

@abraxxas can you help me reproduce this issue? How exactly are you triggering this memleak?
I tried your description with no luck, it just produces stats events, never leak one.

@kamilogorek Thanks for your response. What i did was the following. I checked out this example https://github.com/sheerun/next.js/tree/with-sentry-fix/examples/with-sentry and added memwatch to the server.js

if (!dev) {
  memwatch.on("leak", info => {
    console.log("memwatch::leak");
    console.error(info);
  });

  memwatch.on("stats", stats => {
    console.log("memwatch::stats");
    console.error(Util.inspect(stats, true, null));
  });
}

then i ran the example using node 10.x (with 8.x i observed no memory issues) and requested the following ressources using our gatling testsuite:

    "/",
    "/_next/static/r1zovjaZ1TujaA0hJEp91/pages/_app.js",
    "/_next/static/r1zovjaZ1TujaA0hJEp91/pages/_error.js",
    "/_next/static/r1zovjaZ1TujaA0hJEp91/pages/index.js",
    "/_next/static/runtime/main-1eaa6d1d0c8e7d048efd.js",
    "/_next/static/chunks/commons.b34d260fee0c4a698139.js",
    "/_next/static/runtime/webpack-42652fa8b82c329c0559.js"

(note the hashes might change for you)

but you should be able to achieve the same results with this approach very simple https://www.simonholywell.com/post/2015/06/parallel-benchmark-many-urls-with-apachebench/

after a few thousand requests our memory usage was at almost 1GB and never reduced even when idling. As soon as i remove the request and errorHandler from server.js the memory-leak stops. So it seems to be connected to those 2. Maybe you either had too few requests or used node 8.x?

@abraxxas confirmed. Node 10 + ~300req for each resource does the job. Will investigate further. Thanks!

@kamilogorek intersting, but if you look on the heapsize in your
reproduction you will see that it stays around 20mb without the handlers
but increases rapidly with them.

I think the memory-leak warning without the handlers happens just because
due to the requests there is some constsnt memory increase.

There is still a vast difference in memory usage between the version with
sentry and without.

On Thu, 29 Nov 2018, 12:45 Kamil Ogórek <[email protected] wrote:

@abraxxas https://github.com/abraxxas I successfully reproduced it,
however, it appears that server still leaks request objects on its own,
even without Sentry handlers.

https://streamable.com/bad9j

The growth rate is a tad larger, as we do attach domain and our own scope
object to the request, but it'd be utilized alongside the request by GC.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/getsentry/sentry-javascript/issues/1762#issuecomment-442804709,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AIbrNlgPjPd5Jra1aahR-Dthf7XvbCexks5uz8jjgaJpZM4YvOA2
.

@abraxxas ignore my previous comment (the one I removed), it's totally on our side :)

It appears to be the issue in Node's core itself. Please refer to this comment https://github.com/getsentry/sentry-javascript/issues/1762#issuecomment-444126990

@kamilogorek Have you had chance to look into this yet? It is causing a huge memory leak for us. After looking at the heap, this line looks like it could be the root of problem:

https://github.com/getsentry/sentry-javascript/blob/c27e1e32d88cc03c8474fcb1e12d5c9a2055a150/packages/node/src/handlers.ts#L233

The inspector showed thousands of entries into the eventProcessors list

I don't have any context around how things are architected, but we have noticed that requests are not correctly scoped and are giving the wrong metadata (see #1773) so it appears everything is being managed in global state and not cleaned up when a request ends

@abraxxas @tpbowden there's an issue with leaking domain module in Node's core itself. We'll keep monitoring it and try to come up with a temporary solution before it's fixed in the core. Related issue: https://github.com/nodejs/node/issues/23862

@kamilogorek do you have any ideas for a workaround or temporary fix for this? Progress on the node issue looks quite slow

We're currently using PM2 to restart Node.js processes when memory reaches a certain threshold. https://pm2.io/doc/en/runtime/features/memory-limit/#max-memory-threshold-auto-reload

Using lab for unit testing. The leaks are still present. I know leaks can be painful to debug. Is there an ETA for a fix?

1 tests complete
Test duration: 1832 ms
The following leaks were detected:__extends, __assign, __rest, __decorate, __param, __metadata, __awaiter, __generator, __exportStar, __values, __read, __spread, __await, __asyncGenerator, __asyncDelegator, __asyncValues, __makeTemplateObject, __importStar, __importDefault

npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! [email protected] test: lab build/test
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the [email protected] test script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR! /Users/sunknudsen/.npm/_logs/2019-02-13T14_41_28_595Z-debug.log

@sunknudsen The issue is actually fixed in NodeJS, see https://github.com/nodejs/node/issues/23862 and https://github.com/nodejs/node/pull/25993. We probably need to wait for a release.

@garthenweb Does this affect @sentry/node because of how the package was developed? One of the projects that I am developing on hapi (and relies on many other dependencies) doesn’t produce leaks (at least they are not caught by lab).

@sunknudsen More or less. It's the combination of the (deprecated) domain package and promises as far as I understood. See https://github.com/getsentry/sentry-javascript/blob/master/packages/node/src/handlers.ts#L233

In my case, I just removed the sentry middlewares from my (express) server to fix it.

This isn't an issue in node but in Sentry as disabling the Sentry handlers fixes the issue;

@MartijnHols We are currently working on a major release that should significantly reduce the memory footprint our SDK. If you feel adventurous you could try it out https://github.com/getsentry/sentry-javascript/pull/1919

@HazAT Thanks, I installed it in production last night (at 23:10 in the graph) and re-enabled the handlers with the results below. There's normally a slight spike in CPU usage around 23:00-24:00 (as you can see from the previous day), but this seemed higher. The standard CPU usage is also a lot more spiky than without the handlers. Not sure if this is due to the changes in the new version or it's the handlers being enabled. I'll try disabling the handlers again in a few hours. There are about 2.5 errors caught per hour.

@MartijnHols Thanks for trying it out!

Two things to keep in mind, the memory leak fix for domains in node landed in node only recently in 11.10.
Also, we had to unpublish 5.0.0-beta1 because it was mistakenly tagged as latest, 5.0.0-rc.1 is now the latest next version.
Please try 5.0.0-rc.1 we made a small change how events are queued that should improve load/memory by a lot.

Updating to node 11.12 seems to have stabilized the memory and CPU usage. It seems there isn't any discernible difference now in resource usage when comparing it to having the handlers disabled, maybe even slightly better. It seems to also catch errors fine with all the info I need (it might have more console "breadcrumbs" which is nice). Not sure what else I can check for 5.0.0. I'll let you know if I run into any issues, but probably not.

LGTM. Thanks!

I'd be happy to give this a try as well. @HazAT do you know if the fix in 11.10 is already backported to the active LTS release 10.x?

@adriaanmeuris I've read that someone asked if it will be backported to 10.x, not sure if they will do it.
ref: https://github.com/nodejs/node/pull/25993#issuecomment-463957701

I solved the issue creating the Client and Scope manually in a express middleware

I created a file services/sentry that export a function

import {
  NodeClient as SentryClient, Hub, Integrations, Scope,
} from '@sentry/node';
import config from 'config';

const sentryClient = new SentryClient({
  ...config.sentry,
  frameContextLines: 0,
  integrations: [new Integrations.RewriteFrames()],
});

export default () => {
  const scope = new Scope();
  const client = new Hub(sentryClient, scope);
  return Object.freeze({ client, scope });
};

and in a middleware I save the sentry client/scope on the request object like this

app.use((req, res, next) => {
  req.sentry = getSentry();
  req.sentry.scope.setTag('requestId', req.requestId);
  req.sentry.scope.setExtra('More info', 'XXXXXX');
  next();
});
....
// and use the express error handler
app.use((err, req, res, next) => {
  const code = err.code || 500;
  res.status(code).json({
    code,
    error: err.message,
  });
  if (code >= 500) {
    req.sentry.client.captureException(err);
  }
  next();
});

With this seem that the memory leak its fixed

@couds This implementation is really nice, there is one thing you have to consider tho.

For every request you will get a new client and we do not catch any global errors / automatic breadcrumbs (or whatever else the default integrations do).

@HazAT Thanks, I know that, but is a small price to pay. to solve that huge memory leak, but to minimize this lost I did a few things.

1. I try handle all exceptions I preffer to not rely on the unCoughException/promise events.
2. For breadcrumbs I manually add them as long I have access to the req object..
3. Upload the sourcemaps to sentry with @sentry/webpack-plugin
4. Add Tags/Extra on every request that will help to identify the issue

One more thing, I forgot to paste one more tag I'm adding that is the transaction (to simulate the default Handler)

req.sentry.scope.setTag('transaction', `${req.method}|${req.route ? req.route.path : req.path}`);

I hope this help someone, Sentry its a really great tool and I did what I could to avoid removing it from my stack.

@HazAT @kamilogorek we're still seeing the huge memory growth on [email protected] and [email protected] - are you certain that this nodejs patch fixed it?

@tpbowden Can you please provide a repro OR at least say how your code looks like?
Also, are you using any other plugins related to Sentry?

@HazAT I've tried to reproduce but it's being caused by a fairly complex server with a lot of middleware (server side rendering React). With Sentry middleware installed, we are hitting huge memory growth (under heavy load it can increase by ~10MB / second). When we remove Sentry.Handlers.requestHandler() middleware from our server the memory is constant at ~200MB

We aren't using any plugins, only @sentry/node. Can you think of anything I could try to help reproduce this?

@HazAT Looks like it's related to bundling Sentry using webpack on the server - only happens when the app has been built by Webpack. Adding @sentry/node to externals has fixed our memory problem. Any idea why this happens?

The leak is still reproducible on Node 11.14.0 using @sentry/node 5.1.0

For us, the leak is caused by the interplay between @sentry/node and i18next-express-middleware. Our express app looks similar to https://github.com/i18next/react-i18next/blob/master/example/razzle-ssr/src/server.js.

If we put .use(Sentry.Handlers.requestHandler()); above .use(i18nextMiddleware.handle(i18n)) then we get a memory leak. If we put sentry below it, then we don't get a leak.

We have the same problem. I tried it with node 10.15.3 and 11.14.0. Here is minimal repository to reproduce the problem https://github.com/michalkvasnicak/sentry-memory-leak-reproduction. Just run yarn test or yarn test:watch which will report heap usage. It always increases.

yarn test

yarn test:watch

@michalkvasnicak Thanks for taking the time to create an example, I have a few questions tho.

You are not really testing anything regarding the SDK

const Sentry = require('@sentry/node');

it('works', () => {
  expect(true).toBe(true);
});

You require the package but that's it.
I am not sure since I have no experience yet with jest running leak tests but what can leak by just requiring the package?
Not sure, maybe I am missing something but I would expect the memory to grow when loading a new package.

In regards to what can leak, we do create some global variables but we need them to track the state.

@HazAT yes it is strange but it is enough to make tests leak, jest will fail with leak detected. We have the same problem in our code base where just commenting the import of @sentry/node solved the problem with leak.

@michalkvasnicak I investigated this and it's not directly caused by Sentry.

Our @sentry/node transport has a dependency on https-proxy-agent, which has a dependency on agent-base which requires it's patch-core.js file _and_ this is what creates a leak :shrug:

https://github.com/TooTallNate/node-agent-base/issues/22

You can easily verify this by copying its content into your test and running it with a heap stats:

const url = require("url");
const https = require("https");

https.request = (function(request) {
  return function(_options, cb) {
    let options;
    if (typeof _options === "string") {
      options = url.parse(_options);
    } else {
      options = Object.assign({}, _options);
    }
    if (null == options.port) {
      options.port = 443;
    }
    options.secureEndpoint = true;
    return request.call(https, options, cb);
  };
})(https.request);

https.get = function(options, cb) {
  const req = https.request(options, cb);
  req.end();
  return req;
};

it("works", () => {
  expect(true).toBe(true);
});

We'll have to probably fork it or write a workaround.

any workaround for this one?

https://nodejs.org/en/blog/release/v10.16.0/

fixed some memory leaks, can someone test it?

I have the same problem on node 11.10, but it seems to be fixed on Node 12.3.1

There's a PR open for the agent-base issue: https://github.com/TooTallNate/node-agent-base/pull/25

We can either fork this and override the dependency in Yarn resolutions or find a way to get it merged.

Might be bit offtopic, but could also cause some leaks that there is no domain cleanup in Handlers.ts, only domain.create?
Medium article or SO suggests there should be removeListeners and domain.exit. But did not find some definitive answer to that.

UPDATE: Now I see that handlers are using domain.run which internally calls domain.exit. Still removing the listeners/emitters might make difference, but honestly have no idea.

I upgraded to Node 12.4.0 and I'm still seeing bad behavior that seems to be tied to Sentry.

Here's some node-clinic doctor runs, done with the --autocannon option over 90 seconds.

• Without any sentry initialization
• With init but without express request handlers
• With both handlers and init (our current production setup)

It only seems really leaky when the request handlers are in place. If you look at the bottoms of the GC troughs in the run without the handlers, they're all at about the same level (65-70mb), where the run with the handlers seems to be climbing about 5mb with each GC cycle.

@tstirrat15 this fix has not been released yet, so that's probably why you are still having this issue. Can you try from the latest master if that's an option?

@tstirrat15 5.4.2 with the fix included has been released, give it a try :)

Here's another run with v5.4.2 in place. It still seems a bit leaky...

GC is always kicking in correctly and restores memory to the baseline. Memory usage increase is caused by breadcrumbs collected from the events and event queue, but it'll stop at 100 breadcrumbs and won't increase further. It'd be nice if we could see like ~15-30min dump and see whether peak memory stops at some point.

Hmm... sounds good. I'll get this PR through to production and see if the behavior changes. Thank you!

Looks like it's related to bundling Sentry using webpack on the server - only happens when the app has been built by Webpack. Adding @sentry/node to externals has fixed our memory problem. Any idea why this happens?

@tpbowden You are right about this, I have the same issue. I was running the SDK v5.15.0 and node v12.3.1, both supposed to include all the required fixes mentioned here.

I am bundling all dependencies within my server bundle with webpack. This way I can ship a docker image without node_modules, but something is messing up the sentry SDK and it leaks memory when bundled this way.

It might be a bug made by some optimization process. My wild guess is it's probably terser. Some optimization is probably messing up the usage of the domain module, and the closure of the callback passed to scope.addEventProcessor is no longer garbage collected, so every request made is leaking a good amount of memory.

I'm also using razzle.js which is a bit behind on the webpack/terser versions, maybe it's already fixed.

This doesn't seem to be a bug on sentry's side anymore. I will continue to investigate this and open an issue where appropriate and keep this thread updated.

This doesn't seem to be a bug on sentry's side anymore. I will continue to investigate this and open an issue where appropriate and keep this thread updated.

Keep us posted, thank you!

@kamilogorek Could you point me where in the code the event processors callback added to the _eventProcessors array within the Scope instance are removed? I can't find it. It seems that all the requests are adding an event processor callback to this array, and they are never removed. If I would know how they are supposed to be removed, it might help me understand the bug better.

Or maybe it's the entire scope that is supposed to be unique and garbage collected for each request? It seems that each request are getting the same scope instance 🤔

Or maybe it's the entire scope that is supposed to be unique and garbage collected for each request?

That is correct. However, scope should be cloned for every new domain instance 🤔

See this call stack:

https://github.com/getsentry/sentry-javascript/blob/fd26d9fa273002502706b03fc1a9a46864cd8440/packages/node/src/handlers.ts#L319-L328

https://github.com/getsentry/sentry-javascript/blob/fd26d9fa273002502706b03fc1a9a46864cd8440/packages/hub/src/hub.ts#L442-L457

https://github.com/getsentry/sentry-javascript/blob/fd26d9fa273002502706b03fc1a9a46864cd8440/packages/hub/src/hub.ts#L463-L488

https://github.com/getsentry/sentry-javascript/blob/fd26d9fa273002502706b03fc1a9a46864cd8440/packages/hub/src/hub.ts#L479

Ha! I think I found something.

We use dynamicRequire:
https://github.com/getsentry/sentry-javascript/blob/fd26d9fa273002502706b03fc1a9a46864cd8440/packages/hub/src/hub.ts#L465-L468

But when I step into the dynamicRequire code:
https://github.com/getsentry/sentry-javascript/blob/fd26d9fa273002502706b03fc1a9a46864cd8440/packages/utils/src/misc.ts#L28-L31

require is undefined on mod 🤯

So it enters the catch block of the getHubFromActiveDomain function and instead use getHubFromCarrier()!

Since in my setup _everyting_ is bundled by webpack, there is probably some assumptions made on the mod object that is broken by webpack. Do you have an idea on how this could be fixed? 🤔

Recap

We use dynamicRequire:

mod.require is undefined:

What the mod object looks like:

We end up using getHubFromCarrier:

I manually patched the Hub module directly in my node_modules folder. I removed the line using the dynamicRequire and just added import domain from 'domain'; at the top of the file and... it now works perfectly! No more leaks! 🎉

Maybe the dynamicRequire hack was needed before, but is no longer needed with newer versions of webpack? 🤔

I also tried to replace :

const domain = dynamicRequire(module, 'domain');

with:

const domain = require('domain');

and it also works fine. I don't know which of those two solutions you would prefer.

Would you like me to open a PR with this fix?