Sendgrid-nodejs: Significant memory usage after upgrading from sendgrid to @sendgrid/mail.

Created on 12 Apr 2018  ·  19Comments  ·  Source: sendgrid/sendgrid-nodejs

Issue Summary

As sendgrid is deprecated, we decided to give @sendgrid/mail a try.

We were using the deprecated package without issues, but as soon as we replaced it, we started getting errors from the server running out of memory. It is important to mention that the emails are being delivered but the server completely crashes.

NOTE: We are reverting for the time being but I wanted to report it.

Error after the upgrade:

2018-04-12T19:24:02 heroku[web.1]: Process running mem=1111M(217.0%)
2018-04-12T19:24:02 heroku[web.1]: Error R15 (Memory quota vastly exceeded)
2018-04-12T19:24:02 heroku[web.1]: Stopping process with SIGKILL
2018-04-12T19:24:02 heroku[web.1]: Process exited with status 137
2018-04-12T19:24:02 heroku[web.1]: State changed from up to crashed

Steps to Reproduce

  1. npm uninstall sendgrid
  2. npm install --save @sendgrid/mail
  3. Replace code:

BEFORE:

const router = require("express").Router();
const sg = require("sendgrid")(process.env.SENDGRID_API_KEY);

router.post("/email-something", (req, res) => {

  // ... some constants and validations ...

  const request = sg.emptyRequest({
    method: "POST",
    path: "/v3/mail/send",
    body: {
      personalizations: [{
        to: [{ email: process.env.FORWARDING_EMAIL }],
        subject: "[SOMETHING] Request from..."
      }],
      from: { email: "[email protected]" },
      content: [{
        type: "text/plain",
        value: "add all info here..."
      }]
    }
  });

  sg.API(request)
    .then(response => {
      res.status(200).json({
        success: true
      });
    })
    .catch(error => {
      res.status(500).json({
        success: false,
        message: "There was an error sending the email."
      });
    });
});

AFTER:

const router = require("express").Router();
const sgMail = require("@sendgrid/mail");

sgMail.setApiKey(process.env.SENDGRID_API_KEY);

router.post("/email-something", (req, res) => {

  // ... some constants and validations ...

  const emails = [
    { // REQUEST
      to: process.env.FORWARDING_EMAIL,
      from: "[email protected]",
      subject: "[SOMETHING] Request from...",
      text: "add all info here..."
    }
  ];

  sgMail
    .send(emails)
    .then(response => {
      res.status(200).json({
        success: true
      });
    })
    .catch(error => {
      res.status(500).json({
        success: false,
        message: "There was an error sending the email."
      });
    });
});

Technical details:

We are running a nodejs server in Heroku using expressjs. This small server was created for the sole purpose of sending emails to a specific account, in other words, it is completely isolated and there are no other technologies interfering.

  • "sendgrid": "^5.2.3" (removed package)
  • "@sendgrid/mail": "^6.2.1" (added package)
  • "express": "^4.16.3"
  • node: v6.9.5
unknown or a help wanted question
Source
picture pgarciacamou

Most helpful comment

Good news @pgarciacamou @thinkingserious, I have narrowed it down and it turns out this is not caused by the sendgrid mail package.

In fact, the memory problem comes from heroku-logger, which is trying to log the success response.

I suspect it is an issue in the logger due to circular data in the response object:

image

I recommend raising an issue there or avoid logging the response object, as it's a very large and complex object.

You can test this with my fork of your repo, https://github.com/adamreisnz/sendgrid-memory-issue-677

I stripped out a lot of stuff including the templating engine to try and narrow it down, but in the end if you replace logger.info("sendgrid", response); with a simple console.log you‘ll be able to reproduce it.

Now let's hope I resolve my own memory issue :)

picture adamreisnz on 13 May 2018
❤2

All 19 comments

I downgraded again, here are some visuals before and during the upgrade, and also after the downgrade.

You can see:

  1. The memory spike and then crash.
  2. Memory jump from an average of 4.5% to +200% as soon as the new package was used.
  3. The response time for the deprecated library is way smaller and didn't cause memory to spike.

screen shot 2018-04-12 at 4 27 00 pm

I hope this helps, let me know if you guys would like more information.

pgarciacamou picture pgarciacamou on 13 Apr 2018

Hello @pgarciacamou,

Thanks for reporting this @pgarciacamou!

About how many requests per minute or hour does this server handle?

With Best Regards,

Elmer

cc @adamreisnz

thinkingserious picture thinkingserious on 14 Apr 2018

Hi @thinkingserious,

About how many requests per minute or hour does this server handle?

If you are asking me, how many it is handling right now? Anywhere from 1 to 100 request an hour on a _very_ busy day.

But if you are asking me, how many can it handle? Well, I'm uncertain of the answer to that question, I would have to load test the server but we are using NodeJS running on a single dyno in Heroku, and quoting Heroku:

A single dyno can serve thousands of requests per second, but performance depends greatly on the language and framework you use.
...
Multi-threaded or event-driven environments like Java, Unicorn, EventMachine, and Node.js can handle many concurrent requests. Load testing your app is the only realistic way to determine request throughput.
-- https://devcenter.heroku.com/articles/dynos#dynos-and-requests

I'm unsure of how could this be affecting.

pgarciacamou picture pgarciacamou on 14 Apr 2018

Hi all, I myself am also using the new mail package in my own Node project, which also runs on Heroku, but I have not experienced any memory issues as of yet.

The data from your Heroku performance look really strange; even if there is a memory issue, I would not expect it to have such a large impact. We send out quite a large number of mails, on occasion several hundred at once (sequentially) and have not experienced 30s request times or memory related crashes.

@pgarciacamou What stack does our Heroku app run on? Have you upgraded it to the latest yet? Have you considered running on Node 8?

@thinkingserious have you got anyone on the team who has experience with analysing memory leaks/usage? I am currently in the middle of a move and will not be able to look into this for the next week or two.

adamreisnz picture adamreisnz on 14 Apr 2018

@adamreisnz

Hi all, I myself am also using the new mail package in my own Node project, which also runs on Heroku, but I have not experienced any memory issues as of yet.

This is very good to know, thanks!

Can you tell me:

  1. are you using express? if so, what version?
  2. node version
  3. @sendgrid/mail version
  4. heroku dyno types you are using

What stack does your Heroku app run on? Have you upgraded it to the latest yet? Have you considered running on Node 8?

As mentioned, we are using node and express running on a single hobby dyno, our dependencies in package.json look like this:

  "dependencies": {
    "express": "^4.16.3",
    "helmet": "^3.12.0",
    "sendgrid": "^5.2.3"
  }

I'll try to upgrade the node version and retry this again on Monday, thanks for the idea!

pgarciacamou picture pgarciacamou on 15 Apr 2018

As mentioned, we are using node and express running on a single hobby dyno, our dependencies in package.json look like this

No, I meant the underlying linux version, it's listed as "stack" on the settings page of your app:

image

We're on heroku-16 which is based on Ubuntu 16.04. You may be on an older stack, as you need to initiate the upgrade process manually.

We run on Node 8.10.0, and yes, using express, version ^4.16.3 as well, and @sendgrid/mail version ^6.2.1.

We're presently also using the hobby dyno.

Let me know what you find on your end and yes give upgrading Node and possibly the Heroku stack a go 👍

adamreisnz picture adamreisnz on 15 Apr 2018
👍2

it's listed as "stack" on the settings page

screen shot 2018-04-18 at 10 23 11 am

I'll be upgrading node either today or tomorrow, first to the latest stable version, if I notice any behavior issues, downgrade to 8.10 and retest.

I'll keep you posted.

pgarciacamou picture pgarciacamou on 18 Apr 2018
👍1

We upgraded to node v9.8.0 as we are using that version for other repos. After running the tests we noticed the same behavior (emails are delivered but server crashes or comes close to crashing).

screen shot 2018-04-19 at 2 33 34 pm

Steps we followed:

$ npm uninstall sendgrid
$ npm install @sendgrid/mail

$ nvm install v9.8.0
v9.8.0 is already installed.
Now using node v9.8.0 (npm v5.8.0)

# we updated .nvmrc to v9.8.0 and package.json engines to "node": "9.8.0"
# then deployed to heroku...

Here are a few cleaned up screenshots from the PR (I had to remove comments from the company, etc...)

screen shot 2018-04-19 at 2 40 44 pm
screen shot 2018-04-19 at 2 41 18 pm

pgarciacamou picture pgarciacamou on 19 Apr 2018

I've added this to my backlog to investigate deeper. For now, this may be helpful. Thanks for taking the time to provide all the details!

thinkingserious picture thinkingserious on 20 Apr 2018
😄1

Thanks @thinkingserious and @adamreisnz!

pgarciacamou picture pgarciacamou on 20 Apr 2018
👍1

@pgarciacamou are you able to share a full code example of the application, if it's not proprietary, or otherwise setup a repository with the problem reproducible when deployed to Heroku? That will allow me to analyse further.

adamreisnz picture adamreisnz on 23 Apr 2018
👍1

@adamreisnz I'm unable to share the code from the app (private repo and other info about the company) but I'll create a temp repo with an almost perfect clone, if not this week, next week.

pgarciacamou picture pgarciacamou on 24 Apr 2018
👍1

I decided to do this right away, here is the repo: https://github.com/pgarciacamou/sendgrid-memory-issue-677. I added you guys as collaborators if you do not want to fork it.

A few comments:

First and most important, if you find anything that looks like it should be private, let me know asap.

  1. read the readme file
  2. there might be small bugs/typos here and there because I cleaned it up as fast as I could without testing.
  3. there are 3 commits:

Let me know if it is helpful in any way.

pgarciacamou picture pgarciacamou on 24 Apr 2018
👍1

Great thanks for that. I'm rather busy this week and the next, but will try to look into it when I have a spare moment.

adamreisnz picture adamreisnz on 27 Apr 2018
👍2

I ran into memory issues myself yesterday, however, it appears to have stemmed from an upgrade of bcrypt to 2.x. Nevertheless it has prompted me to investigate this issue and see if I can get to the bottom of it. I'll look into it this weekend and report back if I find anything.

adamreisnz picture adamreisnz on 12 May 2018

Good news @pgarciacamou @thinkingserious, I have narrowed it down and it turns out this is not caused by the sendgrid mail package.

In fact, the memory problem comes from heroku-logger, which is trying to log the success response.

I suspect it is an issue in the logger due to circular data in the response object:

image

I recommend raising an issue there or avoid logging the response object, as it's a very large and complex object.

You can test this with my fork of your repo, https://github.com/adamreisnz/sendgrid-memory-issue-677

I stripped out a lot of stuff including the templating engine to try and narrow it down, but in the end if you replace logger.info("sendgrid", response); with a simple console.log you‘ll be able to reproduce it.

Now let's hope I resolve my own memory issue :)

adamreisnz picture adamreisnz on 13 May 2018
❤2

@adamreisnz first of all, thank you very much for the time spent on this!

I will definitely take a look and will create an issue in heroku-logger if I find that that is the problem. It is still very odd to me that the issue is only happening when upgrading sendgrid.

Anyhow, I didn't close this issue because of the memory issue you are having (although it might be unrelated), feel free to close this.

pgarciacamou picture pgarciacamou on 15 May 2018

@pgarciacamou no problem, I think the reason why this happened after you upgraded, is that the new sendgrid library returns the whole raw response object from request, and i believe the previous library did not, or maybe returned a simplified response. The current response is quite large and contains circular references, causing the logger to fail.

I'll let @thinkingserious close the issue then! 👍

adamreisnz picture adamreisnz on 15 May 2018
👍1

@adamreisnz that makes a lot of sense. I could just filter the information with lodash (_.pick) instead of logging everything.

Thanks again!

pgarciacamou picture pgarciacamou on 15 May 2018
👍1
Was this page helpful?
0 / 5 - 0 ratings

Related issues

Chrischuck picture Chrischuck  ·  3Comments
Loriot-n picture Loriot-n  ·  4Comments
TobiahRex picture TobiahRex  ·  3Comments
mikemaccana picture mikemaccana  ·  4Comments
wooyah picture wooyah  ·  4Comments