Sendgrid-nodejs: Example on using inbound-mail-parser to parse incoming emails

Hello @machinshin,

Thanks for taking the time to report this. I have added this to our backlog for further investigation.

Any update on example? I really need to use this parser but I don't know how to.

Hi @ZeroCho,

We do have a package here, but unfortunately there is no good usage documentation. For that, you might want to check out the Python Inbound Parser.

With Best Regards,

Elmer

SendGrid sends their JSON inbound parse data as form/multi-part data. This will not be accessed immediately by req.body as body-parser doesn't deal with that format. I would use multer as an alternative.

@shishido92 Could you expand on a good example of how to do it. It looks like the documentation for any parsing inbound emails with nodejs is just nonexistent. Thanks!

@stephenfjohnson @machinshin If you are using expressjs this might help

app.use(express.json({limit: '10mb'}));
app.use(express.urlencoded({
  extended: true,
  type: "multipart/form-data",
  limit: '10mb'
}));

app.post('/endpoint',  function (req, res, next) {
  // req.body contains the text fields
  console.log(req.body);
  console.log('body' + req.body);
  res.status(200).json('ok');
})

Doing this will give you the contents of the request in the request body.
This ignores attachments though.

@nathfreder We use thumbs-up reactions on the issue summary as part of our prioritization. Please give a 👍 at the top.

@childish-sambino , @satyajeetjadhav 👍

Later late than never ;)
Thanks for the update, closing this issue

I know this has been closed but I've just spotted this issue and I have express with inbound-mail-parser working. Main thing I needed to do was to use the multer library to handle the multipart/form-data input and then use the keyValues() method of the @sendgrid/inbound-mail-parser library like below

require in multer and Sendgrid:

const multer = require('multer');
const upload = multer();
const mailParse = require('@sendgrid/inbound-mail-parser');

on my router declaration for the POST method use the upload.none() method as I was just getting text (No attachments):

router.post('/mail', upload.none(), function(req, res, next)

Within that POST definition I then use the the keyValues() method to get the relevant parts of the email:

try {
const config = {keys: ['to', 'from', 'subject', 'text',]};
const parsing = new mailParse(config, req.body);
let response = parsing.keyValues();
let to = response.to;
let from = response.from;
let subject = response.subject;
let messageBody = response.text
console.log('This is the subject from the mail: ', subject);
}

Hope that is of some use.

Hi all, can someone share an example how to parse attachments? i am using same configs, but attachment come in unreadeble string

const app = express();
app.use(cors);
app.use(busboy());
app.use(cookieParser());
app.use(bodyParser.urlencoded({ extended: true })); // support encoded bodies.
app.use(bodyParser.json({limit: '5mb'}));

router.post('', upload.any(),function(req, res) {
  // console.log(req.body);
  // console.log(req.files);
  // res.send(req.body);

  const config = {keys: ['to', 'from', 'subject', 'text','attachments']};
  const parsing = new mailParse(config, req.body);
  let response = parsing.keyValues();
  let to = response.to;
  let from = response.from;
  let subject = response.subject;
  let messageBody = response.text
  console.log('This is the subject from the mail: ', response);
  res.send(200);
});

The code posted by @jhorsfield-tw was very helpful since @sendgrid/inbound-mail-parser has no documentation, however it did not quite work for me. The constructor for mailParse takes request as a parameter not the request body.

const parsing = new mailParse(config, req);

Hi @ZeroCho,

We do have a package here, but unfortunately there is no good usage documentation. For that, you might want to check out the Python Inbound Parser.

With Best Regards,

Elmer

So you should write documentation then.

How is this still undocumented

still can't figure it out with nodejs express...

Re-opening as there is still no example for this in the docs.

Pull requests to add this feature are welcome and will be reviewed based on priority, but Twilio SendGrid is not actively building new functionality for the library.

Trying to receive attachments but still no luck with that...

Trying to receive attachments but still no luck with that...

Use RAW

const config = {
          keys: ['to', 'from', 'subject', 'text', 'attachments']
        };
        const parsedMail = new parse(config, req); // req is https.Request type

        parsedMail.getRawEmail((rawMail: any) => {
           console.log(rawMail); // returns null or {}
       }

         parsedMail.attachments((attachmentData: Attachment[]) => {
              console.log('attachments1:'); // returns [ ]
              console.log(attachmentData);
              console.log(JSON.stringify(attachmentData));
            });

I'm trying that but with no luck... it returns an empty object and when I do .attachments() I also get an empty array.

I am using firebase cloud functions, I have not had any success using the inbound mail parser and I am looking forward to some documentation that can help me use the library...

But using busboy (and deselecting "raw" in Sendgrid settings) is working quite well, except for one issue that lead me to try to use the mailparser...

Sendgrid lists the different encodings in the "charsets" field like this:
An email from Gmail:
{"to":"UTF-8","html":"UTF-8","subject":"UTF-8","from":"UTF-8","text":"UTF-8"}
An email from outlook:
{"to":"UTF-8","html":"iso-8859-1","subject":"UTF-8","from":"UTF-8","text":"iso-8859-1"}

When the emails contains international characters (like ÆØÅ) the result is that the text and html fields contains questionmarks instead of the actual nordic characters.

If I change the defCharset on busboy to Windows-1252 the text and html values will display correctly, but the other fields will display wrongly.

Will this be an issue also on the mail parser, or does the mail parser handle different encodings on the fields?

My code for parsing using busboy for anyone interested:
(If anyone can tell me how I solve the issue above using busboy I would be grateful..)

exports.sendgridparse = functions.https.onRequest((req, res) => {
const path = require('path');
const os = require('os');
const fs = require('fs');
const Busboy = require('busboy');

if (req.method !== 'POST') {
    // Return a "method not allowed" error
    return res.status(405).end();
}
const busboy = new Busboy({ headers: req.headers });    
const fields = {};
busboy.on('field', function(fieldname, value, fieldnameTruncated, valTruncated, encoding, mimetype){
    console.log('Busboy fild [' + fieldname + ']: value: ' + val);
fields[fieldname] = value;
});

let imageToBeUploaded = {};
let imagestobeuploaded = [];
let imageFileName;

busboy.on('file', (fieldname, file, filename, encoding, mimetype) => {
console.log(fieldname, file, filename, encoding, mimetype);

file.on('data', function(data) {
console.log('File [' + fieldname + '] got ' + data.length + ' bytes');
});
file.on('end', function() {
console.log('File [' + fieldname + '] Finished');
});

const imageExtension = filename.split('.')[filename.split('.').length - 1];
imageFileName = ${Math.round(Math.random() * 1000000000000).toString()}.${imageExtension};
const filepath = path.join(os.tmpdir(), imageFileName);
const url = https://firebasestorage.googleapis.com/v0/b/${environment.firebase.storageBucket}/o/${imageFileName}?alt=media;
imageToBeUploaded = { filepath, mimetype, imageFileName, imageExtension, url };
imagestobeuploaded.push(imageToBeUploaded);
file.pipe(fs.createWriteStream(filepath));
});

busboy.on('finish', async () => {
//all fields are available (imagestobeuploaded and fields) and can be saved to db
res.send(200);
});
busboy.end(req.rawBody);
});

@hhetland so, were you able to get the files using busboy? I'm also using it to get the other data from the email, but wasn't able to get the files... Does getting the files work for you?

@matheustavaresdev - Yes, I am getting the files using busboy.

First they are saved to a tmp folder in busboy.on('file'), and then using the information in the variable "imagestobeuploaded" I'm saving it to a storage bucket in firebase in busboy.on('finish').

awesome thanks!!!!!!!!!!!!

April 2020, still no official docs, right?

I know this has been closed but I've just spotted this issue and I have express with inbound-mail-parser working. Main thing I needed to do was to use the multer library to handle the multipart/form-data input and then use the keyValues() method of the @sendgrid/inbound-mail-parser library like below

require in multer and Sendgrid:

const multer = require('multer');
const upload = multer();
const mailParse = require('@sendgrid/inbound-mail-parser');

on my router declaration for the POST method use the upload.none() method as I was just getting text (No attachments):

router.post('/mail', upload.none(), function(req, res, next)

Within that POST definition I then use the the keyValues() method to get the relevant parts of the email:

try {
const config = {keys: ['to', 'from', 'subject', 'text',]};
const parsing = new mailParse(config, req.body);
let response = parsing.keyValues();
let to = response.to;
let from = response.from;
let subject = response.subject;
let messageBody = response.text
console.log('This is the subject from the mail: ', subject);
}

Hope that is of some use.

What I find is that the request.body is in form-data and see an error:
[Nest] 24687 - 05/01/2020, 1:29:51 PM [ExceptionsHandler] Reduce of empty array with no initial value
TypeError: Reduce of empty array with no initial value
at Array.reduce ()

The code:
keyValues() {
return this.keys
.filter(key => this.payload[key])
.map(key => ({ [key]: this.payload[key] }))
.reduce((keyValues, keyPayload) => Object.assign(keyValues, keyPayload));
}

returns an error as this.payload isn't a structured data but is multi-part form data.

I wasn't able to get this working with nest-js. The webhook data I receive in NestJS controller is multi-form data. I am unsure how to use Multer in that case.

outer.post('/mail', upload.none(), function(req, res, next)

I tried doing this:

export class MultiformParser implements NestMiddleware { async use(req: Request, res: Response, next: Function) { console.log("before: ", req.body) var upload = multer(); await new Promise((resolve, reject) => { upload.none()(req, res, err => { if (err) { console.log(err) reject(err) } else { resolve() console.log("request: ", req.body) } }); }); console.log("after: ", req.body) next(); } }

However, the req.body returns:
request: [Object: null prototype] {}

The before prints out multiform data.

If you are on serverless (google cloud functions or firebase)

this might help: https://stackoverflow.com/questions/47242340/how-to-perform-an-http-file-upload-using-express-on-cloud-functions-for-firebase/48648805#48648805

This using busboy: https://stackoverflow.com/questions/51484307/how-to-parse-multipart-form-data-on-firebase-cloud-functions

Use Formidable for parsing multipart on serverless: https://github.com/DanisHack/formidable-serverless

Or this: https://github.com/cristovao-trevisan/express-multipart-file-parser

Here are the changes I had to make so I could use multer:

app.use(express.json())
app.use(express.urlencoded({ extended: true }));
const upload = multer();
app.use(upload.none());

Once I did this, I was able to use multer to parse the webhook data into a
structured key, value pair.

On Sun, May 24, 2020 at 4:51 PM Danish Mohd notifications@github.com
wrote:

If you are on serverless (google cloud functions or firebase)
this might help:
https://stackoverflow.com/questions/47242340/how-to-perform-an-http-file-upload-using-express-on-cloud-functions-for-firebase/48648805#48648805

Use Formidable for parsing multipart on serverless:
https://github.com/DanisHack/formidable-serverless

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/sendgrid/sendgrid-nodejs/issues/482#issuecomment-633312295,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AB2ATHHFKBRZZDGGJ2D7SOLRTGQHNANCNFSM4D67CVZQ
.

--

Shweta Bhandare, PhD | Principal Software Engineer

[email protected]

https://hicleo.com/
hicleo.com | Linkedin https://www.linkedin.com/company/hicleo/ | Twitter
https://twitter.com/hi_cleo/ | Instagram
https://www.instagram.com/hicleo.co/ | Facebook
https://www.facebook.com/hicleolabs

--

This message may contain confidential, proprietary, or protected
information.  If you are not the intended recipient, you may not review,
copy, or distribute this message. If you received this message in error,
please notify the sender by reply email and delete this message.

We spent a lot of time trying to get this to work in Google Cloud Functions with Busboy and couldn't make it happen no matter what tutorial or pointers we followed.

So I'll leave this here for the next person in a similar situation where you have a bit of flexibility: use the Python Inbound Parser. It was pretty much a copy/paste job from the code they provide and took under an hour to get it running.

I know this doesn't solve the issue here, but I hope it can save someone else hours of headbanging.

I'm glad I found this ticket... I was going crazy trying to find some documentation on this, and somehow there is none.

I'm actually baffled. I just assumed there would be standard documentation - especially because while this is open source it is backed by SendGrid (and ultimately Twilio, which has fantastic documentation).

For something as simple as this, you'd imagine there would at least be an example.

Nonetheless, for anyone who comes across this...

I recommend using MailParser instead.

I'm pretty sure this uses it under the hood, but at least MailParser has loads of documentation. Just make sure that in your inbound parse settings you have POST the raw, full MIME message checked as shown below (_wow, an example_).

Reading through the MailParser docs, I got this going in minutes:

import { Request, Response } from "express";
import { simpleParser } from "mailparser";

export const inboundEmails = async (req: Request, res: Response) => {
  const parsedEmail = await simpleParser(req.body.email);

  console.log("subject", parsedEmail.subject);
  console.log("from", parsedEmail.from);
  console.log("to", parsedEmail.to);
  console.log("cc", parsedEmail.cc);
  console.log("bcc", parsedEmail.bcc);
  console.log("date", parsedEmail.date);
  console.log("messageId", parsedEmail.messageId);
  console.log("inReplyTo", parsedEmail.inReplyTo);
  console.log("replyTo", parsedEmail.replyTo);
  console.log("references", parsedEmail.references);
  console.log("html", parsedEmail.html);
  console.log("text", parsedEmail.text);
  console.log("textAsHtml", parsedEmail.textAsHtml);
  console.log("attachments", parsedEmail.attachments);

  res.sendStatus(200);
};

All the data is as you would like it.

I also use the Multer middleware to parse the multipart/form-data that SendGrid POSTs to my endpoint.

It's as simple as multer().any() within your middleware flow.

Cheers

I used the raw full MIME message and got it working with simpleParser, but I'm switching back to the processed version and using sengrid's inbound-mail-parser because simpleParser doesn't convert the MIME to text and html as nicely.

Definitely need some documentation though.

@theweiweiway Do you mind posting an example of the difference in text and html between the two methods? I haven't noticed any differences and I'm curious.

for text

Hi 123 Hi 123 Hi 123 www.hello.com

vs

Hi 123 Hi 123 Hi 123 www. <http://www.hello.com/>hello <http://www.hello.com/>.com <http://www.hello.com/>

^ dunno why that happened

for html

<html><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">Hi 123<div class=""><b class="">Hi 123</b></div><div class=""><b class=""><u class="">Hi 123</u></b></div><div class=""><b class=""><u class="">www.</u>hello</b>.com</div></body></html>

vs

<html><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><span style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);" class="">Hi 123</span><div class="" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);"><b class="">Hi 123</b></div><div class="" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);"><b class=""><u class="">Hi 123</u></b></div><div class="" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);"><b class=""><u class=""><a href="http://www.hello.com" class="">www.</a></u><a href="http://www.hello.com" class="">hello</a></b><a href="http://www.hello.com" class="">.com</a></div></body></html>

I think it's more to do with the simpleParser package tho

Here's my repo express-sendgrid-inbound-parse using express/multer. It's an easy app to get you started. I recommend leaving POST the raw, full MIME message unchecked as you can access to more email data.

console.log('dkim: ', body.dkim)
console.log('to: ', body.to)
console.log('cc: ', body.cc)
console.log('from: ', body.from)
console.log('subject: ', body.subject)
console.log('sender_ip: ', body.sender_ip)
console.log('spam_report: ', body.spam_report)
console.log('envelope: ', body.envelope)
console.log('charsets: ', body.charsets)
console.log('SPF: ', body.SPF)
console.log('spam_score: ', body.spam_score)

if (rawFullMimeMessageChecked) {
    console.log('email: ', body.email)
} else {
    console.log('headers: ', body.headers)
    console.log('html: ', body.html)
    console.log('text: ', body.text)
    console.log('attachments: ', body.attachments)
    console.log('attachment-info: ', body['attachment-info'])
    console.log('content-ids: ', body['content-ids'])
}

tried this on the firebase cloud function.
Don't work definitely.
Any another solution?