Next.js: Next.js API routes (and pages) should support reading files

Created on 5 Aug 2019  ·  87Comments  ·  Source: vercel/next.js

Feature request

Is your feature request related to a problem? Please describe.

It's currently not possible to read files from API routes or pages.

Describe the solution you'd like

I want to be able to call fs.readFile with a __dirname path and have it "just work".

This should work in Development and Production mode.

Describe alternatives you've considered

This may need to integrate with @zeit/webpack-asset-relocator-loader in some capacity. This plugin handles these types of requires.

However, it's not a necessity. I'd be OK with something that _only_ works with __dirname and __filename (no relative or cwd-based paths).

Additional context

Example:

// pages/api/test.js
import fs from 'fs'
import path from 'path'

export default (req, res) => {
  const fileContent = fs.readFileSync(
    path.join(__dirname, '..', '..', 'package.json'), 
    'utf8'
  )
  // ...
}

Note: I know you can cheat the above example ☝️ with require, but that's not the point. 😄

story feature request
Source
picture Timer
👍98 🚀26

Most helpful comment

Workaround I'm using:

# next.config.js
module.exports = {
    serverRuntimeConfig: {
        PROJECT_ROOT: __dirname
    }
}

and in the location you need the path

import fs from 'fs'
import path from 'path'
import getConfig from 'next/config'
const { serverRuntimeConfig } = getConfig()

fs.readFile(path.join(serverRuntimeConfig.PROJECT_ROOT, './path/to/file.json'))

I know this doesn't solve the need to reference files with paths relative to the current file, but this solves my very related use case (reading image files from a /public/images folder).

picture jkjustjoshing on 30 Oct 2019
👍20 ❤15 😕5 👎2

All 87 comments

Just wanted to second that, trying to implement file uploading using API routes. I can get the file to upload but then need to be able to access it again to upload it to S3 bucket.

amytych picture amytych on 7 Aug 2019
👍2

I second this! Also, being able to read directories is very important for my company's usage as we keep our data like team members and blog posts in a content directory so we're looking for a way to require all files in the directory.

ScottSmith95 picture ScottSmith95 on 12 Aug 2019

The above PR will fix this! ☝️ 🙏

Timer picture Timer on 12 Aug 2019
👍7 🎉5

How about fs.writeFile is that possible? For example, create and save a JSON file based on a webhook that was posted on an /api/route

marlonmarcello picture marlonmarcello on 30 Aug 2019

Hey @marlonmarcello, this is going to be possible. Stay tuned 😊

huv1k picture huv1k on 30 Aug 2019

It's this already solved?

NicolasHz picture NicolasHz on 15 Sep 2019

Not yet, you can subscribe for #8334

huv1k picture huv1k on 16 Sep 2019
👍2

@huv1k Many thanks!

NicolasHz picture NicolasHz on 16 Sep 2019

Is there a way to help this move forward more quickly?

BrunoBernardino picture BrunoBernardino on 3 Oct 2019

Worth noting: if you're using TypeScript, you can already import a JSON file as a module directly (make sure resolveJsonModule is true in tsconfig.json). E.g.:

import myJson from '../../../some/path/my.json';

The shape of the JSON object is also automatically used as its type, so autocomplete is really nice.

bitjson picture bitjson on 19 Oct 2019

Workaround I'm using:

# next.config.js
module.exports = {
    serverRuntimeConfig: {
        PROJECT_ROOT: __dirname
    }
}

and in the location you need the path

import fs from 'fs'
import path from 'path'
import getConfig from 'next/config'
const { serverRuntimeConfig } = getConfig()

fs.readFile(path.join(serverRuntimeConfig.PROJECT_ROOT, './path/to/file.json'))

I know this doesn't solve the need to reference files with paths relative to the current file, but this solves my very related use case (reading image files from a /public/images folder).

jkjustjoshing picture jkjustjoshing on 30 Oct 2019
👍20 ❤15 😕5 👎2

Saw in the PR this has changed a bit - any update on what the current plans are (or aren't)? Sounds like there are some strategies you don't want pursued, mind listing them + why so contributors can give this a shot?

IanMitchell picture IanMitchell on 30 Oct 2019

This is blocking the usage of nexus with Next.js. It would be great to see this prioritized again.

timsuchanek picture timsuchanek on 10 Dec 2019
👍8

Workaround I'm using:

# next.config.js
module.exports = {
    serverRuntimeConfig: {
        PROJECT_ROOT: __dirname
    }
}

and in the location you need the path

import fs from 'fs'
import path from 'path'
import getConfig from 'next/config'
const { serverRuntimeConfig } = getConfig()

fs.readFile(path.join(serverRuntimeConfig.PROJECT_ROOT, './path/to/file.json'))

I know this doesn't solve the need to reference files with paths relative to the current file, but this solves my very related use case (reading image files from a /public/images folder).

Awesome Man. Worked for me.

karthhhi picture karthhhi on 10 Jan 2020
👍1

I've been using the new getStaticProps method for this (in #9524). The method is currently marked as unstable but there seems to be good support from the Next.js team on shipping it officially.

e.g.:

export async function unstable_getStaticProps() {
  const siteData = await import("../data/pages/siteData.json");
  const home = await import("../data/pages/home.json");

  return {
    props: { siteData, home }
  };
}
ScottSmith95 picture ScottSmith95 on 10 Jan 2020
👍2

@ScottSmith95 Do you have some public source project where you're using this? Curious about what it would look like.

Svish picture Svish on 11 Jan 2020
👍1

The project is not open source, yet but I am happy to share more of my config if you have more questions.

ScottSmith95 picture ScottSmith95 on 15 Jan 2020

@ScottSmith95 I have _all_ the questions 😛

  1. Where in your project do you store the data-files? (outside/inside src?)
  2. What does a next.js page component using them look like?
  3. Is it hard-coded paths only, or can you load a file based on path parameters?
  4. How does build/deployment work, especially if it's not hard-coded paths?
Svish picture Svish on 16 Jan 2020

@Svish We store data files in /data within our project. (Pages are in /pages, not /src/prages.) This page component looks like this (props are sent to the Home component which is the default export):

// /pages/index.js
const Home = ({ siteData, home }) => {
  return (
    <>
      <Head>
        <meta name="description" content={siteData.siteDescription} />
        <meta name="og:description" content={siteData.siteDescription} />
        <meta
          name="og:image"
          content={getAbsoluteUrl(siteData.siteImage, constants.siteMeta.url)}
        />
      </Head>
      <section className={`container--fluid ${styles.hero}`}>
        <SectionHeader section={home.hero} heading="1">
          <div className="col-xs-12">
            <PrimaryLink
              href={home.hero.action.path}
              className={styles.heroAction}
            >
              {home.hero.action.text}
            </PrimaryLink>
          </div>
        </SectionHeader>
        <div className={styles.imageGradientOverlay}>
          <img src={home.hero.image.src} alt={home.hero.image.alt} />
        </div>
      </section>
    </>
  );
};

For more advanced pages, those with dynamic routes, we grab this data like so:

// /pages/studio/[member.js]
export async function unstable_getStaticProps({ params }) {
  const siteData = await import("../../data/pages/siteData.json");
  const member = await import(`../../data/team/${params.member}.json`);

  return {
    props: { siteData, member }
  };
}

Deployment goes really smoothly, with dynamic routes, getStaticPaths() becomes necessary. I encourage you to check out the RFC for the documentation on that, but here's an example of how we handle that by gathering all our team member data and passing it to Next.js. 

// /pages/studio/[member.js]
export async function unstable_getStaticPaths() {
  const getSingleFileJson = async path => await import(`../../${path}`);

  // These utility functions come from `@asmallstudio/tinyutil` https://github.com/asmallstudio/tinyutil
  const directoryData = await getDirectory(
    "./data/team",
    ".json",
    getSingleFileJson,
    createSlugFromTitle
  );
  const directoryPaths = directoryData.reduce((pathsAccumulator, page) => {
    pathsAccumulator.push({
      params: {
        member: page.slug
      }
    });

    return pathsAccumulator;
  }, []);

  return directoryPaths;
}
ScottSmith95 picture ScottSmith95 on 16 Jan 2020
🚀1

@ScottSmith95 Looks promising! A couple of follow-up questions if you have time:

  1. What you're doing here is for static site generation? I.e. when using next export?
  2. Have I understood it correctly, that getStaticPaths returns a list of path parameters, which is then (by next) fed, one by one, into getStaticProps for each render?
  3. Can you use getStaticProps without getStaticPaths, for example for a page without any parameters?
  4. Can you use getStaticProps in _app? For example if you have some site wide config you'd like to load or something like that?
Svish picture Svish on 16 Jan 2020

@svish https://github.com/zeit/next.js/issues/9524

timneutkens picture timneutkens on 16 Jan 2020

What about the apis?? Those hooks are for pages, but what about apis?

NicolasHz picture NicolasHz on 17 Jan 2020
👍5

I'm confused. I was able to set the _dirname as an env variable in the next config. Therefore I was able to access the filesystem from the API, but it only worked locally. After deploying it to now, I got an error. Any ideas why it won't work after deployment?

josias-r picture josias-r on 1 Feb 2020
👍1

@josias-r the main issue is usually that the files to be read are not included the deployment, but it depends on how you include them and which types of files they are (js/json is usually fine, but other file types like .jade will require alternative ways of dealing with his, like using a separate @now/node lambda/deployment for reading/handling those files).

If you can explain more about the error, maybe someone can help you.

BrunoBernardino picture BrunoBernardino on 1 Feb 2020

@BrunoBernardino It was actually referring to JSON files inside my src folder. But it's actually even the fs.readdirSync(my_dirname_env_var) method that already fails in deployment. So that dir doesn't seem to exist at all after deployment. Here is what I get when I try to access the full path to the json vis my API:

ERROR   Error: ENOENT: no such file or directory, open '/zeit/3fc37db3/src/content/somejsonfilethatexists.json'

And as I mentioned, this works locally when I build and then run npm start.

josias-r picture josias-r on 1 Feb 2020

@josias-r Thanks! Have you tried doing the fs.readdirSync with a relative path (no variables) instead (just to debug the deployment)? I've found that to usually work, and if so, you can write that piece of code (just reading the file, not storing it anywhere) somewhere in an initialization process (getInitialProps or something), so that the deployment process picks up that it needs that file, and then keep reading it with the var in the actual code/logic. It's not neat, but it works until this is supported. I believe that also using __dirname works in some cases.

BrunoBernardino picture BrunoBernardino on 2 Feb 2020

@BrunoBernardino I was able to build a file tree starting from the root-relative path ./. What I got was the following JSON (without the node modules listed):

{
  "path": "./",
  "name": ".",
  "type": "folder",
  "children": [
    {
      "path": ".//.next",
      "name": ".next",
      "type": "folder",
      "children": [
        {
          "path": ".//.next/serverless",
          "name": "serverless",
          "type": "folder",
          "children": [
            {
              "path": ".//.next/serverless/pages",
              "name": "pages",
              "type": "folder",
              "children": [
                {
                  "path": ".//.next/serverless/pages/api",
                  "name": "api",
                  "type": "folder",
                  "children": [
                    {
                      "path": ".//.next/serverless/pages/api/posts",
                      "name": "posts",
                      "type": "folder",
                      "children": [
                        {
                          "path": ".//.next/serverless/pages/api/posts/[...id].js",
                          "name": "[...id].js",
                          "type": "file"
                        }
                      ]
                    }
                  ]
                }
              ]
            }
          ]
        }
      ]
    },
    {
      "path": ".//node_modules",
      "name": "node_modules",
      "type": "folder",
      "children": ["alot of children here ofc"]
    },
    { "path": ".//now__bridge.js", "name": "now__bridge.js", "type": "file" },
    {
      "path": ".//now__launcher.js",
      "name": "now__launcher.js",
      "type": "file"
    }
  ]
}
josias-r picture josias-r on 2 Feb 2020

Your JSON file seems to be missing there, did you try including it via the code like I suggested above? The main problem is that the optimizations the deployment runs don’t always pick up dynamic paths, I believe, so forcing a static path has worked for me in the past (not necessarily for the actual code running, but to make sure the relevant files are included). Does that makes sense?

BrunoBernardino picture BrunoBernardino on 2 Feb 2020

@BrunoBernardino I've switched to a non API solution. Since I dynamically want to require files from a folder and I only need the content of these files, I'm able to use the import() method. I just didn't want to do it this way, because it seems hacky, but it's essentially doing the same thing my API endpoint would have done.
... I tried putting the file into the static folder but that didn't work either. But I hope accessing the filesystem will be possible in the future.

josias-r picture josias-r on 3 Feb 2020
👍1

I've also had to resort to hacky solutions, but hopefully this will land soon and more people will start seeing Next as production-ready as these use cases become supported "as expected".

BrunoBernardino picture BrunoBernardino on 3 Feb 2020
👍1

Workaround I'm using:

# next.config.js
module.exports = {
    serverRuntimeConfig: {
        PROJECT_ROOT: __dirname
    }
}

and in the location you need the path

import fs from 'fs'
import path from 'path'
import getConfig from 'next/config'
const { serverRuntimeConfig } = getConfig()

fs.readFile(path.join(serverRuntimeConfig.PROJECT_ROOT, './path/to/file.json'))

I know this doesn't solve the need to reference files with paths relative to the current file, but this solves my very related use case (reading image files from a /public/images folder).

Awesome Man. Worked for me.

It works perfectly on local development, though it does not seem to work when deploying to now. 

ENOENT: no such file or directory, open '/zeit/41c233e5/public/images/my-image.png'
    at Object.openSync (fs.js:440:3)
    at Object.readFileSync (fs.js:342:35)
    at getEmailImage (/var/task/.next/serverless/pages/api/contact/demo.js:123:52)
    at module.exports.7gUS.__webpack_exports__.default (/var/task/.next/serverless/pages/api/contact/demo.js:419:87)
    at processTicksAndRejections (internal/process/task_queues.js:93:5)
    at async apiResolver (/var/task/node_modules/next/dist/next-server/server/api-utils.js:42:9) {
  errno: -2,
  syscall: 'open',
  code: 'ENOENT',
  path: '/zeit/41c233e5/public/images/my-image.png'
}

I understand that the public folder gets moved to the route so I tried to force it to search in the base folder when on production but still got the same result:

ENOENT: no such file or directory, open '/zeit/5fed13e9/images/my-image.png'
    at Object.openSync (fs.js:440:3)
    at Object.readFileSync (fs.js:342:35)
    at getEmailImage (/var/task/.next/serverless/pages/api/contact/demo.js:124:52)
    at module.exports.7gUS.__webpack_exports__.default (/var/task/.next/serverless/pages/api/contact/demo.js:331:87)
    at processTicksAndRejections (internal/process/task_queues.js:93:5)
    at async apiResolver (/var/task/node_modules/next/dist/next-server/server/api-utils.js:42:9) {
  errno: -2,
  syscall: 'open',
  code: 'ENOENT',
  path: '/zeit/5fed13e9/images/my-image.png'
}
PaulPCIO picture PaulPCIO on 5 Feb 2020
👍6

@PaulPCIO the problem you're experiencing there is because it's not a .json, .js, or .ts file. The files under /public are "deployed" to a CDN but not to the lambda (AFAIK), so for that case you need either a dedicated lambda (@now/node) deployment with includeFiles, or, if you only need that single file, convert it to base64 and use that as a var (in a dedicated file or not).

BrunoBernardino picture BrunoBernardino on 5 Feb 2020
👍1

Thanks @BrunoBernardino expected as much, I will use the base64 method

PaulPCIO picture PaulPCIO on 5 Feb 2020
👍1

It's some resolution to the __dirname in deployed environment??

NicolasHz picture NicolasHz on 6 Feb 2020

@NicolasHz can you elaborate? I didn’t quite understand your question.

BrunoBernardino picture BrunoBernardino on 7 Feb 2020

@BrunoBernardino Looking at the last comments, including mine, I'm pretty sure that the "map _dirname in the next config" hack doesn't work in deployment. Even w/ js and JSON files. At least for now deployment, that doesn't count for custom deployments probably.

josias-r picture josias-r on 7 Feb 2020
👍1

@BrunoBernardino I'm not able to use some variables poiting to the local path on the deployed env. __dirname it's undefined once deployed, and I'm unable to read a file from my apis scripts.

NicolasHz picture NicolasHz on 7 Feb 2020

Got it @NicolasHz . Yeah, you'll need to resort to one of the solutions above, depending on which kind of file you need to read/access.

BrunoBernardino picture BrunoBernardino on 7 Feb 2020

Just confirming, the config.js is not working on deployments.

BrunoQuaresma picture BrunoQuaresma on 13 Feb 2020

Workaround I'm using:

# next.config.js
module.exports = {
  env: {
    PROJECT_DIRNAME: __dirname,
  },
}

and in the api definition where i need the path(allPosts folder contains all blogs in markdown format and it is located in project root )

import fs from 'fs'
import { join } from 'path'

const postsDirectory = join(process.env.PROJECT_DIRNAME, 'allPosts')

It is working perfectly on local development.
But it is giving this error when deploying to zeit now.

[POST] /api/postsApi
11:00:13:67
Status:
500
Duration:
8.1ms
Memory Used:
76 MB
ID:
kxq8t-1585546213659-5c3393750f30
User Agent:
axios/0.19.2
{
  fields: [ 'title', 'date', 'slug', 'author', 'coverImage', 'excerpt' ],
  page: 1
}
2020-03-30T05:30:13.688Z    572075eb-4a7a-47de-be16-072a9f7005f7    ERROR   Error: ENOENT: no such file or directory, scandir '/zeit/1cc63678/allPosts'
    at Object.readdirSync (fs.js:871:3)
    at getPostSlugs (/var/task/.next/serverless/pages/api/postsApi.js:306:52)
    at module.exports.fZHd.__webpack_exports__.default (/var/task/.next/serverless/pages/api/postsApi.js:253:86)
    at apiResolver (/var/task/node_modules/next/dist/next-server/server/api-utils.js:48:15)
    at processTicksAndRejections (internal/process/task_queues.js:97:5) {
  errno: -2,
  syscall: 'scandir',
  code: 'ENOENT',
  path: '/zeit/1cc63678/allPosts'
}
sjcodebook picture sjcodebook on 30 Mar 2020

@sjcodebook like @BrunoQuaresma said, that workaround only works locally. I'm still using a separate @now/node deployment for lambdas to access the filesystem, and call that file via a request from the app itself (or generate whatever static result I need before deploying). Kinda insane, but it works.

BrunoBernardino picture BrunoBernardino on 30 Mar 2020

Hi @BrunoBernardino... Do you mean a separate project with a custom node server?

However I don't understand why there's an "includeFiles" setting if then it's impossible to access them 🤔

valse picture valse on 15 Apr 2020

@valse it can be on the same project. Here's a snippet of my now.json:

{
  "builds": [
    {
      "src": "next.config.js",
      "use": "@now/next"
    },
    {
      "src": "lambdas/**/*.ts",
      "use": "@now/node",
      "config": {
        "includeFiles": ["email-templates/**"]
      }
    }
  ],
  "routes": [
    {
      "src": "/lambdas/(.+)",
      "dest": "/lambdas/$1.ts"
    }
  ]
}

That way I can call them via something like:

await ky.post(`${hostUrl}/lambdas/email?token=${someToken}`);

from inside a next api page, assuming I have a lambdas/email.ts file which handles sending emails and reading from template files like pug.

I hope that helps!

BrunoBernardino picture BrunoBernardino on 15 Apr 2020
👍6 ❤2 🎉2

Also, "includeFiles" only works for @now/node (maybe others, but not @now/next)

BrunoBernardino picture BrunoBernardino on 15 Apr 2020

@BrunoBernardino looks like if using node functions, it now can't read ESM!

this is what happens when I try to import a list of mdx pages:

code

import { NextApiRequest, NextApiResponse } from 'next'
import { promises as fs } from 'fs'
import { join } from 'path'
const { readdir } = fs

export default async (req: NextApiRequest, res: NextApiResponse) => {
  const postFiles = await readdir(join(process.cwd(), 'pages', 'blog'))

  const postNames: string[] = postFiles.filter((page: string) => page !== 'index.tsx')

  const posts = []

  for (const post of postNames) {
    const mod = await import(`../pages/blog/${post}`)

    posts.push({ ...mod, link: post.slice(0, post.indexOf('.')) })
  }

  res.status(200).json([])
}

the error I get:

export const title = 'My new website!'
^^^^^^

SyntaxError: Unexpected token 'export'
talentlessguy picture talentlessguy on 4 May 2020

@talentlessguy I'm not on the Zeit/Vercel team, just a happy customer. Seems like that might be better suited to their customer support, as I see a few potential issues just from that snippet:

  1. You might want to use just nothing or __dirname instead of process.cwd() for base path. I haven't used the latter in lambdas, but the others, so I'm not sure if that's an issue or not
  2. You're importing NextApiRequest and NextApiResponse as types, but this should be running from @now/node", right? So the types should be imported like:
import { NowRequest, NowResponse } from '@now/node';
  1. You're importing/reading from pages/... but are you including them via includeFiles? What does your now.json look like?
BrunoBernardino picture BrunoBernardino on 4 May 2020

@BrunoBernardino

I can't use __dirname because it is always /, process.cwd() instead, shows the real path

I accepted ur fixes and it worked:

lambdas/posts.ts

import { NowResponse, NowRequest } from '@now/node'
import { promises as fs } from 'fs'
import { join } from 'path'
const { readdir } = fs

export default async (req: NowRequest, res: NowResponse) => {
  const postFiles = await readdir(join(process.cwd(), 'pages', 'blog'))

  const postNames: string[] = postFiles.filter((page: string) => page !== 'index.tsx')

  const posts = []

  for (const post of postNames) {
    const mod = await import(`../pages/blog/${post}`)

    posts.push({ ...mod, link: post.slice(0, post.indexOf('.')) })
  }

  res.status(200).json([])
}

now.json

{
  "builds": [
    {
      "src": "next.config.js",
      "use": "@now/next"
    },
    {
      "src": "lambdas/**/*.ts",
      "use": "@now/node",
      "config": {
        "includeFiles": ["pages/blog/*.mdx"]
      }
    }
  ],
  "routes": [
    {
      "src": "/lambdas/(.+)",
      "dest": "/lambdas/$1.ts"
    }
  ]
}

error I get:

import Meta from '../../components/Article/Meta.tsx'
^^^^^^

SyntaxError: Cannot use import statement outside a module

Looks like typescript node function can't treat .mdx as a module :(

talentlessguy picture talentlessguy on 5 May 2020
👍2

Alright, so it seems you found the problem. Try reading the file contents and parsing them instead of importing directly. I’ve never seen an import like that work, and it seems like something that would only work with some Babel magic, which you’re also welcome to use instead of plain TS.

BrunoBernardino picture BrunoBernardino on 5 May 2020

@BrunoBernardino you're right, but it's not plain ts... I have the target set to esnext and module to esnext also, it should be able to import everything... but somehow it doesn't

anyways it's not related to the issue, gonna google it somewhere

talentlessguy picture talentlessguy on 5 May 2020

No worries. A couple of tips might be in https://mdxjs.com/advanced/typescript and https://mdxjs.com/getting-started/webpack which might make it so the @now/node deployment needs to be tweaked to use it. Anyway, their support should be of help.

BrunoBernardino picture BrunoBernardino on 5 May 2020

any movement on this? It would be great to be able to include html email templates for use in API Routes. Right now I am including them in JS files but I am not a particular fan of this hack.

jasonsilberman picture jasonsilberman on 12 May 2020
👍6

Another hack is to use webpack raw-loader to embed them into js.

yarn add --dev raw-loader

const templates = {
    verify: require("raw-loader!../template/email/verify.hbs").default,
};

Then use templates.verify as a string.

balthild picture balthild on 27 May 2020

There's an issue going on with next-i18next that seems to be related to this one (vercel/vercel#4271) . Basically now doesn't put the .json files located inside /public/static/locales/ into the serverless function. Can anyone provide a workaround until the feature discussed here is added to next?

borispoehland picture borispoehland on 31 May 2020

@borispoehland have you tried the import/require workarounds from above? That should work.

BrunoBernardino picture BrunoBernardino on 31 May 2020

@borispoehland have you tried the import/require workarounds from above? That should work.

@BrunoBernardino I don't know what exact comment you mean.

Can you give me an example of somehow importing all the .json files inside public/static/locales into the serverless function? And where to do this (in what file)?

I'm using next (as you stated earlier, includeFiles isn't compatible with @now/next, idk if this has any impact on my problem).

Besides, because next-i18next is kind of a blackbox to me (thus I don't want to import the files from there), I search for a way to entirely import them so that next-i18next can directly access them (in other comments above, sometimes only the PROJECT_DIRNAME was defined inside the next.config.json and the import had to be done manually. This is not what I try to reach). Like in vercel/vercel#4271, I just want now to take my .json files into the serverless function somehow.

borispoehland picture borispoehland on 1 Jun 2020

@borispoehland in _any_ file inside pages/api (or that gets called by one there), do something like https://github.com/vercel/next.js/issues/8251#issuecomment-544008976

You don't need to do anything with the import. The point is that the webpack vercel runs will then see those files need to be included, and it should work.

I hope that makes sense.

BrunoBernardino picture BrunoBernardino on 1 Jun 2020

@borispoehland in _any_ file inside pages/api (or that gets called by one there), do something like #8251 (comment)

You don't need to do anything with the import. The point is that the webpack vercel runs will then see those files need to be included, and it should work.

I hope that makes sense.

@BrunoBernardino the problem with this approach is that I have lots of json files. Doing the import manually for every file is kind of cumbersome. Is there a easier way to tell now: "Hey, please pick up all json files inside that directory recursively"? Thanks in advance

Edit: Even manually importing json files results in the same error than before. I'm going to open a new issue for this, I guess

borispoehland picture borispoehland on 1 Jun 2020
👍1

I opened a new issue for my problem, in case someone is interested in joining the discussion. Thanks for now, @BrunoBernardino !

borispoehland picture borispoehland on 1 Jun 2020
❤2 👍2

Another option / workaround to enable the ability to use __dirname as you would normally expect it to behave is to adjust the webpack config.

By default, webpack will alias various Node globals with polyfills unless you tell it not to:
https://webpack.js.org/configuration/node/
And the webpack default settings are to leave __dirname and __filename alone, i.e. not polyfill them and let node handle them as normal.

However, the Next.js webpack config doesn't use / reflect the webpack defaults https://github.com/vercel/next.js/blob/bb6ae2648ddfb65a810edf6ff90a86201d52320c/packages/next/build/webpack-config.ts#L661-L663

All of that said, I have used the below custom Next config plugin to adjust the webpack config.

IMPORTANT: this works for my use case. It has not been tested in a wide range of environments / configurations nor has it been tested against all of the Next.js unit/integration tests. Using it may have unintended side-effects in your environment.
Also, Next may have specific reasons for not using the webpack default settings for __dirname and __filename. So again, the code below may have unintended side-effects and should be used with caution.

Also, the below plugin has been designed for use with the next-compose-plugins package: https://github.com/cyrilwanner/next-compose-plugins

But should work as a normal plugin as well: https://nextjs.org/docs/api-reference/next.config.js/custom-webpack-config

const withCustomWebpack = (nextCfg) => {
  return Object.assign({}, nextCfg, {
    webpack(webpackConfig, options) {
      // We only want to change the `server` webpack config.
      if (options.isServer) {
        // set `__dirname: false` and/or `__filename: false` here to align with webpack defaults:
        // https://webpack.js.org/configuration/node/
        Object.assign(webpackConfig.node, { __dirname: false });
      }

      if (typeof nextCfg.webpack === 'function') {
        return nextCfg.webpack(webpackConfig, options);
      }

      return webpackConfig;
    },
  });
};
aweber1 picture aweber1 on 3 Jun 2020

I implemented the solution by @jkjustjoshing, and while it works great locally, it does not work when I deploy the app to Vercel.

I get the following error:

Error: GraphQL error: ENOENT: no such file or directory, open '/vercel/37166432/public/ts-data.csv'

My code:

const content = await fs.readFile(
  path.join(serverRuntimeConfig.PROJECT_ROOT, "./public/ts-data.csv")
);

Here's a link to the file: https://github.com/bengrunfeld/trend-viewer/blob/master/pages/api/graphql-data.js

bengrunfeld picture bengrunfeld on 13 Jul 2020

@bengrunfeld yes, your solution only works locally.

I had a similar problem lately (wanted to read a file in a API route) and the solution was easier than expected.

Try path.resolve('./public/ts-data.csv')

borispoehland picture borispoehland on 13 Jul 2020
👍15 🎉6 ❤5 🚀4

@borispoehland Thank you SO much!! Your solution worked beautifully!

bengrunfeld picture bengrunfeld on 14 Jul 2020

@bengrunfeld no problem, I also found it out on coincidence (@BrunoBernardino ;)). It's the solution to everyone's problem here, I guess.

borispoehland picture borispoehland on 14 Jul 2020
❤3 😄1

Please note, you still need to set next.config.js. I deleted the file after I saw that @borispoehland's solution worked, and received a similar error.

Then I reset it to @jkjustjoshing's solution above, deployed again to Vercel, and it worked. 

# next.config.js
module.exports = {
    serverRuntimeConfig: {
        PROJECT_ROOT: __dirname
    }
}
bengrunfeld picture bengrunfeld on 14 Jul 2020

Please note, you still need to set next.config.js. I removed it after I saw that @borispoehland's solution worked, and received a similar error.

I reset it to @jkjustjoshing's solution above, deployed again to Vercel, and it worked. 

# next.config.js
module.exports = {
    serverRuntimeConfig: {
        PROJECT_ROOT: __dirname
    }
}

@bengrunfeld Really? Maybe you're still using the PROJECT_ROOT approach at another point in the code, because in my project it works without it. How does the error look like?

borispoehland picture borispoehland on 14 Jul 2020

When deploying to Vercel, how do I write a readFile in a Page that will work both in SSG and SSR/Preview mode?

Demo repository of where it does not work: https://github.com/mathdroid/blog-fs-demo

https://blog-fs-demo.vercel.app/

mathdroid picture mathdroid on 26 Jul 2020

@mathdroid try moving readFile and readdir inside the getStaticProps and getStaticPaths functions, respectively, otherwise the code might run in the browser.

Importing fs should be fine on top, though.

Let us know how that works.

BrunoBernardino picture BrunoBernardino on 26 Jul 2020

@borispoehland Thanks for the wonderful solution. Did not expect path.resolve() to /public would work both locally and on Vercel :eyes:! You're my savior for the day. :+1:

subwaymatch picture subwaymatch on 3 Aug 2020
🎉1 👍1

@borispoehland i tried your solution inside a serverless functionbut still get:
ENOENT: no such file or directory, open '/var/task/public/posts.json'

const postsFile = resolve('./public/posts.json');

const updateCache = async (posts: IPost[]): Promise<IPost[]> => {
    postCache = posts;
    fs.writeFileSync(postsFile, JSON.stringify(postCache)); // <====
    return postCache;
}

i tried with our without the next.config.js

module.exports = {
    serverRuntimeConfig: {
        PROJECT_ROOT: __dirname
    }
}

Maybe your solution does not work on serverless functions?

neckaros picture neckaros on 12 Aug 2020
👍1

@borispoehland i tried your solution inside a serverless functionbut still get:
ENOENT: no such file or directory, open '/var/task/public/posts.json'

const postsFile = resolve('./public/posts.json');

const updateCache = async (posts: IPost[]): Promise<IPost[]> => {
    postCache = posts;
    fs.writeFileSync(postsFile, JSON.stringify(postCache)); // <====
    return postCache;
}

i tried with our without the next.config.js

module.exports = {
    serverRuntimeConfig: {
        PROJECT_ROOT: __dirname
    }
}

Maybe your solution does not work on serverless functions?

I don't know why it doesn't work on your end... Sorry

borispoehland picture borispoehland on 12 Aug 2020

Ok i made it work for read using @bengrunfeld code but unfortunately you cannot write:
[Error: EROFS: read-only file system, open '/var/task/public/posts.json']
So no way to update a cache to avoid too much database calls :(

neckaros picture neckaros on 12 Aug 2020
👍1

@neckaros have you tried using my approach to read a file other than .json, e.g. a .jpg file?

borispoehland picture borispoehland on 12 Aug 2020
👍1

Ok i made it work for read using @bengrunfeld code but unfortunately you cannot write:

[Error: EROFS: read-only file system, open '/var/task/public/posts.json']

So no way to update a cache to avoid too much database calls :(

@neckaros you should be able to write and read from S3 (or some other, external filesystem), but I usually use redis for quick, cached things that can be volatile. https://redislabs.com keeps it "serverless", and I've got production-ready code examples in https://nextjs-boilerplates.brn.sh if you want.

BrunoBernardino picture BrunoBernardino on 12 Aug 2020
👍1

@borispoehland i could read but not write from serveless function. But I ended up having it working by refresh my cache in the incremental builds (revalidate) instead of on add new content . Which I guess is not a bad pattern. Thanks for your help!

@BrunoBernardino thanks i will have a look. Really looking to a fully free hobbyist solution that does not break once you have a few users :)

neckaros picture neckaros on 13 Aug 2020

Really looking to a fully free hobbyist solution that does not break once you have a few users :)

Copy that. RedisLabs and Vercel did that for me. 💯

BrunoBernardino picture BrunoBernardino on 13 Aug 2020

After some digging I got writing files working with the extended os package...

import { tmpdir } from "os";
const doc = new PDFDocument()
const pdfPath = path.join(tmpdir(), `${store.id}${moment().format('YYYYMMDD')}.pdf`)
const writeStream = doc.pipe(fs.createWriteStream(pdfPath)

reading a file works with @subwaymatch solution
const logoPath = path.resolve('./public/logo.png')

Edjevw12 picture Edjevw12 on 25 Aug 2020
🚀2 👍2

After some digging I got writing files working with the extended os package...

import { tmpdir } from "os";
const doc = new PDFDocument()
const pdfPath = path.join(tmpdir(), `${store.id}${moment().format('YYYYMMDD')}.pdf`)
const writeStream = doc.pipe(fs.createWriteStream(pdfPath)

reading a file works with @subwaymatch solution
const logoPath = path.resolve('./public/logo.png')

Nice, are you able to read back the contents of this file? Is the directory accessible and permanent?

marklundin picture marklundin on 1 Oct 2020
👍1

@marklundin With a function named tmpdir I doubt it's permanent, but if this works, then it would be good to know how temporary actually tmpdir is, yeah... 🤔

Svish picture Svish on 1 Oct 2020

Any updates on this? I'm wondering why it works in getInitialProps, but not in API routes 🤷‍♂️

My current workaround 

const data = await import(`../../../../data/de/my-nice-file.json`);
res.json(data.default);
wiesson picture wiesson on 13 Oct 2020
👍2

currently having this issue in API routes too

mattvb91 picture mattvb91 on 14 Oct 2020

currently having this issue in API routes too

There are a few working solutions here, what problem are you having, specifically?

BrunoBernardino picture BrunoBernardino on 14 Oct 2020

I'm struggling to get this working even with suggestions from this thread. My use-case is I'm writing a guide and want to show the source-code for the component alongside the component itself. My method for doing this is using fs to load the component's jsx file inside getServerSideProps and passing the string value of the file contents as a prop.

I was feeling over the moon about having it working locally, but then when I went to deploy it, the joy has gone :(

Please see: https://github.com/ElGoorf/i18next-guide/blob/fix-example-components/pages/plurals.jsx

ElGoorf picture ElGoorf on 17 Oct 2020

@ElGoorf your problem is that the public files are on an edge, and the functions are on a lambda. Now, @vercel/next still doesn't allow for includeFiles, so the easiest way for you to get it working would be to use a lambda function with it.

Here's some sample code that helped others here: https://github.com/vercel/next.js/issues/8251#issuecomment-614220305

BrunoBernardino picture BrunoBernardino on 17 Oct 2020
👍1

Thanks @BrunoBernardino I didn't realise I'd missed the "x hidden items load more..." and thought I was going crazy from the thread losing meaning!

Unfortunately, I struggled with your solution, as it's the first time I've heard of Edge/Lambda, however, I found @balthild's solution was closer to what I was originally after before trying the node.fs method: https://github.com/vercel/next.js/issues/8251#issuecomment-634829189

ElGoorf picture ElGoorf on 17 Oct 2020

Great! Did you get it working? Or are you still having issues?

I'm not sure Vercel even uses that terminology, but by Edge I mean CDN, where static files are served from, and by lambda I mean the "backend" functions that get called from the API routes, which are isolated like AWS Lambda functions.

BrunoBernardino picture BrunoBernardino on 17 Oct 2020
👍1

Hey,

Any update on writing to files using next.js on vercel? I can read no problem. Using the const logoPath = path.resolve('./public/logo.png')

I'm attempting to overwrite the public/sitemap.xml file (due to the size limits on vercel) I can only return it without error as a static file in the public folder. I have previously implemented the sitemap with zlib and streaming the response but it seems to wait until the stream is finished and then return it. This doesn't hit the size limitation error, but unfortunately it's very slow. I'm open to any suggestions people might have. The sitemap is built from an API call to a separate backend and needs to be updated regularly.

Things I have attempted :

  • Zipping and streaming the xml - works but very slow.
  • Building and returning sitemap from an api function, unfortunately this hits the size limit.
  • Reading static xml file from public folder (works) regardless of size, but not updatable.
  • Test writing to this file, doesn't work. Test writing to any file/folder doesnt work
  • Test returning static xml file from "api" function, size error. This works locally.
  • Test returning static xml file from "page" getServerSideProp size error. This works locally.
  • Would appreciate any ideas at all?
emomooney picture emomooney on 23 Oct 2020

Hey @emomooney, I don't imagine Vercel ever allowing to write files in a function (even for caching), since the main "advantage" of serverless is its statelessness, and that would add state to it, so I think you'll need to use the edge/cdn for it.

I have previously implemented the sitemap with zlib and streaming the response but it seems to wait until the stream is finished and then return it.

I'm curious if you were just experiencing this slowness for subsequent calls, or just the first, for the cold start? I imagine this was an API call to Vercel via a next.js api function, or a dedicated lambda, similar to what I do here.

If it was and it was still too slow, is your "separate backend" outside of Vercel? If so, you can potentially use it to build a sitemap.xml file and vercel --prod it into a domain, basically "caching" the file to be readable and accessible, and you'd just need to update the robots.txt to link the sitemap to another domain/subdomain.

BrunoBernardino picture BrunoBernardino on 24 Oct 2020
Was this page helpful?
0 / 5 - 0 ratings

Related issues

renatorib picture renatorib  ·  3Comments
olifante picture olifante  ·  3Comments
havefive picture havefive  ·  3Comments
ghost picture ghost  ·  3Comments
formula349 picture formula349  ·  3Comments