Knex: "Module not found" errors when using knex with webpack

I had this issue, but adding a Regex to my externals in my webpack config fixed it:

const commonLoaders = [
  { test: /\.js$/, loader: 'babel', exclude: [/node_modules/] },
  { test: /\.json$/, loader: 'json' },
];

module.exports = [
  {
    name: 'browser',
    entry: './js/entry.js',
    output: {
      path: './build',
      filename: 'bundle.js',
    },
    module: {
      loaders: commonLoaders.concat([
        { test: /\.styl$/, loader: 'style!css!stylus' },
      ]),
    },
  },
  {
    name: 'server',
    entry: './server.js',
    target: 'node',
    output: {
      path: './build',
      filename: 'server.js',
      libraryTarget: 'commonjs2',
    },
    externals: [
      {
        'socket.io': 'socket.io',
      },
      /^[a-z\-0-9]+$/,
    ],
    module: {
      loaders: commonLoaders.concat([
        { test: /\.styl$/, loader: 'css!stylus' },
      ]),
    },
  },
];

+1

@joeketchi Ignoring the node_modules directory doesn't really solve the issue, we're just working around the issue. It would be awesome if the code would do these requires in a more intelligent way that doesn't throw errors.

Also ran into this. Not quite sure why ignoring node_modules doesn't work.

also ran into this. what works for me;

externals: {
  knex: 'commonjs knex'
}

I can't comment specifically on the mariasql problem but what solved my problem is to define all the drivers I don't use as external. For instance, I use the mysql2 driver for my project so I have this in externals:

  externals: {
    // Possible drivers for knex - we'll ignore them
    'sqlite3': 'sqlite3',
    'mariasql': 'mariasql',
    'mssql': 'mssql',
    'mysql': 'mysql',
    'oracle': 'oracle',
    'strong-oracle': 'strong-oracle',
    'oracledb': 'oracledb',
    'pg': 'pg',
    'pg-query-stream': 'pg-query-stream'
  }

The result is that knex gets bundled without errors or warnings

This issue seems to be just a problem how to setup webpack configuration when including knex... lets reopen if there are changes required in knex to support webpack better.

Actually, I don't think the issue should be closed yet, there are still unresolved problems with webpack, with migration and seed:

WARNING in ./node_modules/knex/lib/migrate/index.js
342:20-81 Critical dependency: the request of a dependency is an expression

WARNING in ./node_modules/knex/lib/migrate/index.js
447:18-49 Critical dependency: the request of a dependency is an expression

WARNING in ./node_modules/knex/lib/seed/index.js
113:13-74 Critical dependency: the request of a dependency is an expression

WARNING in ./node_modules/knex/lib/seed/index.js
150:11-24 Critical dependency: the request of a dependency is an expression

I couldn't fix these

@ifullgaz that doesnt seem to be an error, but just a warning that require() was not used with string literal? https://github.com/webpack/webpack/issues/196

@ifullgaz, I've tried your solution, but I got errors as below. Did you run into a similar issue?

  Error --------------------------------------------------
  Command failed: npm install
gyp ERR! UNCAUGHT EXCEPTION
gyp ERR! stack Error: spawn EACCES
gyp ERR! stack     at exports._errnoException (util.js:1018:11)
gyp ERR! stack     at ChildProcess.spawn (internal/child_process.js:319:11)
gyp ERR! stack     at exports.spawn (child_process.js:378:9)
gyp ERR! stack     at exports.execFile (child_process.js:143:15)

@brianhuangyl have you correct ownership etc. to the node_modules directory and all ist content (you have not ran npm install as root at any moment)?

     [EACCES]       Permission bits of the file mode do not permit the
            requested access, or search permission is denied on a
            component of the path prefix.

@ifullgaz I found the suggestion from https://www.laurivan.com/make-electron-work-with-knex-js/ fixed these errors:

WARNING in ./node_modules/knex/lib/migrate/index.js
342:20-81 Critical dependency: the request of a dependency is an expression

WARNING in ./node_modules/knex/lib/migrate/index.js
447:18-49 Critical dependency: the request of a dependency is an expression

WARNING in ./node_modules/knex/lib/seed/index.js
113:13-74 Critical dependency: the request of a dependency is an expression

WARNING in ./node_modules/knex/lib/seed/index.js
150:11-24 Critical dependency: the request of a dependency is an expression

Specifically, this part worked for me:

config.plugins = [
  // ...
  new NormalModuleReplacementPlugin(/\.\.\/migrate/, '../util/noop.js'),
  new NormalModuleReplacementPlugin(/\.\.\/seed/, '../util/noop.js'),
];

this was the ticket for me libraryTarget: 'commonjs2',

I looked a slightly different approach which allows webpack to process most require statements in the knex files, but will leave the non-literal requires alone so that the migration and seed files provided by the user of the library be found. Here is the approach I took:

config.module.rules = [
  {
    include: [
      /knex\/lib\/migrate\/index/,
      /knex\/lib\/seed\/index/
    ],
    loader: 'string-replace-loader',
    options: {
      search: 'require(\\([^\'"])',
      replace: '__non_webpack_require__$1',
      flags: 'g'
    }
  }
];

Hi folks, I was working to solve the problem of creating a complete webpack bundle for a service using knex and ran into this issue when trying to run migrations programmatically from that bundle. @mdlavin 's suggestion did get it to run the require but, it resulted in webpack not transforming the source file. It also meant I'd need to include that directory with the bundle out of band from the webpack bundle.

I figured out that you can use webpack's ContextReplacementPlugin to resolve the issue with migrations and seeds. Below is a sample from my config (written in typescript):

plugins: [
  //...
  fs.existsSync(path.join(params.targetPackageFilePath, 'migrations')) && new webpack.ContextReplacementPlugin(
    /knex\/lib\/migrate/,
    '.',
    fs.readdirSync(path.join(params.targetPackageFilePath, 'migrations'))
      .reduce((result, file) => {
        return Object.assign(result, {
          [path.join(params.targetPackageFilePath, 'migrations', file)]: path.relative(
            path.dirname(require.resolve('knex/lib/migrate')),
            path.join(params.targetPackageFilePath, 'migrations', file)
          )
        });
      }, {})),

  fs.existsSync(path.join(params.targetPackageFilePath, 'seeds')) && new webpack.ContextReplacementPlugin(
    /knex\/lib\/seed/,
    '.',
    fs.readdirSync(path.join(params.targetPackageFilePath, 'seeds'))
      .reduce((result, file) => {
        return Object.assign(result, {
          [path.join(params.targetPackageFilePath, 'seeds', file)]: path.relative(
            path.dirname(require.resolve('knex/lib/seed')),
            path.join(params.targetPackageFilePath, 'seeds', file)
          )
        });
      }, {}))
  //...
]

In my config params.targetPackageFiletPath is the resolved path to the package being compiled e.g. /Users/yourusername/yourproject. So if a project at that path has migrations and seeds directories the resulting plugin configurations would look something like this:

  fs.existsSync(path.join(params.targetPackageFilePath, 'migrations')) && new webpack.ContextReplacementPlugin(
    /knex\/lib\/migrate/,
    '.',
    {
      '/Users/yourusername/yourproject/201706261234_migration.ts': '../../../migrations/201706261234_migration.ts'
    }),

  fs.existsSync(path.join(params.targetPackageFilePath, 'seeds')) && new webpack.ContextReplacementPlugin(
    /knex\/lib\/seed/,
    '.',
    {
      '/Users/yourusername/yourproject/seeds/201706261234_first-seed.ts': '../../../seeds/201706261234_first-seed.ts'
    }),

To break this down a little, the issue is that knex's migration and seed code calls require to load your migrations and seeds, as it does here https://github.com/tgriesser/knex/blob/master/src/migrate/index.js#L54. It passes the result of a function/expression to require with no hard-coded string prefix or suffix so, when webpack attempts to figure out what files that might actually load it doesn't have enough information to determine that. These plugins provide that needed information to webpack.

I'm not sure if Knex could be changed to make these unnecessary when bundling with webpack. Maybe by doing the requires in user-land?

Anyway, thought I'd share a solution that worked for me.

Any suggestion for this issue in Angular?
I try using express project, it is normal (not showing any error). But when I use it in Angular project, it shows the error:

ERROR in ./node_modules/knex/lib/dialects/oracle/utils.js Module not found: Error: Can't resolve 'crypto' in '/Users/marudits/Documents/CODE/PROJECT/MOJOMARET/app-member/node_modules/knex/lib/dialects/oracle' ERROR in ./node_modules/knex/lib/dialects/mssql/index.js Module not found: Error: Can't resolve 'mssql/package.json' in '/Users/marudits/Documents/CODE/PROJECT/MOJOMARET/app-member/node_modules/knex/lib/dialects/mssql' ERROR in ./node_modules/knex/lib/runner.js Module not found: Error: Can't resolve 'stream' in '/Users/marudits/Documents/CODE/PROJECT/MOJOMARET/app-member/node_modules/knex/lib' ERROR in ./node_modules/knex/lib/dialects/oracledb/index.js Module not found: Error: Can't resolve 'stream' in '/Users/marudits/Documents/CODE/PROJECT/MOJOMARET/app-member/node_modules/knex/lib/dialects/oracledb'

where i can do the configuration to remove this error?
could it be set up on angular.json or tsconfig.json file?

In webpack most probably.

This fixed it for me (credit to @ifullgaz ) I had to add mysql2.

    externals: {
        // Possible drivers for knex - we'll ignore them
        sqlite3: 'sqlite3',
        mysql2: 'mysql2',
        mariasql: 'mariasql',
        mysql: 'mysql',
        oracle: 'oracle',
        'strong-oracle': 'strong-oracle',
        oracledb: 'oracledb',
        pg: 'pg',
        'pg-query-stream': 'pg-query-stream'
    }

Then just remove the driver that you're using from this list and webpack will include it in the build.

@ifullgaz I found the suggestion from https://www.laurivan.com/make-electron-work-with-knex-js/ fixed these errors:

WARNING in ./node_modules/knex/lib/migrate/index.js
342:20-81 Critical dependency: the request of a dependency is an expression

WARNING in ./node_modules/knex/lib/migrate/index.js
447:18-49 Critical dependency: the request of a dependency is an expression

WARNING in ./node_modules/knex/lib/seed/index.js
113:13-74 Critical dependency: the request of a dependency is an expression

WARNING in ./node_modules/knex/lib/seed/index.js
150:11-24 Critical dependency: the request of a dependency is an expression

Specifically, this part worked for me:

config.plugins = [
  // ...
  new NormalModuleReplacementPlugin(/\.\.\/migrate/, '../util/noop.js'),
  new NormalModuleReplacementPlugin(/\.\.\/seed/, '../util/noop.js'),
];

This worked great for me. I did this:

plugins: [
    new webpack.NormalModuleReplacementPlugin(/m[sy]sql2?|oracle(db)?|sqlite3/, "node-noop"),
    new webpack.NormalModuleReplacementPlugin(/\.\.\/migrate/, "node-noop"),
    new webpack.NormalModuleReplacementPlugin(/\.\.\/seed/, "node-noop")
  ]

1) I couldn't get the relative import of noop.js to work because it was trying to import it deep inside knex. I just installed this simple node-noop lib and that worked fine.
2) We use Postgres, so the regex in the first line matches all the other ones it looks for. Modify to your heart's content.

2019/10 I managed to get it working, see my answer in another issue https://github.com/tgriesser/knex/issues/1446#issuecomment-537715431

I put pg dependency as "external" and it solved the problem thanks a lot, does it mean that webpack parse the back-end dependencies ?

Hi folks, I was working to solve the problem of creating a complete webpack bundle for a service using knex and ran into this issue when trying to run migrations programmatically from that bundle. @mdlavin 's suggestion did get it to run the require but, it resulted in webpack not transforming the source file. It also meant I'd need to include that directory with the bundle out of band from the webpack bundle.

I figured out that you can use webpack's ContextReplacementPlugin to resolve the issue with migrations and seeds. Below is a sample from my config (written in typescript):

plugins: [
  //...
  fs.existsSync(path.join(params.targetPackageFilePath, 'migrations')) && new webpack.ContextReplacementPlugin(
    /knex\/lib\/migrate/,
    '.',
    fs.readdirSync(path.join(params.targetPackageFilePath, 'migrations'))
      .reduce((result, file) => {
        return Object.assign(result, {
          [path.join(params.targetPackageFilePath, 'migrations', file)]: path.relative(
            path.dirname(require.resolve('knex/lib/migrate')),
            path.join(params.targetPackageFilePath, 'migrations', file)
          )
        });
      }, {})),

  fs.existsSync(path.join(params.targetPackageFilePath, 'seeds')) && new webpack.ContextReplacementPlugin(
    /knex\/lib\/seed/,
    '.',
    fs.readdirSync(path.join(params.targetPackageFilePath, 'seeds'))
      .reduce((result, file) => {
        return Object.assign(result, {
          [path.join(params.targetPackageFilePath, 'seeds', file)]: path.relative(
            path.dirname(require.resolve('knex/lib/seed')),
            path.join(params.targetPackageFilePath, 'seeds', file)
          )
        });
      }, {}))
  //...
]

In my config params.targetPackageFiletPath is the resolved path to the package being compiled e.g. /Users/yourusername/yourproject. So if a project at that path has migrations and seeds directories the resulting plugin configurations would look something like this:

  fs.existsSync(path.join(params.targetPackageFilePath, 'migrations')) && new webpack.ContextReplacementPlugin(
    /knex\/lib\/migrate/,
    '.',
    {
      '/Users/yourusername/yourproject/201706261234_migration.ts': '../../../migrations/201706261234_migration.ts'
    }),

  fs.existsSync(path.join(params.targetPackageFilePath, 'seeds')) && new webpack.ContextReplacementPlugin(
    /knex\/lib\/seed/,
    '.',
    {
      '/Users/yourusername/yourproject/seeds/201706261234_first-seed.ts': '../../../seeds/201706261234_first-seed.ts'
    }),

To break this down a little, the issue is that knex's migration and seed code calls require to load your migrations and seeds, as it does here https://github.com/tgriesser/knex/blob/master/src/migrate/index.js#L54. It passes the result of a function/expression to require with no hard-coded string prefix or suffix so, when webpack attempts to figure out what files that might actually load it doesn't have enough information to determine that. These plugins provide that needed information to webpack.

I'm not sure if Knex could be changed to make these unnecessary when bundling with webpack. Maybe by doing the requires in user-land?

Anyway, thought I'd share a solution that worked for me.

Save my days, thanks so much.

For those looking for a solution, this one worked for me using knex@^0.21.1, I wanted mysql2 driver retained. I took 2 existing solutions, which worked together for me.

In webpack config merge in a modified version of https://github.com/knex/knex/issues/1128#issuecomment-515573845

externals: [
      {
        // Possible drivers for knex - we'll ignore them
       // comment the one YOU WANT to use
        sqlite3: 'sqlite3',
        // mysql2: 'mysql2', // << using this one
        mariasql: 'mariasql',
        mysql: 'mysql',
        mssql: 'mssql',
        oracle: 'oracle',
        'strong-oracle': 'strong-oracle',
        oracledb: 'oracledb',
        pg: 'pg',
        'pg-query-stream': 'pg-query-stream',
      }

Also add in a webpack plugin, that will exclude the offending packages
ref https://github.com/knex/knex/issues/3130#issuecomment-573293311

I modified the original to correctly exclude in my use case,including the correct set of mssql sub-dirs & also to allow some pg packages that are required for Knex to work

    new webpack.IgnorePlugin(
        new RegExp('^(mssql*|mariasql|.oracle.|sqlite3|mssql/.*|tedious|node-pre-gyp)$')
      ),