Yarn: `yarn install --production` doesn't install correct dependencies

@Daniel15 I guess this is because of nodemon is having latest version of minimatch.

The linker function currently takes in both deps and dev deps into it. For argument production, this should be prevented.

Even on normal yarn install with out production argument. Only latest version is installed in the actual path. This has to checked too.

I'm getting a similar problem when running yarn install --production and then trying to run my build using webpack (running yarn install works fine).

> NODE_ENV=production webpack -p --config webpack/production.config.js

module.js:457
    throw err;
    ^

Error: Cannot find module 'graceful-fs'
    at Function.Module._resolveFilename (module.js:455:15)
    at Function.Module._load (module.js:403:25)
    at Module.require (module.js:483:17)
    at require (internal/module.js:20:19)

and if I recall correctly, previous attempts showed similar errors with another package (not only graceful-fs)

I'm getting very similar as well... yarn install works just fine. but with the --production flag I get this:

> yarn install --production

yarn install v0.15.1
error npm-shrinkwrap.json found. This will not be updated or respected. See [TODO] for more information.
[1/4] Resolving packages...
[2/4] Fetching packages...
warning [email protected]: The engine "rhino" appears to be invalid.
warning [email protected]: The platform "win32" is incompatible with this module.
info "[email protected]" is an optional dependency and failed compatibility check. Excluding it from installation.
warning [email protected]: The engine "rhino" appears to be invalid.
warning [email protected]: The engine "rhino" appears to be invalid.
[3/4] Linking dependencies...
[4/4] Building fresh packages...
[1/1] ⠐ node-sass:     at Module.require (module.js:367:17)
[-/1] ⠐ waiting...
[-/1] ⠐ waiting...
[-/1] ⠐ waiting...
error C:\vagrant\ebroker-quoteengine\node_modules\node-sass: Command failed.
Exit code: 1
Command: C:\WINDOWS\system32\cmd.exe
Arguments: /d /s /c node scripts/install.js
Directory: C:\vagrant\ebroker-quoteengine\node_modules\node-sass
Output:
module.js:341
    throw err;
    ^

Error: Cannot find module 'tough-cookie'
    at Function.Module._resolveFilename (module.js:339:15)
    at Function.Module._load (module.js:290:25)
    at Module.require (module.js:367:17)
    at require (internal/module.js:16:19)
    at Object.<anonymous> (C:\vagrant\ebroker-quoteengine\node_modules\node-sass\node_modules\request\lib\cookies.js:3:13)
    at Module._compile (module.js:413:34)
    at Object.Module._extensions..js (module.js:422:10)
    at Module.load (module.js:357:32)
    at Function.Module._load (module.js:314:12)
    at Module.require (module.js:367:17)
    at SpawnError (C:\Users\nathan.white\AppData\Roaming\npm\node_modules\yarnpkg\lib\errors.js:18:1)
    at ChildProcess.<anonymous> (C:\Users\nathan.white\AppData\Roaming\npm\node_modules\yarnpkg\lib\util\child.js:107:15)
    at emitTwo (events.js:100:13)
    at ChildProcess.emit (events.js:185:7)
    at maybeClose (internal/child_process.js:827:16)
    at Socket.<anonymous> (internal/child_process.js:319:11)
    at emitOne (events.js:90:13)
    at Socket.emit (events.js:182:7)
    at Pipe._onclose (net.js:471:12)

Can reproduce a similar issue with:

npm init --yes
yarn add --dev nodemon
yarn add gulp
rm -rf node_modules
yarn install --production

This will install is-glob but not its dependency is-extglob:

> yarn why is-glob
yarn why v0.16.0
# ...
info Reasons this module exists
   - "nodemon#chokidar" depends on it
   - "gulp#liftoff#findup-sync" depends on it

> yarn why is-extglob
yarn why v0.16.0
#  ...
info This module exists because "nodemon#chokidar#is-glob" depends on it.

It seems to "forget" the gulp#liftoff dependency path while traversing..?

EDIT: Smaller example:

npm init --yes
yarn add --dev [email protected]
yarn add [email protected]
rm -rf node_modules
yarn --prod
node -e "require('is-glob')"

Also confirmed that removing devDependencies before running yarn --prod installs the correct dependency tree.

My software team ran into this issue, specifically with the package prr which is a dependency of both less and pouchdb. Many other packages were also missing from the --production build but prr was the first to cause a failure in our product. This issue has been a showstopper for us as the size of our installer would significantly increase if we included the dev packages, so we've returned to using npm.

FWIW: I can work around the issue by deleting devDependencies section from package.json prior to running yarn in production.

as @gihrig said running npm prune --production can remove the devDependencies which helps to workaround this issue.

as @gihrig said running npm prune --production can remove the devDependencies which helps to workaround this issue.

Yarn's main advantage over npm is a deterministic node_modules dir i.e. same in dev, CI, and production out of the box. Does running npm prune --production yield that same behavior?

My current workaround is to just install devDependencies in production as well. Disk is cheap (especially on AWS) and a deterministic install is much more important to me than disk space. So my "workaround" is to just act like yarn --production does not exist right now .

@tanx npm prune --production just remove the devDependencies. And in my tests always the same modules were removed. On the other hand, yes disk space is cheap, so maybe acting as if yarn --production does not exits is a better workaround :)

@tanx npm prune --production just remove the devDependencies. And in my tests always the same modules were removed.

This is precisely the "works on my machine" mentality described in the Yarn blog post. The problem is you're letting npm change the state of node_modules without yarn's integrity check via theyarn.lock file.

Hopefully the workarounds discussed will be rendered moot soon by an yarn update to respect dev vs production deps. In the meantime, indeed there is much to groan about with the "npm prune" post-processing hack.

The yarn why thing described above is unrelated afaict. It looks to be just a side effect of how the why code is searching for the packages.

Tried to find a nice way of doing this without adding an additional pass to propagate the visibility after the graph was walked once. Not sure if it would be acceptable to just split out the visibility into a separate step..?

There's some interesting edge cases as well, it's not just about properly resolving visibility:

• A is an optional dependency of a production dependency
• B is a non-optional dependency of a dev dependency
• C is a non-optional dependency of both

In that case the optional flag of C depends on dev vs. prod. In dev it would be non-optional, in prod it would be optional. Just inheriting the optional flag from one of the parents (or always inheriting it from the prod parent) could lead to weirdness.

This is still not fixed in 0.17.2 😢

Repro: https://gist.github.com/SimenB/2b179f3b6bca73ba824e1273ea38aed3

yarn

node index.js # works

yarn --prod

node index.js # explodes

/cc @jkrems

Doesn't seem fixed for me either in 0.17.2 (HearthSim/Joust#169).

I am going to reopen it as it can be easily reproduced with @SimenB's instructions.

@wyze The problem might be the install itself, and not the pruning?

rm -rf node_modules/ && yarn && npm prune --production && node index.js fails with the same error.

rm -rf node_modules/ && npm i && npm prune --production && node index.js works though.

I guess yarn and npm are not meant to be used at the same time though, so might be a coincidence that it produces the same error.

Diffing node_modules after npm i and yarn shows that yarn doesn't output "_requiredBy", probably why npm prune messes up after a yarn install. That info is available in the lockfile though, so it shouldn't be an issue, should it?

Same issue here, we were testing production building on docker and find out that with yarn --production the package mime was missing even if the parent module send (used by express) was installed.

I think this issue should be handled with maximum priority because cause unpredictable builds.

As a workaround, I am simply removing the devDependencies section from package.json in my build script.

$ jq 'del(.devDependencies)' package.json > tmp.json && mv tmp.json package.json

Following @dy-dx advice i've written a custom entrypoint for Docker to monkeypatch this issue during development:

First of all you should install jq in your Dockerfile adding this line somewhere:

RUN apt-get update && \
    apt-get install -y jq

Then add this script somewhere and use it as Dockerfile [ENTRYPOINT] or docker-compose entrypoint entrypoint.sh

Use your preferred command to Dockerfile [CMD] or docker-compose command E.g. npm start

The same script could be used in CI with some edit to build the image

@SimenB Can you please remove the node_modules from your entries-test package and try?

https://registry.yarnpkg.com/entries-test/-/entries-test-1.0.1.tgz#1bf192e414ceadd0cf4b77b3969df32de2985d50

Extract v1.0.1 tar ball, there's a node_modules folder with define-properties and other modules. And, none of them have any *.js file.

@torifat Huh, how did that even make it in? It shouldn't be possible to include a node_modules without using bundledDependencies...
Will try to push a clean one (I've deleted the projects, will have to recreate).

@torifat It's yarn's fault it looks like.

$ mkdir some-dir && cd some-dir && yarn init -y && yarn add object.entries && yarn pack && tar -ztvf some-dir-v1.0.0.tgz
drwxr-xr-x  0 0      0           0 Nov 27 10:36 package
-rw-r--r--  0 0      0         972 Oct 15  2015 package/node_modules/define-properties/CHANGELOG.md
-rw-r--r--  0 0      0        1080 Oct 15  2015 package/node_modules/define-properties/LICENSE
-rw-r--r--  0 0      0        2725 Oct 15  2015 package/node_modules/define-properties/README.md
-rw-r--r--  0 0      0        1593 Oct 15  2015 package/node_modules/define-properties/package.json
-rw-r--r--  0 0      0        3798 Aug 21 11:09 package/node_modules/es-abstract/CHANGELOG.md
-rw-r--r--  0 0      0        1080 Jul 29  2015 package/node_modules/es-abstract/LICENSE
-rw-r--r--  0 0      0        1812 Aug 13  2015 package/node_modules/es-abstract/README.md
-rw-r--r--  0 0      0        1989 Aug 21 11:09 package/node_modules/es-abstract/package.json
-rw-r--r--  0 0      0        1207 Jan  4  2016 package/node_modules/es-to-primitive/CHANGELOG.md
-rw-r--r--  0 0      0        1082 Nov  1  2015 package/node_modules/es-to-primitive/LICENSE
-rw-r--r--  0 0      0        2180 Nov  1  2015 package/node_modules/es-to-primitive/README.md
-rw-r--r--  0 0      0        1558 Jan  4  2016 package/node_modules/es-to-primitive/package.json
-rw-r--r--  0 0      0        1074 Sep 22  2014 package/node_modules/foreach/LICENSE
-rw-r--r--  0 0      0         593 Sep 22  2014 package/node_modules/foreach/Readme.md
-rw-r--r--  0 0      0        1297 Sep 22  2014 package/node_modules/foreach/package.json
-rw-r--r--  0 0      0        1052 Feb 14  2016 package/node_modules/function-bind/LICENSE
-rw-r--r--  0 0      0        1488 Feb 14  2016 package/node_modules/function-bind/README.md
-rw-r--r--  0 0      0        1619 Feb 14  2016 package/node_modules/function-bind/package.json
-rw-r--r--  0 0      0        1060 Jul 24  2015 package/node_modules/has/LICENSE-MIT
-rw-r--r--  0 0      0         239 Jul 24  2015 package/node_modules/has/README.mkd
-rw-r--r--  0 0      0         782 Jul 24  2015 package/node_modules/has/package.json
-rw-r--r--  0 0      0        1839 Feb 28  2016 package/node_modules/is-callable/CHANGELOG.md
-rw-r--r--  0 0      0        1082 May 19  2015 package/node_modules/is-callable/LICENSE
-rw-r--r--  0 0      0        1978 Aug 12  2015 package/node_modules/is-callable/README.md
-rw-r--r--  0 0      0        1983 Feb 28  2016 package/node_modules/is-callable/package.json
-rw-r--r--  0 0      0         421 Sep 27  2015 package/node_modules/is-date-object/CHANGELOG.md
-rw-r--r--  0 0      0        1082 Mar 13  2015 package/node_modules/is-date-object/LICENSE
-rw-r--r--  0 0      0        1751 Aug 12  2015 package/node_modules/is-date-object/README.md
-rw-r--r--  0 0      0        1420 Sep 27  2015 package/node_modules/is-date-object/package.json
-rw-r--r--  0 0      0         482 Jan 30  2015 package/node_modules/is-regex/CHANGELOG.md
-rw-r--r--  0 0      0        1081 Jan 15  2014 package/node_modules/is-regex/LICENSE
-rw-r--r--  0 0      0        1623 Jan 28  2015 package/node_modules/is-regex/README.md
-rw-r--r--  0 0      0        1512 Jan 30  2015 package/node_modules/is-regex/package.json
-rw-r--r--  0 0      0         121 Jan 26  2015 package/node_modules/is-symbol/CHANGELOG.md
-rw-r--r--  0 0      0        1082 Jan 24  2015 package/node_modules/is-symbol/LICENSE
-rw-r--r--  0 0      0        1469 Jan 24  2015 package/node_modules/is-symbol/README.md
-rw-r--r--  0 0      0        1214 Jan 26  2015 package/node_modules/is-symbol/package.json
-rw-r--r--  0 0      0        6992 Jul  5 19:14 package/node_modules/object-keys/CHANGELOG.md
-rw-r--r--  0 0      0        1080 Oct 15  2015 package/node_modules/object-keys/LICENSE
-rw-r--r--  0 0      0        2460 Oct 15  2015 package/node_modules/object-keys/README.md
-rw-r--r--  0 0      0        1955 Jul  5 19:14 package/node_modules/object-keys/package.json
-rw-r--r--  0 0      0         560 Oct  6  2015 package/node_modules/object.entries/CHANGELOG.md
-rw-r--r--  0 0      0        1082 Sep  2  2015 package/node_modules/object.entries/LICENSE
-rw-r--r--  0 0      0        2339 Sep  2  2015 package/node_modules/object.entries/README.md
-rw-r--r--  0 0      0        1636 Oct  6  2015 package/node_modules/object.entries/package.json
-rw-r--r--  0 0      0         145 Nov 27 10:36 package/package.json

Using npm pack works as expected (in same dir).

$ npm pack && tar -ztvf some-dir-1.0.0.tgz
-rw-r--r--  0 501    20        145 Nov 27 10:36 package/package.json
-rw-r--r--  0 501    20       2460 Nov 27 10:36 package/yarn.lock

Seems like Yarn is so hell bent on including changelog, readme and package.json it even includes them from node_modules...

Using [email protected]

@torifat Published 1.0.2 now (using npm to avoid the bug just mentioned), still the same issue

Opened up #2047 for the bug regarding node_modules, but it's a red herring in this issue, as my repro is still valid with a proper tarball published.

(sorry about the spam to subscribed people, I'll stop now)

@SimenB Thanks for your time. I have figured out the bug.

This appears to be related to #2104, which I just opened. The node_modules/.bin after the OP's install:

$ ll node_modules/.bin
total 16
lrwxr-xr-x  1 samuelreed  staff    22B Dec  1 11:16 forever -> ../forever/bin/forever
lrwxr-xr-x  1 samuelreed  staff   109B Dec  1 11:16 nodemon -> ../../../../../Library/Caches/Yarn/npm-nodemon-1.11.0-226c562bd2a7b13d3d7518b49ad4828a3623d06c/bin/nodemon.js

Fixed via #2116.

Has #2116 been merged in yet? I can't see it in the commit history. Seems premature to close a bunch of issues before a fix is available at least on master if not in a tagged release. Also, it looks like #2116 is failing all three checks. Am I missing something?

This is still an issue in v0.18.0, which includes (#2116).

yes, i can confirm that this issue is still present in 0.18.0

From what i see the #2116 should have introduced testing for this issue ( test.concurrent('--production flag ignores dev dependencies'... or i'm wrong?

The test doesn't verify correct behaviour for transitive dependencies:
In my case the issue is a shared dependant (lru-cache) between a prod (minimatch v2.0.0) and a dev (useragent v2.1.9) dependency. That shared dependant is not installed in --production, even though the prod dependency requires it.

@beheh I don't see were minimatch uses lru-cache, maybe that's why is not installed in production?

I'm doing some tests with 0.18.0

dep { A->B }
devDep { B }
OK
A,B are installed.

dep { A->C->D }
devDep { B->C->D }
OK
A,C,D are installed.

dep { E->A->C->D }
devDep { B->C->D }
KO
E,A,C are installed but D is missing.

this is the case illustrated by @SimenB

"dependencies": {
    "entries-test": "^1.0.1"
  },
  "devDependencies": {
    "object.values": "^1.0.3"
  }

@SharpEdgeMarshall Thanks for testing. I will add it as a test case.

@torifat might consider reopening this as well

@SharpEdgeMarshall Tried the following and it's working. Need to figure out the actual problem.

@SimenB need to verify before reopening. This might happen due optionalDependencies too. Which has another open issue.

@torifat my repro still happens: https://github.com/yarnpkg/yarn/issues/761#issuecomment-260975012

EDIT: Which has no optional deps, grep optional yarn.lock exits with 1

It now fails on Error: Cannot find module 'object-keys' instead of missing define-properties though.

$ yarn why object-keys
yarn why v0.18.0
[1/4] 🤔  Why do we have the module "object-keys"...?
[2/4] 🚚  Initialising dependency graph...
[3/4] 🔍  Finding dependency...
[4/4] 🚡  Calculating file sizes...
info This module exists because "object.values#define-properties" depends on it.
✨  Done in 0.09s.

Seems like it now handles one level deeper, but then fails

@SimenB Just tried with your:

{
  "dependencies": {
    "entries-test": "^1.0.1"
  },
  "devDependencies": {
    "object.values": "^1.0.3"
  }
}

And, it's working fine for me. Can you do a yarn cache clean and try again?

Nope, fails

@SimenB Can you share your yarn.lock file?

$ rm -rf node_modules && rm yarn.lock && yarn cache clean && yarn && node index.js && yarn --prod && node index.js
yarn cache v0.18.0
success Cleared cache.
✨  Done in 0.07s.
yarn install v0.18.0
info No lockfile found.
[1/4] 🔍  Resolving packages...
[2/4] 🚚  Fetching packages...
[3/4] 🔗  Linking dependencies...
[4/4] 📃  Building fresh packages...
success Saved lockfile.
✨  Done in 0.92s.
yarn install v0.18.0
[1/4] 🔍  Resolving packages...
[2/4] 🚚  Fetching packages...
[3/4] 🔗  Linking dependencies...
[4/4] 📃  Building fresh packages...
✨  Done in 0.18s.
module.js:471
    throw err;
    ^

Error: Cannot find module 'object-keys'
    at Function.Module._resolveFilename (module.js:469:15)
    at Function.Module._load (module.js:417:25)
    at Module.require (module.js:497:17)
    at require (internal/module.js:20:19)
    at Object.<anonymous> (/Users/simbekkh/repos/ugh/node_modules/define-properties/index.js:3:12)
    at Module._compile (module.js:570:32)
    at Object.Module._extensions..js (module.js:579:10)
    at Module.load (module.js:487:32)
    at tryModuleLoad (module.js:446:12)
    at Function.Module._load (module.js:438:3)

For me SimenB's example doesn't work with 0.18.0 even after yarn cache clean

# THIS IS AN AUTOGENERATED FILE. DO NOT EDIT THIS FILE DIRECTLY.
# yarn lockfile v1


define-properties@^1.1.2:
  version "1.1.2"
  resolved "http://npm.office.crweb.it/define-properties/-/define-properties-1.1.2.tgz#83a73f2fea569898fb737193c8f873caf6d45c94"
  dependencies:
    foreach "^2.0.5"
    object-keys "^1.0.8"

entries-test@^1.0.1:
  version "1.0.2"
  resolved "http://npm.office.crweb.it/entries-test/-/entries-test-1.0.2.tgz#f1039aba3a2effc9c3a56b6b1180694b2789e4d5"
  dependencies:
    object.entries "^1.0.3"

es-abstract@^1.6.1:
  version "1.6.1"
  resolved "http://npm.office.crweb.it/es-abstract/-/es-abstract-1.6.1.tgz#bb8a2064120abcf928a086ea3d9043114285ec99"
  dependencies:
    es-to-primitive "^1.1.1"
    function-bind "^1.1.0"
    is-callable "^1.1.3"
    is-regex "^1.0.3"

es-to-primitive@^1.1.1:
  version "1.1.1"
  resolved "http://npm.office.crweb.it/es-to-primitive/-/es-to-primitive-1.1.1.tgz#45355248a88979034b6792e19bb81f2b7975dd0d"
  dependencies:
    is-callable "^1.1.1"
    is-date-object "^1.0.1"
    is-symbol "^1.0.1"

foreach@^2.0.5:
  version "2.0.5"
  resolved "http://npm.office.crweb.it/foreach/-/foreach-2.0.5.tgz#0bee005018aeb260d0a3af3ae658dd0136ec1b99"

function-bind@^1.0.2, function-bind@^1.1.0:
  version "1.1.0"
  resolved "http://npm.office.crweb.it/function-bind/-/function-bind-1.1.0.tgz#16176714c801798e4e8f2cf7f7529467bb4a5771"

has@^1.0.1:
  version "1.0.1"
  resolved "http://npm.office.crweb.it/has/-/has-1.0.1.tgz#8461733f538b0837c9361e39a9ab9e9704dc2f28"
  dependencies:
    function-bind "^1.0.2"

is-callable@^1.1.1, is-callable@^1.1.3:
  version "1.1.3"
  resolved "http://npm.office.crweb.it/is-callable/-/is-callable-1.1.3.tgz#86eb75392805ddc33af71c92a0eedf74ee7604b2"

is-date-object@^1.0.1:
  version "1.0.1"
  resolved "http://npm.office.crweb.it/is-date-object/-/is-date-object-1.0.1.tgz#9aa20eb6aeebbff77fbd33e74ca01b33581d3a16"

is-regex@^1.0.3:
  version "1.0.3"
  resolved "http://npm.office.crweb.it/is-regex/-/is-regex-1.0.3.tgz#0d55182bddf9f2fde278220aec3a75642c908637"

is-symbol@^1.0.1:
  version "1.0.1"
  resolved "http://npm.office.crweb.it/is-symbol/-/is-symbol-1.0.1.tgz#3cc59f00025194b6ab2e38dbae6689256b660572"

object-keys@^1.0.8:
  version "1.0.11"
  resolved "http://npm.office.crweb.it/object-keys/-/object-keys-1.0.11.tgz#c54601778ad560f1142ce0e01bcca8b56d13426d"

object.entries@^1.0.3:
  version "1.0.4"
  resolved "http://npm.office.crweb.it/object.entries/-/object.entries-1.0.4.tgz#1bf9a4dd2288f5b33f3a993d257661f05d161a5f"
  dependencies:
    define-properties "^1.1.2"
    es-abstract "^1.6.1"
    function-bind "^1.1.0"
    has "^1.0.1"

object.values@^1.0.3:
  version "1.0.4"
  resolved "http://npm.office.crweb.it/object.values/-/object.values-1.0.4.tgz#e524da09b4f66ff05df457546ec72ac99f13069a"
  dependencies:
    define-properties "^1.1.2"
    es-abstract "^1.6.1"
    function-bind "^1.1.0"
    has "^1.0.1"

$ rm -rf node_modules && rm yarn.lock && yarn cache clean && yarn && node index.js && rm -rf node_modules && rm yarn.lock && yarn cache clean && yarn --prod && node index.js
yarn cache v0.18.0
success Cleared cache.
✨  Done in 0.07s.
yarn install v0.18.0
info No lockfile found.
[1/4] 🔍  Resolving packages...
[2/4] 🚚  Fetching packages...
[3/4] 🔗  Linking dependencies...
[4/4] 📃  Building fresh packages...
success Saved lockfile.
✨  Done in 0.93s.
yarn cache v0.18.0
success Cleared cache.
✨  Done in 0.07s.
yarn install v0.18.0
info No lockfile found.
[1/4] 🔍  Resolving packages...
[2/4] 🚚  Fetching packages...
[3/4] 🔗  Linking dependencies...
[4/4] 📃  Building fresh packages...
success Saved lockfile.
✨  Done in 0.76s.
module.js:471
    throw err;
    ^

Error: Cannot find module 'object-keys'
    at Function.Module._resolveFilename (module.js:469:15)
    at Function.Module._load (module.js:417:25)
    at Module.require (module.js:497:17)
    at require (internal/module.js:20:19)
    at Object.<anonymous> (/Users/simbekkh/repos/ugh/node_modules/define-properties/index.js:3:12)
    at Module._compile (module.js:570:32)
    at Object.Module._extensions..js (module.js:579:10)
    at Module.load (module.js:487:32)
    at tryModuleLoad (module.js:446:12)
    at Function.Module._load (module.js:438:3)

Lockfile:

# THIS IS AN AUTOGENERATED FILE. DO NOT EDIT THIS FILE DIRECTLY.
# yarn lockfile v1


define-properties@^1.1.2:
  version "1.1.2"
  resolved "https://registry.yarnpkg.com/define-properties/-/define-properties-1.1.2.tgz#83a73f2fea569898fb737193c8f873caf6d45c94"
  dependencies:
    foreach "^2.0.5"
    object-keys "^1.0.8"

entries-test@^1.0.1:
  version "1.0.2"
  resolved "https://registry.yarnpkg.com/entries-test/-/entries-test-1.0.2.tgz#f1039aba3a2effc9c3a56b6b1180694b2789e4d5"
  dependencies:
    object.entries "^1.0.3"

es-abstract@^1.6.1:
  version "1.6.1"
  resolved "https://registry.yarnpkg.com/es-abstract/-/es-abstract-1.6.1.tgz#bb8a2064120abcf928a086ea3d9043114285ec99"
  dependencies:
    es-to-primitive "^1.1.1"
    function-bind "^1.1.0"
    is-callable "^1.1.3"
    is-regex "^1.0.3"

es-to-primitive@^1.1.1:
  version "1.1.1"
  resolved "https://registry.yarnpkg.com/es-to-primitive/-/es-to-primitive-1.1.1.tgz#45355248a88979034b6792e19bb81f2b7975dd0d"
  dependencies:
    is-callable "^1.1.1"
    is-date-object "^1.0.1"
    is-symbol "^1.0.1"

foreach@^2.0.5:
  version "2.0.5"
  resolved "https://registry.yarnpkg.com/foreach/-/foreach-2.0.5.tgz#0bee005018aeb260d0a3af3ae658dd0136ec1b99"

function-bind@^1.0.2, function-bind@^1.1.0:
  version "1.1.0"
  resolved "https://registry.yarnpkg.com/function-bind/-/function-bind-1.1.0.tgz#16176714c801798e4e8f2cf7f7529467bb4a5771"

has@^1.0.1:
  version "1.0.1"
  resolved "https://registry.yarnpkg.com/has/-/has-1.0.1.tgz#8461733f538b0837c9361e39a9ab9e9704dc2f28"
  dependencies:
    function-bind "^1.0.2"

is-callable@^1.1.1, is-callable@^1.1.3:
  version "1.1.3"
  resolved "https://registry.yarnpkg.com/is-callable/-/is-callable-1.1.3.tgz#86eb75392805ddc33af71c92a0eedf74ee7604b2"

is-date-object@^1.0.1:
  version "1.0.1"
  resolved "https://registry.yarnpkg.com/is-date-object/-/is-date-object-1.0.1.tgz#9aa20eb6aeebbff77fbd33e74ca01b33581d3a16"

is-regex@^1.0.3:
  version "1.0.3"
  resolved "https://registry.yarnpkg.com/is-regex/-/is-regex-1.0.3.tgz#0d55182bddf9f2fde278220aec3a75642c908637"

is-symbol@^1.0.1:
  version "1.0.1"
  resolved "https://registry.yarnpkg.com/is-symbol/-/is-symbol-1.0.1.tgz#3cc59f00025194b6ab2e38dbae6689256b660572"

object-keys@^1.0.8:
  version "1.0.11"
  resolved "https://registry.yarnpkg.com/object-keys/-/object-keys-1.0.11.tgz#c54601778ad560f1142ce0e01bcca8b56d13426d"

object.entries@^1.0.3:
  version "1.0.4"
  resolved "https://registry.yarnpkg.com/object.entries/-/object.entries-1.0.4.tgz#1bf9a4dd2288f5b33f3a993d257661f05d161a5f"
  dependencies:
    define-properties "^1.1.2"
    es-abstract "^1.6.1"
    function-bind "^1.1.0"
    has "^1.0.1"

object.values@^1.0.3:
  version "1.0.4"
  resolved "https://registry.yarnpkg.com/object.values/-/object.values-1.0.4.tgz#e524da09b4f66ff05df457546ec72ac99f13069a"
  dependencies:
    define-properties "^1.1.2"
    es-abstract "^1.6.1"
    function-bind "^1.1.0"
    has "^1.0.1"

@SimenB Thanks. It seems, my cache had problem. After clearing my cache now it's failing. But, with a different error. Give me some time to investigate.

Btw, I recommend using https://github.com/Mottie/Octopatcher in this thread

Very practical with many lines of output

I'll stop spamming now

@SimenB Failing in v0.18.0 but can't reproduce in latest master.

UPDATE: Weird! It's failing again 😕

@torifat I can confirm that doesn't work with master (v0.19.0)
rm -rf node_modules && rm yarn.lock && ../yarn/bin/yarn cache clean && ../yarn/bin/yarn && node index.js && rm -rf node_modules && rm yarn.lock && ../yarn/bin/yarn cache clean && ../yarn/bin/yarn --prod && node index.js

yarn cache v0.19.0-0
success Cleared cache.
Done in 0.58s.
yarn install v0.19.0-0
info No lockfile found.
[1/4] Resolving packages...
[2/4] Fetching packages...
[3/4] Linking dependencies...
[4/4] Building fresh packages...
success Saved lockfile.
Done in 10.18s.
yarn cache v0.19.0-0
success Cleared cache.
Done in 0.09s.
yarn install v0.19.0-0
info No lockfile found.
[1/4] Resolving packages...
[2/4] Fetching packages...
[3/4] Linking dependencies...
[4/4] Building fresh packages...
success Saved lockfile.
Done in 4.26s.
module.js:457
    throw err;
    ^

Error: Cannot find module 'object-keys'
    at Function.Module._resolveFilename (module.js:455:15)
    at Function.Module._load (module.js:403:25)
    at Module.require (module.js:483:17)
    at require (internal/module.js:20:19)
    at Object.<anonymous> (/home/sharpedge/git/Utility/YarnBug/node_modules/define-properties/index.js:3:12)
    at Module._compile (module.js:556:32)
    at Object.Module._extensions..js (module.js:565:10)
    at Module.load (module.js:473:32)
    at tryModuleLoad (module.js:432:12)
    at Function.Module._load (module.js:424:3)

I have same problems (some packages do not installed with --production flag) with current stable 0.17.10:

curl -o- -L https://yarnpkg.com/install.sh | bash && ~/.yarn/bin/yarn install --production && rm -rf ~/.yarn

But when I try current nightly build Yarn 0.19.0-20161207.1241 all needed packages were installed correctly for my app:

wget https://yarnpkg.com/install.sh && chmod +x install.sh && ./install.sh --nightly && rm -f install.sh && ~/.yarn/bin/yarn install --production && rm -rf ~/.yarn

@SharpEdgeMarshall @SimenB can you try latest nightly build and confirm that problem persists.

Used in my docker containers: https://gist.github.com/nodkz/b843d65a3430a4f510e5f5eb0cc759d2

@nodkz 18 is out and stable (I think?), but as can be seen in the post directly above yours, master (as of 2 days ago at least) still has the bug

I think it still requires install yarn@rc:

'dist-tags': { rc: '0.18.0', latest: '0.17.10' },

That's new then, I got 0.18 a few days ago by just install. Anyways, the bug is still reproducible in 0.18.

@nodkz Repro is:

{
  "dependencies": {
    "entries-test": "^1.0.1"
  },
  "devDependencies": {
    "object.values": "^1.0.3"
  }
}

Yep, running into similar issues with our projects as well:

rm -rf package.json yarn.lock node_modules && npm init --yes && yarn add --dev nodemon && yarn add glob-stream && yarn --prod && node -p "require('glob-stream')"

Fails with both 0.18 and latest master branch.

Agreed. Still able to repro with latest.

I think my issue is similar or the same as this one. Build is failing on Heroku, but only for production environment. Caching is disabled.

Resolving node version ^7.2.1 via semver.io...
       Downloading and installing node 7.2.1...
       Using default npm version: 3.10.10
       Resolving yarn version (latest) via semver.io...
       Downloading and installing yarn (0.18.1)...
       Installed yarn 0.18.1
-----> Restoring cache
       Skipping cache restore (disabled by config)
-----> Building dependencies
       Installing node modules (yarn)
       yarn install v0.18.1
       [1/4] Resolving packages...
       [2/4] Fetching packages...
       warning [email protected]: The platform "linux" is incompatible with this module.
       info "[email protected]" is an optional dependency and failed compatibility check. Excluding it from installation.
       [3/4] Linking dependencies...
       [4/4] Building fresh packages...
       error /tmp/build_0e4ff736ad0f25dc816a47543687fefc/bbb7b6e751dde291f65dea175f41a26862eef28f/node_modules/bcrypt: Command failed.
       Exit code: 1
       Command: sh
       Arguments: -c node-pre-gyp install --fallback-to-build
       Directory: /tmp/build_0e4ff736ad0f25dc816a47543687fefc/bbb7b6e751dde291f65dea175f41a26862eef28f/node_modules/bcrypt
       Output:
       module.js:472
       throw err;
       ^

       Error: Cannot find module 'abbrev'
       at Function.Module._resolveFilename (module.js:470:15)
       at Function.Module._load (module.js:418:25)
       at Module.require (module.js:498:17)
       at require (internal/module.js:20:19)
       at Object.<anonymous> (/tmp/build_0e4ff736ad0f25dc816a47543687fefc/bbb7b6e751dde291f65dea175f41a26862eef28f/node_modules/nopt/lib/nopt.js:10:14)
       at Module._compile (module.js:571:32)
       at Object.Module._extensions..js (module.js:580:10)
       at Module.load (module.js:488:32)
       at tryModuleLoad (module.js:447:12)
       at Function.Module._load (module.js:439:3)
       info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.
-----> Build failed

I also have this issue as well. I'm still using yarn install for production as of the moment since yarn install --production does not correctly install the right dependencies.

I'm afraid i can't do that, as it's hardcoded to yarn install --production in the Heroku buildpack I think (ref https://github.com/heroku/heroku-buildpack-nodejs/issues/337)

@adamreisnz Sorry, I was referring to the original issue not yours.

For a workaround of your issue, I suggest you put all devDependencies to dependencies for now until this issue is fixed.

@dashmug Ah ok, no problem.

Anyway, I prefer using npm on Heroku for now until Yarn is more stable, rather than mucking around with hacky solutions. I've put yarn.lock in my .gitignore so it doesn't get uploaded to the repo/heroku. This way, I can still use Yarn locally, but it won't affect the builds on Heroku.

@adamreisnz That defeats your purpose for using yarn, don't you think?

@dashmug Not really, at least, not for us. I am not using it for locking package versions in place. We have very up-to-date dependencies and no issues around "works on my machine". My main reason for moving to yarn over npm was for the speed, which for complex apps with a lot of dependencies I saw going from 5 minutes for an npm install to 22 seconds with yarn.

As long as yarn is unstable, I can live with a slightly slower build process on Heroku, as long as I can use yarn locally for development and have fast installs :)

Yarn is advertised as being pretty much a drop-in replacement for npm. However, some of the issues we've encountered and that are still unresolved, like this one, prevent us from using it as such. Therefore I see it as an extra tool at present, which is useful for one thing, but not there yet for another. I have no doubt in time it will work great though :)

This appears to be fixed for me in 0.18.1.

Heroku was using 0.18.1 at the time when it failed, so not fixed yet afaik.

This issue's fixed for me on 0.18.1.

My earlier repro is fixed with 0.18.1 🎉

I'll try tomorrow with a real app

0.18.1 fixes my issues. I'm a happy camper 🎉

Tried 0.18.1 with a non-trivial app that was previously failing and it seems to work now! 🎉

I guess I'll give it another go soon :)

Sorry, still not working with 0.18.1;

-----> Node.js app detected
-----> Creating runtime environment

       NPM_CONFIG_LOGLEVEL=error
       NPM_CONFIG_PRODUCTION=true
       NODE_ENV=production
       NODE_MODULES_CACHE=true
-----> Installing binaries
       engines.node (package.json):  ^7.2.1
       engines.npm (package.json):   unspecified (use default)

       Resolving node version ^7.2.1 via semver.io...
       Downloading and installing node 7.2.1...
       Using default npm version: 3.10.10
       Resolving yarn version (latest) via semver.io...
       Downloading and installing yarn (0.18.1)...
       Installed yarn 0.18.1
-----> Restoring cache
       Loading 2 from cacheDirectories (default):
       - node_modules
       - bower_components (not cached - skipping)
-----> Building dependencies
       Installing node modules (yarn)
       yarn install v0.18.1
       [1/4] Resolving packages...
       [2/4] Fetching packages...
       warning [email protected]: The platform "linux" is incompatible with this module.
       info "[email protected]" is an optional dependency and failed compatibility check. Excluding it from installation.
       [3/4] Linking dependencies...
       [4/4] Building fresh packages...
       error /tmp/build_c86802dccae94b3fb074d3b88f3f63f2/9512deeed23c0eca48d68fb2c8850a28f76692ea/node_modules/bcrypt: Command failed.
       Exit code: 1
       Command: sh
       Arguments: -c node-pre-gyp install --fallback-to-build
       Directory: /tmp/build_c86802dccae94b3fb074d3b88f3f63f2/9512deeed23c0eca48d68fb2c8850a28f76692ea/node_modules/bcrypt
       Output:
       module.js:472
       throw err;
       ^

       Error: Cannot find module 'abbrev'
       at Function.Module._resolveFilename (module.js:470:15)
       at Function.Module._load (module.js:418:25)
       at Module.require (module.js:498:17)
       at require (internal/module.js:20:19)
       at Object.<anonymous> (/tmp/build_c86802dccae94b3fb074d3b88f3f63f2/9512deeed23c0eca48d68fb2c8850a28f76692ea/node_modules/nopt/lib/nopt.js:10:14)
       at Module._compile (module.js:571:32)
       at Object.Module._extensions..js (module.js:580:10)
       at Module.load (module.js:488:32)
       at tryModuleLoad (module.js:447:12)
       at Function.Module._load (module.js:439:3)
       info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.
-----> Build failed

       We're sorry this build is failing! You can troubleshoot common issues here:
       https://devcenter.heroku.com/articles/troubleshooting-node-deploys

       Some possible problems:

       - A module may be missing from 'dependencies' in package.json
       https://devcenter.heroku.com/articles/troubleshooting-node-deploys#ensure-you-aren-t-relying-on-untracked-dependencies

       - This module may be specified in 'devDependencies' instead of 'dependencies'
       https://devcenter.heroku.com/articles/nodejs-support#devdependencies

       Love,
       Heroku

 !     Push rejected, failed to compile Node.js app.
 !     Push failed

Setting NODE_MODULES_CACHE=false didn't help either.

Here is the dependency tree:

├─┬ [email protected]
│ └─┬ [email protected]
│   └─┬ [email protected]
│     └─┬ [email protected]
│       └─┬ [email protected]
│         └── [email protected] 
├─┬ [email protected]
│ └─┬ @google-cloud/[email protected]
│   └─┬ @google-cloud/[email protected]
│     └─┬ [email protected]
│       └─┬ [email protected]
│         └─┬ [email protected]
│           └── [email protected] 
└─┬ [email protected]
  └── [email protected] 

I think the problem is the deep dependency via google-cloud. That is a production module, and both bable-cli and instanbul are dev only.

Moreover, when I use yarn why abbrev, it fails to pick up the google-cloud and babel-cli parent dependents:

yarn why v0.18.1
[1/4] 🤔  Why do we have the module "abbrev"...?
[2/4] 🚚  Initialising dependency graph...
[3/4] 🔍  Finding dependency...
[4/4] 🚡  Calculating file sizes...
info Reasons this module exists
   - "istanbul" depends on it
   - "istanbul#nopt" depends on it

@jkrems, @SimenB Shall I raise a new issue for that or not?

istanbul#nopt also looks wrong, in the why output.
I'm onnmy way to work now, will test in a real app then

@SimenB thanks, let me know if you need more information, e.g. my entire package.json modules list.

Edit: actually here it is just in case, as I'm going to sleep now;

"dependencies": {
    "bcrypt": "^1.0.1",
    "bluebird": "^3.4.6",
    "body-parser": "^1.15.2",
    "chalk": "^1.1.3",
    "compression": "^1.6.2",
    "cookie-parser": "^1.4.3",
    "cors": "^2.8.1",
    "express": "^4.14.0",
    "glob": "^7.1.1",
    "google-cloud": "^0.45.1",
    "handlebars": "^4.0.6",
    "html-pdf": "^2.1.0",
    "http-as-promised": "^1.1.0",
    "meanie-express-error-handling": "git+https://github.com/meanie/express-error-handling#2.0.0",
    "meanie-express-github-service": "^2.0.2",
    "meanie-express-jwt-service": "^1.0.2",
    "meanie-express-raven-service": "^1.0.1",
    "meanie-mail-composer": "^1.2.0",
    "meanie-mongoose-only-id": "^1.0.1",
    "meanie-mongoose-set-properties": "^1.0.1",
    "meanie-mongoose-to-json": "^1.1.0",
    "meanie-multer-mime-types-filter": "^1.0.1",
    "meanie-passport-refresh-strategy": "^1.1.2",
    "moment": "^2.17.1",
    "mongoose": "^4.7.3",
    "morgan": "^1.7.0",
    "multer": "^1.1.0",
    "passport": "^0.3.2",
    "passport-http-bearer": "^1.0.1",
    "passport-local": "^1.0.0",
    "phantomjs-prebuilt": "2.1.14",
    "sendgrid": "^4.7.1",
    "sendgrid-mailer": "^1.0.7",
    "socket.io": "^1.7.2",
    "yargs": "^6.5.0"
  },
  "devDependencies": {
    "babel-cli": "^6.16.0",
    "babel-preset-es2015": "^6.18.0",
    "chai": "^3.5.0",
    "chai-as-promised": "^6.0.0",
    "dirty-chai": "^1.2.2",
    "eslint": "^3.12.1",
    "express-simulate-latency": "0.0.2",
    "istanbul": "^1.0.0-alpha.2",
    "mocha": "^3.2.0",
    "mocha-clean": "^1.0.0",
    "nodemon": "^1.11.0",
    "sinon": "^1.17.6",
    "sinon-as-promised": "^4.0.0",
    "sinon-mongoose": "^1.3.0"
  }

This still fails for the app at work. Seems like Yarn's unable to follow deep paths. Here is npm ls entities after yarn --prod

$ npm ls entities
[email protected] /Users/simbekkh/repos/frontpage
└─┬ @finn-no/[email protected]
  └─┬ [email protected]
    └─┬ [email protected]
      └─┬ [email protected]
        └─┬ [email protected]
          └─┬ [email protected]
            └── UNMET DEPENDENCY entities@~1.1.1

npm ERR! missing: entities@~1.1.1, required by [email protected]

Same as @adamreisnz, yarn why doesn't pick up the correct tree.

$ yarn why entities
yarn why v0.18.1
[1/4] 🤔  Why do we have the module "entities"...?
[2/4] 🚚  Initialising dependency graph...
[3/4] 🔍  Finding dependency...
[4/4] 🚡  Calculating file sizes...
info Reasons this module exists
   - "cheerio" depends on it
   - "cheerio#htmlparser2" depends on it
info Disk size without dependencies: "108kB"
info Disk size with unique dependencies: "108kB"
info Disk size with transitive dependencies: "108kB"
info Amount of shared dependencies: 0
✨  Done in 0.40s.

istanbul#nopt also looks wrong, in the why output.

You're right, that seems likely to be the core of this problem. It seems to think that nopt is part of istanbul package, instead of google-cloud and/or babel-cli, and maybe that's why it's not installing it for production environments, because istanbul is not a prod dependency.

Hey everyone, sorry that you are having this problem for quite a while now.
I'll assign this issue to myself and it is now a high priority, I'll try to fix it during the holidays.
Help and PRs with isolated breaking tests or a fix (ideally) are very much welcome.

We have the same issue in prod env with the lib bl that is a dependency of gulp-imagemin's optional dependencies 😕

[~/Workspaces/my-project 12:05:33] NODE_ENV=production yarn
yarn install v0.18.1
info No lockfile found.
[1/4] 🔍  Resolving packages...
warning algoliasearch > [email protected]: Just use Array.isArray directly
warning gulp-file > through2 > xtend > [email protected]:
warning raven > [email protected]: use uuid module instead
warning wiredep > bower-config > [email protected]: graceful-fs v3.0.0 and before will fail on node releases >= v7.0. Please update to graceful-fs@^4.0.0 as soon as possible. Use 'npm ls graceful-fs' to find it in the tree.
warning chromedriver > [email protected]: this package has been reintegrated into npm and is now out of date with respect to npm
warning mversion > [email protected]: Please update to minimatch 3.0.2 or higher to avoid a RegExp DoS issue
warning wiredep > glob > [email protected]: Please update to minimatch 3.0.2 or higher to avoid a RegExp DoS issue
warning webdriverio > request > [email protected]: use uuid module instead
warning gulp > vinyl-fs > glob-watcher > gaze > globule > [email protected]: Please update to minimatch 3.0.2 or higher to avoid a RegExp DoS issue
warning gulp > vinyl-fs > glob-watcher > gaze > globule > glob > [email protected]: graceful-fs v3.0.0 and before will fail on node releases >= v7.0. Please update to graceful-fs@^4.0.0 as soon as possible. Use 'npm ls graceful-fs' to find it in the tree.
warning sprity-lwip > lwip > decree > [email protected]: This package is discontinued. Use lodash@^4.0.0.
[2/4] 🚚  Fetching packages...
warning [email protected]: The engine "ender" appears to be invalid.
[3/4] 🔗  Linking dependencies...
[4/4] 📃  Building fresh packages...
[1/7] ⠂ fsevents: GET https://fsevents-binaries.s3-us-west-2.amazonaws.com/v1.0.15/fse-v1.0.15-node-v51-darwi
[2/7] ⠂ gifsicle
[3/7] ⠂ jpegtran-bin
[4/7] ⠂ optipng-bin
error /Users/fdubost/Workspaces/my-project/node_modules/gifsicle: Command failed.
Exit code: 1
Command: sh
Arguments: -c node lib/install.js
Directory: /Users/fdubost/Workspaces/my-project/node_modules/gifsicle
Output:
module.js:474
    throw err;
    ^

Error: Cannot find module 'bl'
    at Function.Module._resolveFilename (module.js:472:15)
    at Function.Module._load (module.js:420:25)
    at Module.require (module.js:500:17)
    at require (internal/module.js:20:19)
    at Object.<anonymous> (/Users/fdubost/Workspaces/my-project/node_modules/tar-stream/extract.js:2:10)
    at Module._compile (module.js:573:32)
    at Object.Module._extensions..js (module.js:582:10)
    at Module.load (module.js:490:32)
    at tryModuleLoad (module.js:449:12)
    at Function.Module._load (module.js:441:3)

Thanks for your help 😊

It works if I manually add bl to our package.json...

Any news on this?

Not yet, I am building an ad-hoc commonJS checker that verifies node_modules structure independent of Yarn's hoisting and resolution algorithms https://github.com/yarnpkg/yarn/pull/2419.
It would be able to catch all the cases described in this bug and guard us from future regressions.

@kittens is having a look at what is happening here.
The bug is not trivial, so any additional insight is welcome

All right, collecting all the repros now with the latest trunk.
The example in the first comment does not reproduce anymore and yarn check --verify-tree passes

This one does not reproduce too https://github.com/yarnpkg/yarn/issues/761#issuecomment-260975012 and https://github.com/yarnpkg/yarn/issues/761#issuecomment-265823529

Yeah, that repro was fixed with 0.18.1.

2 ideas:

Is there some log with the resolutions happening that I can share with you?

Additionally, I can give you the lockfile that reproduces it at work, but you won't be able to install it because of private deps. You might be able to mess around and skip the fetching of packages, and just mess about with the tree?

@SimenB, so you you have another example where the resolution is broken for --production install?

For that case you can try:

yarn install --production --verbose
yarn check --production --verify-tree

With latest master branch.
If you don't want to send logs publicly, message me at [email protected]

Yeah, 0.18.1 was still broken, haven't tested 0.19 (or master). If it still reproduces (hope not!), I'll send you the logs privately

Let's close this task because the title issue got solved.
There are 2 open ones that I haven't rechecked yet: #2263 and #2141 feel free to comment there or create a new one for your case and cc me.

2all: when you comment that installation is incorrect please include a package.json so that others could reproduce that.
Kudos to @jkrems for making a step further and submitting an awesome repro script: https://github.com/yarnpkg/yarn/issues/761#issuecomment-265823529

@bestander have you also re-checked https://github.com/yarnpkg/yarn/issues/761#issuecomment-268130124?

@adamreisnz, can you share the package.json again please?

https://github.com/yarnpkg/yarn/issues/761#issuecomment-268130124 is a different issue.

That one fails for yarn install --production.

This issue is about yarn install --production finishing successfully but is not doing the right thing.

@bestander I've shared it in the comment below that, https://github.com/yarnpkg/yarn/issues/761#issuecomment-268201708, cheers

Still fails with master (c98df16b) for me...

yarn check --verify-tree throws though, that's promising. Lots of those are dev deps though.

yarn check v0.20.0-0
error "babel-preset-es2015" not installed
error "browserify-middleware" not installed
error "cheerio" not installed
error "codeceptjs" not installed
error "del-cli" not installed
error "eslint" not installed
error "eslint-config-finn" not installed
error "espower-loader" not installed
error "hashmark" not installed
error "interfake" not installed
error "nightmare" not installed
error "nightmare-upload" not installed
error "nock" not installed
error "nodemon" not installed
error "nyc" not installed
error "power-assert" not installed
error "sinon" not installed
error "supertest" not installed
error "uglifyify" not installed
error "@finn-no/express-base#nunjucks#chokidar#anymatch" not installed
error "@finn-no/express-base#unleash-client#request#json-stringify-safe" not installed
error "@finn-no/express-base#pretty-error#renderkid#css-select#domutils#dom-serializer#entities" not installed
error Found 22 errors.

Running the app, it fails on missing anymatch.

npm ls shows other missing deps, which makes more sense

npm ERR! extraneous: [email protected] /Users/simbekkh/repos/frontpage/node_modules/node-pre-gyp
npm ERR! missing: anymatch@^1.3.0, required by [email protected]
npm ERR! missing: entities@~1.1.1, required by [email protected]
npm ERR! missing: json-stringify-safe@~5.0.1, required by [email protected]

And this was the observation/cause of the issue: https://github.com/yarnpkg/yarn/issues/761#issuecomment-268331340

You're right, that seems likely to be the core of this problem. It seems to think that nopt is part of istanbul package, instead of google-cloud and/or babel-cli, and maybe that's why it's not installing it for production environments, because istanbul is not a prod dependency.

Oh, sorry, @SimenB

yarn check --prodution --verify-tree

I'll edit my comment

Doing yarn check --verify-tree --production gives a good output (it agrees with npm ls):

yarn check v0.20.0-0
error "@finn-no/express-base#nunjucks#chokidar#anymatch" not installed
error "@finn-no/express-base#unleash-client#request#json-stringify-safe" not installed
error "@finn-no/express-base#pretty-error#renderkid#css-select#domutils#dom-serializer#entities" not installed
error Found 3 errors.

@bestander I'll email you package.json, yarn.lock and the verbose install log 😄

@dashmug do you want me to create a new ticket for that issue? It's still a problem of incorrect dependencies being installed (albeit just for production), so I think it's related to this ticket.

@bestander Email sent.
While @finn-no/express-base is not publicly available, the other 3 packages in the output are, so hopefully you can reproduce with solely public packages.

Should I open a new issue?

@adamreisnz, could you please create a new issue then?
I can reproduce it but it is a different issue from the title one

It is related. Probably the same cause. Just a different symptom. I would put that as a different issue since it's not exactly the same as what the title says.

@SimenB, thanks, I'll have a look

Ok will do guys.

@bestander Making a package wth just those 3 in the output that fails makes it reproducible on master for me with just public deps. It's not a minimal reproduce, but still

{
  "name": "app",
  "version": "1.0.0",
  "dependencies": {
    "brakes": "^2.5.1",
    "compression": "^1.6.2",
    "envalid": "^2.4.0",
    "express": "^4.14.0",
    "object.entries": "^1.0.4",
    "prom-client": "^7.0.0",
    "response-time": "^2.3.2",
    "spaden": "^7.13.1",
    "yarn-issue-repro-package": "^1.0.0"
  },
  "devDependencies": {
    "babel-preset-es2015": "^6.18.0",
    "browserify": "^13.1.1",
    "browserify-middleware": "^7.1.0",
    "cheerio": "^0.22.0",
    "codeceptjs": "^0.4.13",
    "del-cli": "^0.2.1",
    "eslint": "^3.12.2",
    "eslint-config-finn": "^1.0.1",
    "espower-loader": "^1.0.1",
    "hashmark": "^4.1.0",
    "interfake": "^1.19.0",
    "mocha": "^3.2.0",
    "nightmare": "^2.9.0",
    "nightmare-upload": "^0.1.1",
    "nock": "^9.0.2",
    "nodemon": "^1.11.0",
    "nyc": "^10.0.0",
    "power-assert": "^1.4.1",
    "sinon": "^1.17.6",
    "supertest": "^2.0.1",
    "uglify-js": "^2.7.5",
    "uglifyify": "^3.0.4"
  }
}

package.json of yarn-issue-repro-package

{
  "name": "yarn-issue-repro-package",
  "version": "1.0.0",
  "main": "index.js",
  "license": "MIT",
  "dependencies": {
    "nunjucks": "^2.5.2",
    "pretty-error": "^2.0.2",
    "unleash-client": "^1.0.0-beta.7"
  }
}

Interestingly, it produces 4 errors instead of 3...

$ yarn check v0.20.0-0                                                                                                   │├─ [email protected]
error "yarn-issue-repro-package#nunjucks#chokidar#anymatch" not installed                                              │└─ [email protected]
error "yarn-issue-repro-package#unleash-client#request#json-stringify-safe" not installed                              │✨  Done in 2.65s.
error "yarn-issue-repro-package#nunjucks#yargs#string-width#code-point-at" not installed                               │ ~/repos/yarn-issue-repro-package  vim package.json
error "yarn-issue-repro-package#pretty-error#renderkid#css-select#domutils#dom-serializer#entities" not installed      │ ~/repos/yarn-issue-repro-package  npm publish
error Found 4 errors.

I'm not sure is this the same issue or not, but here is the output (tested with 0.19.1)

I think I have found the root cause for the installation issue.
The failure to install a package can be reproduced by the following package.json:

{ "dependencies": {
    "bcrypt": "^1.0.1",
    "gamepad": "1.4.2"
  },
  "devDependencies": {
     "istanbul": "^1.0.0-alpha.2"
  }
}

And then the commands

rm -rf node_modules yarn.lock
yarn install
rm -rf node_modules
yarn install --production
npm ls abbrev

In this configuration, abbrev is not installed.

abbrev is used by istanbul and by nopt (as can be seen from yarn why abbrev). nopt is used by istanbul and node-pre-gyp (which is used by bcrypt and gamepad).

When deduping abbrev in the package hoister, the following code is used to determine the new isIgnored function of the hoisting record:

          // switch to non ignored if earlier deduped version was ignored
          if (existing.isIgnored() && !info.isIgnored()) {
            existing.isIgnored = info.isIgnored;
          }

abbrev is one of the first hoisting records to be processed. At that moment, the existing record is istanbul#abbrev (ignored because istanbul is ignored), and the duplicate record is istanbul#nopt#abbrev, which is also ignored at the time for the same reason.

Because both records are ignored at the time, the ignore function isn't adjusted as it was supposed to - because nopt will become non-ignored in a later dedup because of the dependency by node-pre-gyp. The ignore status of both records can change at any time, so the new ignore function should mix them.

And indeed, the installation problem goes away when we replace those lines with

          // switch to non ignored if earlier deduped version was ignored
          if (existing.isIgnored()) {
            if (info.isIgnored()) {
              // both are ignored now, but any one could become non ignored later on.
              let oldIsIgnored = existing.isIgnored;
              existing.isIgnored = () => oldIsIgnored() && info.isIgnored();
            } else {
              existing.isIgnored = info.isIgnored;
            }
          }

@blexrob, great find!
Would you send a PR?
There is a disabled test for "should not lose dependencies when installing with --production" in integration.js that should be fixed now

@bestander, just tested it, and this fix causes a stack overflow in the test you mentioned, so it can't be applied. The following cycle pops up:

d#es5-ext -> es6-symbol#es5-ext -> es6-set#es5-ext -> es6-iterator#es5-ext -> es6-map#es5-ext -> es5-ext#es6-iterator -> es6-set#es6-iterator -> es6-weak-map#es6-iterator -> event-emitter#es5-ext -> d#es5-ext

So, the naive recursive call approach is out...

Yeah, I believe it needs a bit tweaking but the idea seems correct

I have such a problem with phantomjs-prebuilt module (as dependence of dependence) with yarn 0.27.5-1.
So now I do dummy yarn add phantomjs-prebuilt, before yarn install --production.

I regret to say that this still seems to be a problem in Yarn 1.3.2.
My builds on Netlify are failing when I use Yarn 1.3.2 but succeeding with Yarn 0.18.2.
The build errors with a cannot find module 'are-we-there-yet' and only with production flag.

@adamreisnz, this thread is too big to track all issues.
Could you raise a new one with repro script?

@bestander done, thanks.

For someone who still can't get it work and doesn't want to install jq can use

$ python -c "import json; p = json.loads(open('package.json').read()); del p['devDependencies']; open('package.json', 'w').write(json.dumps(p, indent=2));"

i am on yarn 1.17.3 and node v10.16.2 in lerna monorepo. still facing same issue.

I can confirm it too.
I have a lot of dependencies, but when I use yarn install --production, only two modules are installed.
Notable though, I am on a Lerna monorepo similar to @hannadrehman with Yarn workspaces, which may explain the extreme behavior.

Yarn version: 1.22.0
Node: v12.16.1

npm install --production works perfectly though.

@hannadrehman is the project in question a package of your monorepo?

Same problem as @TAnas0