Yarn: Native packages rebuilt every time

Can't reproduce this in Mac OSX (10.11.6) so might be a Ubuntu specific issue?

I could repro on Windows 10, but only once. The second time I ran "yarn", it didn't rebuild the native libraries. Strange.

I was playing with it some more and came up with a few more details:

1. If I run yarn add leveldown bcrypt, leveldown will be compiled as the first item in the list and the hash in node_modules/.yarn-integrity will equal to 595306... when it's finished (cut for brevity; I assume that's a checksum which determines whether something needs to be done). Now if I run just yarn again, both of the packages will be rebuilt with bcrypt being compiled as the first in the list (the order is different). The resulting checksum will equal to cbe480.... Subsequent invocation of yarn will not rebuilt the packages and the checksum will stay the same. This contradicts my original report (I probably made a mistake somewhere) but aligns with what @Daniel15 was seeing.
2. When I reverse the order of the packages right from the start (yarn add bcrypt leveldown), bcrypt will be first in the list and the resulting checksum will equal to cbe480... immediately. Subsequent invocation of yarn will not rebuilt the packages.
3. The bcrypt package will always finish first (as expected since there's simply not that much to compile) regardless of the position in the list.

I'm not at all familiar with the internals but it seems to me that the order in which the packages are compiled matter and they are simply not being sorted when first installed (and they _are_ sorted when later invoking yarn) which affects the checksum in some way.

Thank you for investigating! That sounds like a good lead. The hash is written here: https://github.com/yarnpkg/yarn/blob/f04b157a0162114de7252b682ecf4b66895d7e87/src/cli/commands/install.js#L581-L583. It might be worth debugging that code and seeing what's different in the lockfile, as the hash in .yarn-integrity is based off the lockfile.

It might be worth debugging that code and seeing what's different in the lockfile, as the hash in .yarn-integrity is based off the lockfile.

I suspected that but what threw me off was the fact that the lock file doesn't change, it's always the same. It's just the hash in .yarn-integrity that changes.

$ yarn add leveldown bcrypt
$ cat node_modules/.yarn-integrity
59530680cc0a4ee12feb373980b5abf2edf2fe2aefa16d556bfb531af8299e71
$ cp yarn.lock leveldown_bcrypt_initial.lock
$ yarn
$ cat node_modules/.yarn-integrity
cbe48058288527f95dfe5643976b0735ee784860663e93ed782b98477c824ec1
$ cp yarn.lock leveldown_bcrypt_afterwards.lock
$ diff -s leveldown_bcrypt_initial.lock leveldown_bcrypt_afterwards.lock
Files leveldown_bcrypt_initial.lock and leveldown_bcrypt_afterwards.lock are identical

$ yarn add bcrypt leveldown
$ cat node_modules/.yarn-integrity
cbe48058288527f95dfe5643976b0735ee784860663e93ed782b98477c824ec1
$ cp yarn.lock bcrypt_leveldown_initial.lock
$ yarn
$ cat node_modules/.yarn-integrity
cbe48058288527f95dfe5643976b0735ee784860663e93ed782b98477c824ec1
$ cp yarn.lock bcrypt_leveldown_afterwards.lock
$ diff -s bcrypt_leveldown_initial.lock bcrypt_leveldown_afterwards.lock
Files bcrypt_leveldown_initial.lock and bcrypt_leveldown_afterwards.lock are identical

$ diff -s leveldown_bcrypt_initial.lock bcrypt_leveldown_initial.lock
Files leveldown_bcrypt_initial.lock and bcrypt_leveldown_initial.lock are identical

I have the same issue: I use bcrypt too. Every time when I install some new modules or upgrade exist ones, I have to run npm rebuild to make my app runnable.

macOS 10.12 && node v7.0.0 && yarn v0.16.1

I can no longer reproduce the original issue. It seems to have been fixed :+1:. @Daniel15 Can you confirm?

@hustcer I don't think that's the same issue. You might want to test with the latest version and file a new bug if it's still not working for you.

@jiripospisil Thanks, It's okay now after upgrade to yarn v0.17.4.

I'm still seeing this, or something very similar, on 0.18.1. Incidentally, it's also leveldown that keeps getting rebuilt repeatedly. Using leftpad as a package with no dependencies (and that is not depended on by leveldown) for demonstrative purposes, repro steps are as follows:

mkdir test && cd test
echo '{ "name": "test", "version": "1.0.0" }' > package.json
yarn add leveldown 
yarn add leftpad
yarn remove leftpad

My output when I run this follows. Note that removing leftpad takes almost 40 seconds, the majority of which is rebuilding leveldown. This happens consistently, with and without leveldown or leftpad in the Yarn cache, though only during remove and never add.

$ mkdir test && cd test
$ echo '{ "name": "test", "version": "1.0.0" }' > package.json
$ yarn add leveldown 
yarn add v0.18.1
info No lockfile found.
[1/4] 🔍  Resolving packages...
[2/4] 🚚  Fetching packages...
[3/4] 🔗  Linking dependencies...
[4/4] 📃  Building fresh packages...
success Saved lockfile.
success Saved 145 new dependencies.
<... package list omitted for brevity ...>
✨  Done in 49.93s.
$ yarn add leftpad
yarn add v0.18.1
[1/4] 🔍  Resolving packages...
[2/4] 🚚  Fetching packages...
[3/4] 🔗  Linking dependencies...
[4/4] 📃  Building fresh packages...
success Saved lockfile.
success Saved 1 new dependency.
└─ [email protected]
✨  Done in 2.18s.
$ yarn remove leftpad
yarn remove v0.18.1
[1/2] Removing module leftpad...
[2/2] Regenerating lockfile and installing missing dependencies...
success Uninstalled packages.
✨  Done in 38.76s.
$ yarn why leftpad
<... just to confirm that leftpad and leveldown are entirely unrelated ...>
yarn why v0.18.1
[1/4] 🤔  Why do we have the module "leftpad"...?
[2/4] 🚚  Initialising dependency graph...
warning [email protected]: No license field
[3/4] 🔍  Finding dependency...
error We couldn't find a match!
✨  Done in 0.30s.

Versions:

Node v7.3.0
Yarn 0.18.1
OS X El Capitan (10.11.6)

Please reopen as this still happens.
Just did yarn add redux and it rebuilt bcrypt, node-sass and several others.

Node v6.9.4
Yarn v0.20.3
Windows 10

@seansfkelley I followed your steps with the latest version and it worked. Could you try again?

/tmp/tp-20170222100611/test ∴ echo '{ "name": "test", "version": "1.0.0" }' > package.json
/tmp/tp-20170222100611/test ∴ yarn add leveldown
yarn add v0.20.3
info No lockfile found.
warning [email protected]: No license field
[1/4] 🔍  Resolving packages...
[2/4] 🚚  Fetching packages...
[3/4] 🔗  Linking dependencies...
[4/4] 📃  Building fresh packages...
success Saved lockfile.
success Saved 54 new dependencies.
<cut>
warning [email protected]: No license field
✨  Done in 1.64s.
/tmp/tp-20170222100611/test ∴ yarn add leftpad
yarn add v0.20.3
warning [email protected]: No license field
[1/4] 🔍  Resolving packages...
[2/4] 🚚  Fetching packages...
[3/4] 🔗  Linking dependencies...
[4/4] 📃  Building fresh packages...
success Saved lockfile.
success Saved 1 new dependency.
└─ [email protected]
warning [email protected]: No license field
✨  Done in 0.69s.
/tmp/tp-20170222100611/test ∴ yarn remove leftpad
yarn remove v0.20.3
[1/2] Removing module leftpad...
[2/2] Regenerating lockfile and installing missing dependencies...
warning [email protected]: No license field
success Uninstalled packages.
✨  Done in 0.66s.
/tmp/tp-20170222100611/test ∴ yarn why leftpad
yarn why v0.20.3
[1/4] 🤔  Why do we have the module "leftpad"...?
[2/4] 🚚  Initialising dependency graph...
warning [email protected]: No license field
[3/4] 🔍  Finding dependency...
error We couldn't find a match!
✨  Done in 0.20s.

@Nexxado Could you please add a few reproduction steps? I tried a few combinations but it worked.

Node 7.3.0
Yarn 0.20.3
MacOS 10.12.3

@jiripospisil I have no reproduction steps to provide, simply installing an additional package causes yarn to link and rebuild everything.

Here's the output of adding one package (lock file already exists):

Linking dependencies:

Rebuilding:

@jiripospisil I am also still seeing this, but during my repro I got tripped up because it looks like leveldown (or a dependency thereof) may have started shipping an OS X-compatible binary, so the install times dropped suspiciously from 50s to 3s. If you're on OS X and you specifically yarn add [email protected] instead of just yarn add leveldown, you should see the same behavior as before.

I have an indirect dependency on ttf2woff2, which also rebuilds every time.

However, it only rebuilds every time when there is a change to yarn.lock. That is, yarn with a new yarn.lock, yarn upgrade, yarn upgrade-interactive. In the yarn upgrade-interactive case, if both devdeps and regular deps were updated, ttf2woff2 gets rebuilt twice(!).

I'm also seeing this issue, though I could not reproduce it with the steps above. However I can reproduce it with these steps:

yarn install pouchdb-node
````

which builds leveldown. Then if I add another package:

yarn add co
```

then again it builds leveldown.

It doesn't seem to matter what package I add, it always rebuilds leveldown.

I'm using Yarn v0.21.3, Windows 10, and Node v7.7.1

I'm seeing this as well. I'm using BuckleScript (bs-platform)....

I'm encountering this issue, too, with sharp. Every time I run yarn add or yarn remove, sharp would get rebuilt, even with non-native packages.

Tested in yarn v0.21.3, Node 7.0.0, under Windows 10 and Ubuntu Linux 16.04.

Here is package.json dependencies, if it helps:

{
  "devDependencies": {
    "auto-reload-brunch": "^2.7.1",
    "babel-brunch": "^6.1.1",
    "babel-preset-env": "^1.2.1",
    "brunch": "^2.10.8",
    "chai": "^3.5.0",
    "clean-css-brunch": "^2.10.0",
    "css-brunch": "^2.10.0",
    "express-mysql-session": "^1.2.0",
    "javascript-brunch": "^2.10.0",
    "jquery": "^3.1.1",
    "less-brunch": "^2.10.0",
    "mocha": "^3.2.0",
    "nodemon": "^1.11.0",
    "npm-run-all": "^4.0.2",
    "postcss-brunch": "^2.0.5",
    "postcss-cssnext": "^2.9.0",
    "postcss-font-magician": "^1.6.1",
    "uglify-js-brunch": "^2.10.0"
  },
  "dependencies": {
    "body-parser": "^1.17.1",
    "connect-redis": "^3.2.0",
    "cookie-parser": "^1.4.3",
    "debug": "^2.6.1",
    "express": "^4.15.2",
    "express-session": "^1.15.1",
    "jstransformer-marked": "^1.0.2",
    "md5": "^2.2.1",
    "morgan": "^1.8.1",
    "multer": "^1.3.0",
    "node-mysql": "^0.4.2",
    "passport": "^0.3.2",
    "passport-local": "^1.0.0",
    "pug": "^2.0.0-beta11",
    "serve-favicon": "^2.4.1",
    "sharp": "^0.17.2"
  }
}

also seeing this with bs-platform

Also experiencing this with ttf2woff2 every call to yarn add rebuilds ttf2woff2 even though it hasn't been published in over a year.

I have this with couchbase as well

edit: seems to be fixed in 0.23.2

It still happens to me on 0.23.2 (argon2 and node-sass get rebuilt every time I add/remove some unrelated package like moment which has no dependencies)

OS: Windows 10
Node: 7.9.0
Yarn: 0.23.2

To add some more color, my perception of this happening on yarn add <some-package> was much greater than reality as many cases for me were actually triggered by combining with yarn remove <unrelated-package> immediately before due to the force: true on this line.

Certainly convenient to reuse the install logic in remove to generate the lockfile, but it'd be nice if it didn't come with all the baggage of a forced install :)

For me this started happening again when I upgraded to 0.23.x. I reverted to 0.21.3 and it no longer builds each time. Also this led to this issue where my IP was blocked by unicode.org after upgrading a few packages in a row https://github.com/dodo/node-unicodetable/issues/16

While 0.21.3 does not rebuild all the packages on add a new package, it rebuilds all the packages on remove. And it seems yarn doesn't regard it a failure if rebuilding fails.

For me, downgrading to 0.21.3 did not help. bs-platform gets rebuilt on every add/remove.

Seeing this on macOS with Yarn 0.23.4. Rebuilds sqlite3 every time I run yarn add.

$ yarn add gulp-rename gulp-notify gulp-sass -D                                                                                         1 ↵
yarn add v0.23.4
[1/4] 🔍  Resolving packages...
[2/4] 🚚  Fetching packages...
warning [email protected]: The platform "darwin" is incompatible with this module.
info "[email protected]" is an optional dependency and failed compatibility check. Excluding it from installation.
warning [email protected]: The platform "darwin" is incompatible with this module.
info "[email protected]" is an optional dependency and failed compatibility check. Excluding it from installation.
[3/4] 🔗  Linking dependencies...
[4/4] 📃  Building fresh packages...
success Saved lockfile.
success Saved 42 new dependencies.
├─ [email protected]
├─ [email protected]
├─ [email protected]
├─ [email protected]
├─ [email protected]
├─ [email protected]
├─ [email protected]
├─ [email protected]
├─ [email protected]
├─ [email protected]
├─ [email protected]
├─ [email protected]
├─ [email protected]
├─ [email protected]
├─ [email protected]
├─ [email protected]
├─ [email protected]
├─ [email protected]
├─ [email protected]
├─ [email protected]
├─ [email protected]
├─ [email protected]
├─ [email protected]
├─ [email protected]
├─ [email protected]
├─ [email protected]
├─ [email protected]
├─ [email protected]
├─ [email protected]
├─ [email protected]
├─ [email protected]
├─ [email protected]
├─ [email protected]
├─ [email protected]
├─ [email protected]
├─ [email protected]
├─ [email protected]
├─ [email protected]
├─ [email protected]
├─ [email protected]
├─ [email protected]
└─ [email protected]
$ install-app-deps
Rebuilding native production dependencies for darwin:x64
Rebuilding native dependency sqlite3
✨  Done in 56.61s.

Seeing this issue with Ubuntu 16.04LTS, with latest yarn v0.24.6, with node 8.1.2

I am seeing gdal, node-sass rebuilding every time I add a module, this is causing yarn add to take way longer to run than necessary.

I'm seeing this too, it's super annoying on a Raspberry Pi Zero W where building packages (like bleno) can take several minutes.

Still seeing this with Yarn v0.27.5 and uws. Every time something in my packages changes, uws is rebuilt.

The only interesting thing I can see about uws is that it does not have a dependencies field in package.json.

This is becoming quite a frustration for me over the past few days. I had windows-build-tools installed globally at one stage (only really needs to build itself once to set up the Windows dev environment for native packages) which kept re-building itself every time I added a package. As you can imagine that took quite a while and I eventually realised I didn't need it installed anymore and removed it.

Now it seems that node-sass keeps wanting to rebuild on another projected when adding packages.

This behaviour happens on both yarn add and yarn remove for me. Surely no rebuild is necessary for these steps as the native packages are only built once as per the Node version?

Edit: Using Node v8.2.1 and Yarn v0.27.5 on Windows 10.

I can't count the times that uws was rebuilt for me :) Note that uws
has zero dependencies and is even missing the field in package.json

On Mon, Jul 31, 2017, 10:50 PM Paul Myburgh notifications@github.com
wrote:

This is becoming quite a frustration for me over the past few days. I had
windows-build-tools installed globally at one stage (only really needs to
build itself once to set up the Windows dev environment for native
packages) which kept re-building itself every time I added a package. As
you can imagine that took quite a while and I eventually realised I didn't
need it installed anymore and removed it.

Now it seems that node-sass keeps wanting to rebuild on another projected
when adding packages.

This behaviour happens on both yarn add and yarn remove for me. Surely no
rebuild is necessary for these steps as the native packages are only built
once as per the Node version?

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/yarnpkg/yarn/issues/932#issuecomment-319192291, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AADWlgSNhi-V9-yGWsWHBDFYdyJW8Arjks5sTj4UgaJpZM4KVN87
.

I'm on 0.27.5 and keep seeing this behavior with bs-platform.

Quite a frustration this, ditto here with bs-platform.

Nice sense of entitlement there… How about a PR Tim?

On Tue, Aug 22, 2017, 10:44 AM Tim Shnaider notifications@github.com
wrote:

FFS can this be fixed please.
At least provide a command line option or env setting to disable it.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/yarnpkg/yarn/issues/932#issuecomment-323960245, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AADWltQedG4owTlcC8koC4RL-vTiCE0Hks5sapTbgaJpZM4KVN87
.

It doesn't seem to have been mentioned so far, but I can also reproduce this bug with yarn global list.

Steps to reproduce:

1. Use a fresh global env: :warning: run this only if you know what you're doing

   rm -rf ~/.config/yarn/
   

2. Add a problematic package (_i.e._ zeppelin-solidity):

   → yarn global add node-gyp zeppelin-solidity
   yarn global v0.27.5
   [1/4] Resolving packages...
   warning zeppelin-solidity > truffle-hdwallet-provider > web3-provider-engine > ethereumjs-block > merkle-patricia-tree > level-ws > xtend > [email protected]:
   [2/4] Fetching packages...
   [3/4] Linking dependencies...
   [4/4] Building fresh packages...
   success Installed "[email protected]" with binaries:
         - node-gyp
   warning "[email protected]" has no binaries
   Done in 20.53s.
   

3. Run yarn global list and see some native packages being recompiled

4. Repeat as much as you want, yarn global list will always recompile those native packages
5. 😭

I hope this helps.

ℹ️ MacOS 10.12.6 with Yarn 0.27.5 installed via Homebrew.

The entitlement is unnecessary, but I can understand the frustration. This is wasting a lot of people a lot of time (long reinstalls add up over time and break flow). Sure, you can say "make a pull request" - and that's fair. But it would be a learning process on top of changing potentially just a few lines of code... What we're hoping for is that people who know the ins and outs of this project might prioritise this issue on the next release as it really does seem rather major (and potentially an easy fix? Prevent reinstalling binaries if the Node version hasn't changed maybe).

It's been almost a year now since it was first reported.

I hope I don't sound entitled in saying that... I would like to add that I'm very thankful for this project and the usefulness it already brings to me. This issue has been one of very few pain points I've had with it.

EDIT: Just did a yarn remove for a random package and it tried and (this time) failed to rebuild my binaries. The side-effect being that my binaries are completely missing now and it seems the only way to fix it is with an npm rebuild. So not only does it seem this issue causes unnecessary rebuilds - but if it fails that process, it seems you have to resort back to npm to fix it up again.

I am seeing the same behavior as @zhangjyr and @lostpebble. Running yarn add works fine but yarn remove causes native packages to be rebuilt.

Mac Sierra 10.12.6
Yarn 0.27.5

I'll try to see what I can do about this since this also affects me. That said Yarn is not that hard to contribute to so if someone wants to send a PR, please give it a shot and we'll try to support you as much as we can.

I won't be able to work on this for a few weeks.

I am seeing this, too.
Working on site build on gatsby. ANY yarn operation causes rebuild.
Try creating site with gatsby and yarning with it. Hope this helps

Seeing this in a project using gatsby:
yarn 1.0.2
node 7.10.1
ubuntu 16.04

This issue is becoming quite serious now. Not only are my binaries always re-built. But it is often happening that my binaries are completely removed after a yarn add or yarn install (after changing version numbers of a package in package.json to update / rollback a module). And no matter how much I run yarn install or yarn install --check-files afterwards, those binaries are just lost and gone and never come back. The only way to retrieve them is with an npm rebuild.

It's starting to feel like yarn doesn't have knowledge about the state of native packages at all. Whether they are already installed / built, or if they are not installed / built properly.

Has there perhaps been any progress made on this?

I think the recent artifacts field addition to yarn integrity file by @arcanis would help fix this issue. No progress yet but open to PRs :)

I have a similar problem with bs-platform when installing packages in my reasonml project

Same... literally every yarn add will rebuild the gyp project.

Happens here as well.
(yarn 1.2.1)

I see this with node-sass.

Happens to mee to with Cypress for instance.

node -v
v8.8.1
yarn -v
1.2.1

✋ Instead of writing "Me too" with no additional info, use the +1 feature in Github on the top post. Every time you write a "me too" comment 35 subscribers are needlessly notified.

+1

Please just click Subscribe if you want to receive updates. I hope everyone wants to be notified when it's resolved but not when somebody new faced this issue too.

@BYK can you please lock thread, and make it available only for owners.

I'm currently investigating this issue and trying to boil it down to a minimum-reproducible test case. A "great" package for demonstrating this issue is htmlstrip-native - it takes a solid few minutes to compile so you can't miss it.

I'd love to have someone else try this package.json in an empty folder:

{
  "name": "yarn-test",
  "version": "1.0.0",
  "dependencies": {
    "htmlstrip-native": "^0.1.3",
    "left-pad": "1.1.3"
  }
}

Try running these commands:

yarn
yarn upgrade [email protected]

On my machine, with latest yarn 1.2.1, the upgrade command results in a rebuild of htmlstrip-native which takes forever. Yarn shouldn't be rebuilding this since upgrading left-pad has no effect on htmlstrip-native or its dependencies.

Now try npm:

rm -rf node_modules
npm install
npm install [email protected]

On my machine, the second command (correctly) does not result in a fresh rebuild of htmlstrip-native.

EDIT: From re-reading all the posts above it sounds like this case will be of no surprise to anyone - most people, myself included, are finding that simply asking yarn to do _anything_ results in a rebuild. I'm guessing this isn't a bigger issue in the community because most people don't use native packages that take a long time to build - so they either don't have a rebuild step or it finishes so quickly that who cares.

OK, so after much debugging, it appears that this behavior is intentional. Tracing through the code, it looks like, for instance, yarn upgrade [email protected] results in the following steps:

1) upgrade module is called, which runs a new Add() operation for left-pad.
2) Add.init() calls up to its superclass, Install.init()
3) Install.init() queues a rebuildingPackages step
4) In PackageInstallScripts.init() it simply collects _all_ packages and adds them to the workQueue to be rebuilt.
5) PackageInstallScripts then discovers that there's an install command for htmlstrip-native and just runs it - this is the super-slow native rebuild that we're all seeing.

From everything I've seen so far, there seems to be no logic that is intending to filter out packages that don't "need" to be rebuilt. It's simply rebuilding everything, as stated in the console output.

I'd love someone from the Yarn team to chime in here - if this behavior is truly intentional, then I'd recommend closing this issue!

For my situation personally, I'm going to just swap out my htmlstrip-native dependency since there is no reason it should be taking minutes to build (it's like, a couple tiny .c files). The rest of my native deps build quickly, so it's not a big deal if it happens all the time.

It sounds like an unintentional side effect of the design, but maybe someone on @yarnpkg/core can comment about it. I don't think it's intended to rebuild packages that don't need to be rebuilt.

This is not intentional, it is probably just easier to implement that way.
There is a comment above from BYK that says this issue is looking for a PR:
https://github.com/yarnpkg/yarn/issues/932#issuecomment-332498506

Certainly native-heavy packages are not common enough to surface as highest priority for Yarn however Yarn has a capability to not rebuild packages on every install.
This seems like a bug that should be straight forward to fix, send a PR.
There may be caveats because Yarn can't know for sure if a built package could have been corrupted, that is why it is eagerly rerunning install scripts now.

There is a Yarn fork that is used by the team behind Reason https://github.com/esy-ocaml/esy-install that works around a lot of native compilation issues because Reason/Ocaml dependencies require heavy compilation.
As that approach matures I hope it will be possible to merge the changes upstream.

So fundamentally, native packages are rebuilt because either:

1) The force=true flag has been set, or
2) The package has been marked fresh=true.

It seems like with some commands (like upgrade and remove), the force flag is set to true "just in case". This flag promises to "rebuild all packages regardless of if they've changed" so it doesn't make sense to add in some change-detection code because that would break this promise.

So to fix this, it seems like we will need to challenge the assumptions of the various places in code that set force=true.

The first one I tracked down happens when running yarn upgrade whatever. It was introduced in this commit as part of #2780 by @juanca. The commit notes say:

"Ensure force flag is enabled when using `yarn upgrade` Otherwise exotic packages are not updated."

Maybe @juanca or someone more familiar with yarn's history could chime in with the problem this was intended to solve?

@nfarina thanks, that brings clarity. For upgrades, maybe we really do want to force-build native packages (if they're being upgraded). But for installs (yarn add), I'd say we legitimately need to challenge, as you said, the assumption that makes already-installed native packages rebuild when something else is installed.

I suppose force flag is set so that Yarn resolver would not skip the existing dependency because the old version is in yarn.lock.

I think it was a quick solution to make upgrade work.
A proper way would be to pass another flag that affects only resolution of specified packages and won't be as nuclear as force.
Send a PR :)

Maybe @juanca or someone more familiar with yarn's history could chime in with the problem this was intended to solve?

Correct, I was working with just the Add API -- it does (did?) nothing when adding an existing package.

I also recommend using a better technique in controlling the procedural flow.

I'm using yarn on my raspberry pi zero in a project that has a dependency to node-opencv. Every time I add / remove / update a unrelated package, I have to wait 35 Minutes for the rebuild.

@Torsten85, I use it on pi as well, yeah unnecessary rebuilds is something we need to tackle.
Can you provide a repro script for your rebuilds?

same here, on lubuntu 17.10, when I use yarn, developing on with https://github.com/mui-org/material-ui

every single yarn add/remove (-D) <pkg> rebuild everything, like a fresh install, takes more than 1 minute

Is this how npm handles it as well? If not would it be a possible workaround to switch temporarily?

is happening also over here

• node 9.2.0
• yarn 1.4.0

each time I add some dependency, modules like bcrypt or couchbase are rebuilded every time

Hey everyone, I am working on a small hack https://github.com/yarnpkg/yarn/pull/5314.
The idea is to cache a built package into offline mirror to avoid running install scripts all the time.

So if you use offline-mirror and run yarn add node-sass:

1. yarn would build and install node-sass as usual,
2. after the install scripts are run it will generate node-sass-v4.7.2-darwin-x64-57.tgz from the modified node_modules/node-sass folder
3. Next time you run yarn install on the same platform it would just unpack node-sass-v4.7.2-darwin-x64-57.tgz instead of the original one and won't run install scripts

This won't work for every case but it could be a solution for offline CI systems in which you don't want packages to reach our into the internet and run same install scripts every time.

I am trying to install typescript as global package, and yarn spends all time rebuilding stuff instead of installing what I really need now.

I moved to yarn thinking it's faster, and better. I removed any trace of npm (except what comes with node installation); and reinstalled all global packages to yarn.

Yarn is now testing my patience by making me wait for 1-15 minutes for simple package installation. Yarn should be smarter enough to install what's request first before rebuilding other stuff unless requested-package explicitly depends on any of native packages which requires rebuilding.

Environment:

• Node: 9.4.0
• Yarn (global): 1.3.2
• macOS Sierra: 10.12.6
• Macbook pro 15 inch
  
  
  
  • Memory: 16 GB
  
  
  • CPU: 2GHz - Intel Core i7
  
  
  • Storage: magentic (plenty of free space)
  
  
  • Apps running at time of installation: Firefox (6 tabs), Mail.app, and iTerm (with 2 tabs)
  
  

Logs

Fetching Packages takes a lot of time

yarn global upgrade typescript
yarn global v1.3.2
[1/4] 🔍 Resolving packages...
[2/4] 🚚 Fetching packages...
[#############################################################---------------------------------------------------------------------------------------------------------------------------------------] 673/2166

Linking Dependencies is stuck without any progress for minutes

yarn global upgrade typescript
yarn global v1.3.2
[1/4] 🔍 Resolving packages...
[2/4] 🚚 Fetching packages...
[3/4] 🔗 Linking dependencies...
[------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------] 0/2566

Rebuilding all packages -- takes a lot of time

yarn global upgrade typescript
yarn global v1.3.2
[1/4] 🔍 Resolving packages...
[2/4] 🚚 Fetching packages...
[3/4] 🔗 Linking dependencies...
[4/4] 📃 Rebuilding all packages...
[12/17] ⠐ websocket
[2/17] ⠐ expresso: checking for gcc option to accept ISO C99...
[8/17] ⠐ serialport: using [email protected] | darwin | x64
[6/17] ⠐ now: > For the source code, check out: https://github.com/zeit/now-cli

yarn global upgrade typescript
yarn global v1.3.2
[1/4] 🔍 Resolving packages...
[2/4] 🚚 Fetching packages...
[3/4] 🔗 Linking dependencies...
[13/17] ⠈ serverless
[13/17] ⠂ serverless
[14/17] ⠠ solgraph
[14/17] ⠁ solgraph
[14/17] ⡀ solgraph
[-/17] ⠈ waiting...
[-/17] ⠄ waiting...
[-/17] ⠠ waiting...
[-/17] ⠈ waiting...
[-/17] ⠄ waiting...
[-/17] ⢀ waiting...
[-/17] ⠈ waiting...

I keep waiting :-)

another side-effect is that libraries that download and cache precompiles also get wiped and have to be downloaded again: namely nwjs-builder-phoenix's cache of build-target sdks

1. The native package is always rebuilt when updating a package version which is non-native package.

Does yarn cache the binary to the global like npm?

It starts to frustrate me a bit having to rebuild nwjs in every simple installation. Makes no sense

i'm seeing this also. uws is rebuilt every time i do an add/remove operation. yarn v1.3.2

Hi @bestander. We've tried #5314 on our company's monorepo, however it had no effect on stopping the rebuilds of some dependencies for us. We've enabled yarn-offline-mirror and pack-built-packages in .yarnrc as shown in the added tests.

The crux is that running yarn always takes around 40-50 seconds for us, even if the command we ran directly before was yarn too.

@bazyli-brzoska, I've changed the flag to be more explicit and forgot to update the description.
Can you try with flag experimental-pack-script-packages-in-mirror set as "true"?

Is there any progress on this? I see that the pull request is already merged and that it is in the latest release 1.5.1. I'm on 1.5.1 and nothing changed for me. Nevertheless, I'm considering moving back to npm, because waiting 322.70s or 282.69s or even 683.41s (these are really my last three yarn adds) for installing a small rollup plugin or lodash or well, pretty much anything, is just insane.

It would be awesome if major caveats like these would be on your README.md. Users install yarn because it's allegedly "faster" and making this transition from npm to yarn is not a small step, so it would be nice if there was any advance warning for developers, so they can rather think about it one more time, before converting their scripts, polluting their globals and learning the yarn cli.

I thought this is just a problem with my old 32 bit machine but after seeing it the 237th time happen, I googled it and oh, yarn just isn't faster. Great.

Sorry for probably being mean, but I think you get the frustration.

We do accept contributions.

That being said, something to keep in mind regarding packages that get rebuilt when they don't "need" to: build steps are executed after dependencies have been installed. Meaning that the build scripts are allowed to use those dependencies. Meaning that if these dependencies change for whatever reason (which might occur 'randomly', for instance if we optimized two compatible versions of a package into one), then we actually need to re-run the build steps even if those dependencies are not actually used during the build (because how could we know?).

So it's not exactly an easy problem. At least part of the issue comes from the package.json design itself. Tell me about frustration 🙂

@arcanis I don't get this part:

even if those dependencies are not actually used during the build (because how could we know?).

I thought that YARN design was to keep static dependencies+version (yarn-lock) and so there will be no random "updates".
Even though, package json's give a full tree, so why do you say "how could we know" ? Once the tree is resolved, it is really easy to say, if any of dependencies (even deep-level ones) to build X changed or not.

Let's say I have a dependency foo that depends on babel-core and left-pad@^1.0.0, and that this foo dependency has a build script that runs babel on its index file.

After running yarn add foo in my project folder I end up having [email protected] in the node_modules, right? Now let's say a new left-pad version is released (1.1.0), and I want to use it in my project. I'll run yarn add left-pad, which will then resolve to latest, ie 1.1.0.

Now what can happen is that Yarn will "see" that there is two copies of left-pad in the tree, and will also see that they could be optimized: after all, foo depends on left-pad@^1.0.0, so 1.1.0 is compatible as well. So it will remove the previous version, and only use 1.1.0. Because the dependency changed, we need to run the build script again because Yarn has no way to know that left-pad is a runtime dependency rather than a build-time dependency.

Now you might ask "but why not just skip rebuilding when the transitive dependencies changed?". The answer is that some packages (in particular native packages) have dependencies that radically affect how they are built. When it happens, we really, really want to rebuild those packages, or we would end up with incompatible artifacts.

@arcanis, i thikn you misunderstand the issue. the problem here is that these native packages are being re-built _every_ time, even when yarn is run twice (or more) in quick succession. it has _nothing_ to do with dependencies updating - it _always_ rebuilds.

@Spongman I agree with Your latest comment. But as @Spongman correctly mentioned, rebuild happens every time. Even if you do just yarn && yarn — you get two full rebuilds, despite the fact nothing changed in dependency structure.

I tried running yarn add leveldown bcrypt && yarn && yarn as in the OP and only got one build :/ Do you have a command that I can use to repro this behavior?

You can try for example:

mkdir foobar
cd foobar
yarn init
....
yarn add socket.io
      (5 native packages build)
yarn add react
      (5 native packages build again)
yarn remove react
      (5 native packages build again)

(this is under Ubuntu 14 LTS)

foobar billli$ yarn -v
1.3.2
foobar billli$ node -v
v8.9.1

yarn & yarn 

Is not causing two rebuild, but the example with the socket.io, adding react, removing react is causing two rebuilds

I tried the socket.io example with yarn v1.5.1 and didn't get rebuilds when using the new feature to cache native builds. In order to do that you need to use the offline cache. In my ~/.yarnrc I have:

experimental-pack-script-packages-in-mirror true
pack-built-packages true                                         
yarn-offline-mirror "/home/skomorokh/yarn-offline-cache"

When I try as another user without that config, it rebuilds every time still.

Yes, no rebuild now when added these options.
But if only experimental-pack-script-packages-in-mirror true, it still rebuilds.
Why not set yarn-offline-mirror to a default path like "~/.yarn/cache".

Still experimental, but it's an interesting suggestion (cc @bestander). I guess the issue I have with this is that I personally don't like the idea of enabling things without the user explicit consent. It also has other implications: what would it mean to have yarn-offline-mirror set to false, and experimental-pack-script-packages-in-mirror to true? Should experimental-pack-script-packages-in-mirror overwrite the yarn-offline-mirror settings? A bit confusing imo.

That being said, the bug of rebuilding uws when adding left-pad is kinda annoying indeed, and the experimental-pack-script-packages-in-mirror settings only is a workaround. I'm not sure I'll have the bandwidth to look at this in the next week or so, but if someone is interested to land a fix it would be quite impactful.

@arcanis Thank you for the kind response. The frustration comes from the way yarn is presented. At the moment it's just not "fast", definitely not faster than npm and kinda unusable in real world projects. Imho, there should be an info about this or even the workaround by @skomorokh in your README.md file until this is fixed. Or just remove the "fast" from your description.

I would feel the same frustration, if the readme of the angularjs repo would state that it's a lightweight framework. It's just not.

It always repeat to download binary file after yarn-offline-mirror config.

cd \tmp\t
yarn init
yarn add node-sass
Donwnloading Binary from: https://github.com/sass/node-sass/releases/download/v4.7.2/linux-x64-57_binding.node
yarn add node-sass
Donwnloading Binary from: https://github.com/sass/node-sass/releases/download/v4.7.2/linux-x64-57_binding.node

Going to try to debug into this a bit today. I think it actually might be as simple as PackageInstallScripts not checking pkg.fresh to only rebuild newly installed packages. Instead it looks like it's just looping over them all.

After some fiddling around, I'm starting to think that we rebuild things all the time incase someone rm -rf node_modules or the module is deleted from there.

We have a list of detected build artifacts in the .yarn-integrity file, so I'm thinking that a rebuild should only occur if fresh package || module destination dir does not exist || any file in integrity artifacts does not exist || --force flag was passed

So it's a bit more involved than I thought.

I've just yarn added moment, a zero dependency package, to my up-to-date react app, it took 377.97s.

Did it again right after that, it took 389.63s and rebuilt binaries again.

Just for comparison, I then deleted node_modules and entered npm install:
added 1751 packages and updated 124 packages in 360.595s

Switched back to npm now.

OK, some more info for all the eyes on this issue... I've been messing with this thing for a day and a half now and finally realized a few things.

First, most of these rebuild problems are fixed. There is already code in PackageInstallScripts that only re-runs the install scripts for a package is it is marked "fresh":

    if (!ref.fresh && !this.force) {
      // this package hasn't been touched
      return false;
    }

So for example if you use node-sass:

yarn add node-sass <-- builds sass because first install
yarn add left-pad <-- does NOT rebuild node-sass

Also consecutive yarn installs don't rebuild.

yarn add uws <-- builds because first install
rm -rf node_modules
yarn install <-- builds uws because dir was deleted
yarn install <-- does NOT rebuild uws

... but of course there are a couple exceptions ...

On a yarn remove the force flag is set (to fix some other bug I assume, but it's been like that for >2 yrs) so rebuilds always occur.

However, this is arguably "correct" because it is what npm does too:

~/Projects/yarn-test 🐒   npm uninstall left-pad

> [email protected] install /Users/me/Projects/yarn-test/node_modules/node-sass
> node scripts/install.js

Cached binary found at /Users/me/.npm/node-sass/4.7.2/darwin-x64-57_binding.node

> [email protected] postinstall /Users/me/Projects/yarn-test/node_modules/node-sass
> node scripts/build.js

Binary found at /Users/me/Projects/yarn-test/node_modules/node-sass/vendor/darwin-x64-57/binding.node
Testing binary
Binary is fine
npm WARN [email protected] No description
npm WARN [email protected] No repository field.

added 180 packages, removed 1 package and updated 7 packages in 4.03s

You can see that on uninstall of left-pad, npm ran the install and postinstall scripts for node-sass.

The uws package (which is used by quite a few other packages, like socket.io, browser-sync, etc...

something during it's install process changes one of the files (size and timestamp are different).

~/Projects/yarn-test 🐒   ls -l /Users/me/Library/Caches/Yarn/v1/npm-uws-9.14.0-fac8386befc33a7a3705cbd58dc47b430ca4dd95/uws_darwin_57.node
-rwxr-xr-x  1 me  891112136  383636 Nov 21 00:43 uws_darwin_57.node

~/Projects/yarn-test 🐒   ls -l node_modules/uws/uws_darwin_57.node
-rwxr-xr-x  1 me  891112136  378672 Mar  4 11:18 uws_darwin_57.node

yarn sees a file that has "changed" compared to the cache so marks the package "fresh" for re-installation

This then triggers the above logic that re-runs the install scripts because it is "fresh". Yarn is trying to be helpful and "fix" the incorrect files, but of course it doesn't know that the file changed due to the install script. We might have to investigate some way of fixing this, but there is also talk of changing the way that these files are compared (stop running stat on every file), so it might get resolved with that rework.

Hopefully that explains some of the cases here where some people say it works and others say it doesn't.

Thanks for diving deeper there, @rally25rs.

1. Re: using force in delete command.
   That was clearly a quick cut corner bug fix that achieved correctness with a nuclear approach IIRC https://github.com/yarnpkg/yarn/pull/323.
   Should not be hard to remove force and work around the issue.

2. node_modules/uws/uws_darwin_57.node: this file should be in artifacts field of .yarn-integrity and then uws should not be marked as not fresh during linking.
   Something is probably wrong here

@bestander ah good point on 2. I'll try to see if there is a sane way to work that in to the check.

@stereokai In my comment above I note that npm also rebuilds packages on uninstall/remove, so that case is arguably "correct" if you consider npm's behavior to be the standard.

@rally25rs Thanks for pointing that out :)

However, I consider this behavior buggy regardless of the package manager. It's hard to understand why this kind of behavior is even necessary for correct operation. Your OS doesn't reinstall local programs when you add/remove another one, because the apps are already there. I wouldn't expect anything else from a static dependency manager, nahmean?

@stereokai I somewhat agree, which is why I used the phrase "arguably correct" 😆
Your OS also doesn't change where other applications are installed when you uninstall an unrelated one.

Assume you have some dependency tree like:

myProj
  |- depA v1
  |    |-depB v2
  |- depB v1

If you install this, then it would form that same directory structure under node_modules, because it can't hoist depB to the root.

myProj
  |- node_modules
       |- depA v1
       |   |- node_modules
       |        |-depB v2
       |- depB v1

But of you yarn remove depB then the new dep structure is:

myProj
  |- depA v1
       |-depB v2

myProj
  |- node_modules
       |- depA v1
       |- depB v2

so the actual directory where depB v2 is installed to may change whenever a package is added or removed. Those directories aren't just copied from one location to another. The old one is deleted and the new one is copied from cache to the new destination, which means build artifacts that were in node_modules/depA/node_modules/depB would no longer exist, and would need re-built in node_modules/depB.

Likewise, yarn add [email protected] would change the path where depB v2 is installed (in fact I should test that my PR to not rebuild on yarn add actually works in this case)

I suspect that is why these packages are rebuilt every time.

The actual change happened in this commit: https://github.com/yarnpkg/yarn/commit/5300b482c851e2578ac1f3aa4908be4d0289dca8
back in 2016. The file was named uninstall.js at the time, but force: true was added to the flags passed to install. There doesn't seem to be anything specific in the commit comment to indicate _why_.

Would be awesome if there’s a way to avoid the rebuilds in at least _some_ cases.

Anyone is welcome to work on a PR. 😸As I pointed out above, rebuilds already don't happen on yarn install and yarn add (in most cases). I have #5470 open that should eliminate unnecessary rebuilds for yarn add when you have uws in your dependencies (or other packages that change it's own files during install). The only remaining case that I know of is yarn remove.

After having read most of this thread, I don't understand what is happening here.

I have several native packages being rebuilt every time I use yarn add for any other (unrelated) module. It takes around 20 minutes of a very high load on CPUs on my laptop. I just cannot work that way.

Apparently, from this thread, it has been an issue for 19 months.

Is it a bug? Is it being worked on? Is there a workaround? Should I switch back to npm?

Should I switch back to npm?

yes, until #5470 is released.

@vibl

Apparently, from this thread, it has been an issue for 19 months.

It was actually fixed (mostly) a long time ago. This thread just remains open for one edge case that I found a few weeks ago where yarn will rebuild a package if that package changed any of it's files (it thinks the file is wrong compared to what was in the original download, so wants to fix it). And for some open discussion around if remove and upgrade should do a rebuild (it does in npm, but maybe it shouldn't)

Is it a bug?

Maybe. Which packages are rebuilding? The only one I know of that has been a constant issue is uws, so knowing about more would be helpful.

Is it being worked on?

The specific case I mentioned above is fixed in #5470

Is there a workaround?

If the package you are adding does not have any install scripts, then you could use --ignore-scripts

Or, you could check out the branch from the above PR and use that.

It takes around 20 minutes

Wow. I'm curious what package that is.

Thanks for answering.

The packages noise-search, level-rocksdb and jq recompile every time I add any unrelated package with yarn add. My laptop is a bit old, so it takes a very long time to compile them at the same time.

I used Yarn 1.5.1 and node 9.8.0.

@vibl yeesh, noise-search is definitely the culprit behind the long builds...

~/Projects/yarn-test 🐒   yarn add noise-search
yarn add v1.5.1
[1/4] 🔍  Resolving packages...
[2/4] 🚚  Fetching packages...
[3/4] 🔗  Linking dependencies...
[4/4] 📃  Building fresh packages...
success Saved lockfile.
success Saved 73 new dependencies.
✨  Done in 426.15s.

over 7 min on a MacbookPro 😢

Anyway, I installed noise-search then ran a yarn add left-pad with the code from #5470 and it did _not_ cause a rebuild, so I think you should be good to go once we get that merged 👍

Thanks :-)

Should I check back on Yarn in a few days, a few weeks or a few months?

I just discovered that this bug, with the same exact project on the same laptop (dual boot), doesn't occur on Windows 10. But it's there across the latest three Debian versions, Ubuntu, Arch Linux and Fedora. Weird. Didn't try out MacOS yet, but it seems some people also have problems there.

@vibl I'm not sure when our next release will be (I'm not involved in that planning)

@nnmrts yeah I discovered that it's an OS specific thing. From my comments in #5470:

on linux with node 8, when files are copied from the cache to node_modules, the timestamps are updated. yarn decides the files are different and marks the reference fresh.
This only seems to happen on linux node 8 though.
this is because Node v8.5 introduced fs.copyFile which made copies much faster. IIRC that pipes down to the native filesystem copy, so that would explain why it works differently across OSs and only in node 8.

@rally25rs @nnmrts It's most definitely not a MacOS specific thing. Also happens on my Win10 PC

@stereokai Well, this whole issue is not specific. Sometimes packages need to be rebuilt and people still think it's a bug and post it here. Without a solid reproducible repo for every os, we cannot know anything.

Locally caching built modules (from #5314) may help?:

Add .yarnrc to your project with the following:

yarn-offline-mirror "./<your-offline-mirror-path>"
experimental-pack-script-packages-in-mirror true

After the first install, prebuilt modules will live in ./<your-offline-mirror-path>/prebuilt. yarn.lock is also updated w/prebuilt variants

Pulled the latest 66a0143a753cd4ade1a0fffee2174890d564c129, and it seems to be working properly😎

STILL download binary repeatly.

• node v6.13.1
• yarn v1.6.0-20180327.1507
• OS: Ubuntu 17.10 Linux 4.13.7-041307-generic
• ~/.yarnrc:

yarn-offline-mirror "./<your-offline-mirror-path>"
experimental-pack-script-packages-in-mirror true

yarn add node-sass
yarn remove node-sass
yarn add node-sass

On Thu, Mar 29, 2018 at 2:30 AM, Andrew Lane notifications@github.com
wrote:

Locally caching built modules (from #5314
https://github.com/yarnpkg/yarn/pull/5314) may help?:

Add .yarnrc to your project with the following:

yarn-offline-mirror "./"
experimental-pack-script-packages-in-mirror true

After the first install, prebuilt modules will live in
.//prebuilt. yarn.lock is also updated
w/prebuilt variants

Pulled the latest 66a0143
https://github.com/yarnpkg/yarn/commit/66a0143a753cd4ade1a0fffee2174890d564c129,
and it seems to be working properly😎

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/yarnpkg/yarn/issues/932#issuecomment-376989174, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAUAzz07Js-JQyj9n9_rsYq3cpd9Rp8qks5ti9a2gaJpZM4KVN87
.

@snowyu did you delete yarn.lock, node_modules and yarn cache clean ? Does ./yarn-offline-mirror/prebuilt get populated post install?

It's a new project in temp. Yes I can see the node-sass-4.8.3.tgz file in the cache folder.
Now, I run yarn cache clean. BUT STILL download binary repeatly*.

```bash

yarn init -y
yarn add node-sass
yarn add v1.6.0-20180327.1507
info No lockfile found.
[1/4] Resolving packages...
[2/4] Fetching packages...
[3/4] Linking dependencies...
[4/4] Building fresh packages...Downloading binary from https://github.com/sass/node-sass/releases/download/v4.8.3/linux-x64-57_binding.node
success Saved lockfile.
success Saved 152 new dependencies.
Done in 11.98s.

yarn add node-sass
yarn add v1.6.0-20180327.1507
[1/4] Resolving packages...
[2/4] Fetching packages...
[3/4] Linking dependencies...
[4/4] Building fresh packages...Downloading binary from https://github.com/sass/node-sass/releases/download/v4.8.3/linux-x64-57_binding.node
success Saved lockfile.
success Saved 1 new dependency.
info Direct dependencies
└─ [email protected]
info All dependencies
└─ [email protected]
Done in 13.45s.
``

OK, i did simple git repo to reproduce this bug.

https://github.com/vlmonk/yarn-bug-test

yarn perform unnecessary rebuild ttf2woff2 when I try to add zero-dependency left-pad

git clone https://github.com/vlmonk/yarn-bug-test
cd yarn-bug-test
yarn
[ ... building binary package here ... ]
yarn add left-pad
[ ... rebuilding binary packages here ... ]

I can reproduce this in both host OSX system and in docker container with latest node image

NPM works correctly in this case:

git clone https://github.com/vlmonk/yarn-bug-test
cd yarn-bug-test
npm i 
[ ... building binary package here ... ]
npm i left-pad
[ ... don't rebuild binary packages here ... ]

My yarn version 1.5.1

@vlmonk does this still happen if you clone https://github.com/rally25rs/yarn from @rally25rs and use the code in #5470 (branch fix-linking-rebuilding-uws-932)?

@karlhorky yes, yarn still rebuild ttf2woff2 after adding left-pad

# which yarn
yarn: aliased to node /Users/monk/work/yarn/lib/cli/index.js
# yarn --version
1.6.0-0

The ttf2woff2 package comes with files that are changed in the build step. On the next run yarn sees those files changed and reinstalls the package.

Yarn should handle this situation better: It should see that those files changed during the build step and it should accept those changed files as the "correct" files, not treat them as a reason for a reinstall.

I checked this with additional logging (https://github.com/sth/yarn/tree/trace-rebuild). On first install it shows:

build artifacts for ttf2woff2
  modified file: build
  modified file: build/Makefile
  modified file: build/Release
  modified file: build/Release/.deps
  modified file: build/Release/.deps/Release
  modified file: build/Release/.deps/Release/addon.node.d
  modified file: build/Release/.deps/Release/obj.target
  modified file: build/Release/.deps/Release/obj.target/addon
  modified file: build/Release/.deps/Release/obj.target/addon/csrc
  modified file: build/Release/.deps/Release/obj.target/addon/csrc/addon.o.d
  modified file: build/Release/.deps/Release/obj.target/addon/csrc/enc
  modified file: build/Release/.deps/Release/obj.target/addon/csrc/enc/backward_references.o.d
  modified file: build/Release/.deps/Release/obj.target/addon/csrc/enc/block_splitter.o.d
  modified file: build/Release/.deps/Release/obj.target/addon/csrc/enc/brotli_bit_stream.o.d
  modified file: build/Release/.deps/Release/obj.target/addon/csrc/enc/encode.o.d
  modified file: build/Release/.deps/Release/obj.target/addon/csrc/enc/encode_parallel.o.d
  modified file: build/Release/.deps/Release/obj.target/addon/csrc/enc/entropy_encode.o.d
  modified file: build/Release/.deps/Release/obj.target/addon/csrc/enc/histogram.o.d
  modified file: build/Release/.deps/Release/obj.target/addon/csrc/enc/literal_cost.o.d
  modified file: build/Release/.deps/Release/obj.target/addon/csrc/enc/metablock.o.d
  modified file: build/Release/.deps/Release/obj.target/addon/csrc/enc/streams.o.d
  modified file: build/Release/.deps/Release/obj.target/addon/csrc/woff2
  modified file: build/Release/.deps/Release/obj.target/addon/csrc/woff2/font.o.d
  modified file: build/Release/.deps/Release/obj.target/addon/csrc/woff2/glyph.o.d
  modified file: build/Release/.deps/Release/obj.target/addon/csrc/woff2/normalize.o.d
  modified file: build/Release/.deps/Release/obj.target/addon/csrc/woff2/table_tags.o.d
  modified file: build/Release/.deps/Release/obj.target/addon/csrc/woff2/transform.o.d
  modified file: build/Release/.deps/Release/obj.target/addon/csrc/woff2/variable_length.o.d
  modified file: build/Release/.deps/Release/obj.target/addon/csrc/woff2/woff2_common.o.d
  modified file: build/Release/.deps/Release/obj.target/addon/csrc/woff2/woff2_enc.o.d
  modified file: build/Release/.deps/Release/obj.target/addon.node.d
  modified file: build/Release/addon.node
  modified file: build/Release/obj.target
  modified file: build/Release/obj.target/addon
  modified file: build/Release/obj.target/addon/csrc
  modified file: build/Release/obj.target/addon/csrc/addon.o
  modified file: build/Release/obj.target/addon/csrc/enc
  modified file: build/Release/obj.target/addon/csrc/enc/backward_references.o
  modified file: build/Release/obj.target/addon/csrc/enc/block_splitter.o
  modified file: build/Release/obj.target/addon/csrc/enc/brotli_bit_stream.o
  modified file: build/Release/obj.target/addon/csrc/enc/encode.o
  modified file: build/Release/obj.target/addon/csrc/enc/encode_parallel.o
  modified file: build/Release/obj.target/addon/csrc/enc/entropy_encode.o
  modified file: build/Release/obj.target/addon/csrc/enc/histogram.o
  modified file: build/Release/obj.target/addon/csrc/enc/literal_cost.o
  modified file: build/Release/obj.target/addon/csrc/enc/metablock.o
  modified file: build/Release/obj.target/addon/csrc/enc/streams.o
  modified file: build/Release/obj.target/addon/csrc/woff2
  modified file: build/Release/obj.target/addon/csrc/woff2/font.o
  modified file: build/Release/obj.target/addon/csrc/woff2/glyph.o
  modified file: build/Release/obj.target/addon/csrc/woff2/normalize.o
  modified file: build/Release/obj.target/addon/csrc/woff2/table_tags.o
  modified file: build/Release/obj.target/addon/csrc/woff2/transform.o
  modified file: build/Release/obj.target/addon/csrc/woff2/variable_length.o
  modified file: build/Release/obj.target/addon/csrc/woff2/woff2_common.o
  modified file: build/Release/obj.target/addon/csrc/woff2/woff2_enc.o
  modified file: build/Release/obj.target/addon.node

The package file https://registry.npmjs.org/ttf2woff2/-/ttf2woff2-2.0.3.tgz indeed contains those files.

We see this with OS X as well, adding any package with yarn add triggers a recompile of any dependent packages. We have a node-gyp package with native code, it is taking over 1 minute each time another package is added, and there isn't much code in the native module yet (it will get much worse). This is with yarn 1.5.1.

We are using yarn add ../a with relative paths if that makes a difference.

Please let me know if there are any workarounds, or when it will be fixed.

This is with yarn 1.5.1.

This issue was fixed in 1.6.0 which is out quite recently.

I still see this with 1.6. Since moving from npm to yarn a long time ago uws as always rebuilt (or at least yarn has been stuck on uws for roughly 5-10 seconds).

Steps to reproduce:

1. $ yarn outdated
2. pick an outdated package
3. $ yarn upgrade [package]

@grantila can you provide a complete package.json or repo with steps that reproduce this with Yarn 1.6.0?

this is still happening on 1.6.0

you can repro this using this https://github.com/yarnpkg/yarn/issues/932#issuecomment-377112784

I have a simple project and I'm seeing this too. Adding or removing a package seems to trigger a complete rebuild of at least one package every time.

"dependencies": {
    "bufferutil": "3.0.4",
    "chance": "1.0.16",
    "discord.js": "11.3.2",
    "dog-facts": "1.0.3",
    "erlpack": "discordapp/erlpack",
    "flickr-sdk": "3.7.0",
    "html-entities": "1.2.1",
    "node-opus": "0.2.7",
    "snekfetch": "4.0.0",
    "sodium": "2.0.3",
    "utf-8-validate": "4.0.1"
  }

After these were installed, I added the unescape package, which triggered a rebuild of sodium. Then I removed it which triggered a rebuild of what seemed like every packaged that needed to be compiled. Adding this simple package took 36s and removing it took 100s!

EDIT: using Node 8.11.1 and yarn 1.6.0 on Debian Stretch.

@arcanis @rally25rs pleaase reopen the issue, multiple reports of this still happening, even with the merged fix.

I think this is more of a @rally25rs issue :)

@grantila an upgrade will _always_ rebuild all. This is intentional. npm does the same thing (I mention this in a comment somewhere in this long thread.) though we could potentially try to stop doing that. I'm not sure what repercussions it may have.

Everyone else:

In #5680 I point out that this still happens if a package deletes it's own files _(why oh why do they do these things 😿)_ and yarn doesn't track that anywhere (we track what files are created or modified), so it just thinks the package is missing files and rebuilds it.

I suppose we can reopen this, but this has been fixed for most packages. If anyone wants to add "me too" to this, _please_ either provide your package.json, or mention specifically which package is continually rebuilding, since you may have some dependencies that do rebuild and some that do not.

Anyone is welcome to make a PR for this! (see my debugging comments in #5680 )

Sorry for adding more noise, but I'd like to suggest locking this issue and pointing to a new one with this latest information at the top.

I think the issue here has shifted quite a bit and is at least partially resolved. With all the posts here it's difficult for someone new to get up-to-date with what has been fixed and what is still an issue.

I agree with @james-rabbit

Yep, you're right. I'm going to lock this one so that @rally25rs' answer stays visible.

If you have an issue with native packages:

• If it happens with every native dependency, please open a generic issue. This problem should have been resolved, so I don't expect to see one any time soon - still, if you can provide reproduction steps, feel free to open a new issue (and you can link to this one if you like).

• If it happens with one specific native dependency, please also open an issue but don't forget to specify the name of the dependency in the title (as explained, different packages might be rebuilt for different reasons - keeping one issue for each of those will make it easier for everyone to share information).