Yarn: Idea: support package-lock.json from npm 5

I expect this will change in the future, but... What is the current best practice for handling both package-lock.json and yarn.lock?

These are the options I'm seeing...

1. Don't, and only support one of these in your project.
2. Everyone manually keeps them in sync for every dependency change
3. Use some tool to rebuild one of the locks when you change the other. _(I'm hoping for this one :)_
4. Just eat the pain and wait?

I would opt for option 1 or 4.. keeping multiple files for the same in sync is tedious and error prone.
On the other side, a downside of choosing to support one of both is that you somewhat force your users to use either yarn or npm, but I think that this would be less harmful than different packages for different package managers.

So our projects use either a package-lock.json or an yarn.lock file, the first option.

At work, we are currently in the following weird situation:

• using yarn for 95% of our CI, but one step is using npm, because it is the only way we could make it work inside docker.
• as we knew no way of syncing the files we decided to .gitignore package-lock.json for now, because our minimal version of npm is LTS, which doesn't support ti anyway.
• we will soon (days or weeks) upgrade to use node v8 and npm v5 as minimal version, then we might switch to only use/support package-lock.json and .gitignore yarn.lock, in case there is no option to sync them.

The following approaches would work for us:
A) yarn using the values from package-lock.json for installing
B) yarn using the values from package-lock.json to create/update yarn.lock
C) an extra tool to create+sync yarn.lock from package-lock.json

In the general packager case Yarn could potentially be used for Packagist (there's a prototype floating around), and perhaps even CocoaPods, in addition to NPM. A PHP project, for example, might have a composer.lock and a package.json for managing NPM scripts. Might leaning too far into NPM locking behaviors limit some of these future possibilities for packager extensibility?

If yarn starts managing other types of dependencies, it is still always expected to generate the exact same dependency versions as any other managers of the same dependencies. (yarn install or npm install will give you identical dependencies).

So I would have thought it should always follow the prevailing standard for the type of dependency. I.e., for PHP dependencies it should use composer.lock, for NPM it should use package-lock.json.

The way I see it Composer is to Pear as Yarn is to NPM. Difference being NPM uses NPM where Composer uses Packagist. If Yarn were to maintain its own standards we could prevent gridlock and tooling complexity which could hamstring Yarn and inhibit forward momentum in other packagers.

Is it naïve to think 80% case could boil down to one packager, with additional packagers to add value for 20% and ancillary use cases? It seems such an approach might serve the majority while limiting complexity, allowing innovation on the fringes which could then be generalized and pulled into Yarn as the value proposition becomes clear longer-term.

It has always seemed to me this is what NPM itself is doing right now. I could be wrong.

Yes I agree, Yarn wants to maintain its independent space as far as possible, so it has the flexibility to provide a better experience. Which is why, for example, Yarn's CLI is deliberately not compatible with NPM's.

However, I think lock files are outside of the space where Yarn can reasonably maintain independence. package-lock.json and composer.lock are committed to the repository, alongside package.json and composer.json. These are not tool-specific files, they are instead project-specific files specifying the exact dependency versions that the project is guaranteed to work with.

When NPM wasn't offering the ability to lock dependencies, it made sense for Yarn to create its own yarn.lock. But now that NPM has defined a standard way for projects to specify the explicit dependency versions that they are known to work with, and furthermore it is named similarly to the package.json that Yarn already relies on, it doesn't make sense for Yarn to continue to go its own way.

For Yarn to be a useful tool, it has to allow developers to follow standard models in their projects, so the project can remain tool-agnostic. Which, after all, is why Yarn was built on top of package.json and not its own separate dependency file.

mark

Stack Overflow thread discussing one of the problems resulting from having 2 flavors of lock files:

https://stackoverflow.com/questions/44552348/should-i-commit-yarn-lock-and-package-lock-json-files

Been doing some reading around this today, stumbled across this in the npm docs.

One key detail about package-lock.json is that it cannot be published, and it will be ignored if found in any place other than the toplevel package.

https://docs.npmjs.com/files/package-lock.json

Maybe this is common knowledge, but I certainly thought npm would use the lock file in dependencies. I thought that might be interesting to other people that land on this issue, since I was mostly concerned about people installing my packages with npm 5 because we're not using a package lock (we use Yarn instead).

From reading some documentation on Renovate, it seems to make sense that it doesn't use the package lock in dependencies to avoid bloating node_modules and issues with installing two separate versions of packages.

A second reason for using ranges applies to “libaries” that are published as npm packages with the intention that they are used/require()‘d by other packages. In this case, it is usually a bad idea to pin all your dependencies because it will introduce an unnecessarily narrow range (one release!) and cause most users of your package to bloat their node_modules with duplicates.

For example, you might have pinned foobar to version 1.1.0 and another author pinned his/her foobar to dependency to 1.2.2. Any user of both your packages will end up with npm attempting to install two separate versions of foobar, which might not even work.

https://renovateapp.com/docs/deep-dives/dependency-pinning

Maybe this is common knowledge, but I certainly thought npm would use the lock file in dependencies.

@ryansmith94 this behaviour in package-lock.json mimics the exact same behaviour in yarn.lock:

When you publish a package that contains a yarn.lock, any user of that library will not be affected by it. When you install dependencies in your application or library, only your own yarn.lock file is respected. Lockfiles within your dependencies will be ignored.

Thanks @nottrobin, that's good to know 😄

Can someone of yarns core contributors comment on this issue? I would be very interested to hear their opinion. :v:

@nicojs I mentioned it on Discord before I replied here, if I remember correctly, they hadn't had time to properly look into it, but did want to get to it. I'm assuming I was talking to a core contributor in the support channel.

From Irish's comment above

1. Use some tool to rebuild one of the locks when you change the other. (I'm hoping for this one :)

So who has a tool they can share?

I've used this sequence of commands to sync yarn.lock changes to package-lock.json without manual edits.

# clean up
rm -rf node_modules

# remove previous if exists
rm package-lock.json

# create node_modules tree according to yarn.lock file
yarn install

# generate lock file for currently installed node_modules tree
npm shrinkwrap

# rename shrinkwrap to lock
mv npm-shrinkwrap.json package-lock.json

# adds network metadata to package-lock.json (resolved and integrity sha-1)
npm install

Maybe it would help somebody...

I don't understand why anyone would use yarn over npm knowing they are so inconsistent. We really don't need another lock file.

I can't believe that in 2017 we're working to sync two separate lock files! This shouldn't be needed at all, and more so, we shouldn't even think about doing such thing!

@revolter Could you keep discussion about the state of the ecosystem and exclamations of frustrations limited to e.g. reddit or Twitter? Then this issue can remain focused on whether and how to implement this feature, and those who are subscribed to it will only receive email notifications about that. Thanks!

@ahz Why are you using npm-shrinkwrap? Isn't the point of this thread that NPM 5 now manages its own lockfile?

@nottrobin I am guessing that package-lock.json is only generated if:

package-lock.json is automatically generated for any operations where npm modifies either the node_modules tree, or package.json.

To anyone who's intersted, I created a small tool that can convert package-lock.json to yarn.lock and vice versa. Would love to know if it works out for you: https://github.com/imsnif/synp

@nottrobin My scenario was to generate package-lock.json file for project where there is already yarn.lock file for longer period of time (and newer versions of dependencies exist). Therefore I've used npm-shrinkwrap to generate lock file for current node_modules tree (installed by yarn). npm install would generate lock file with possibly newer versions of packages. And the trick is that npm-shrinkwrap.json has same structure as package-lock.json (excepts some metadata that are added by npm install at the end).

Now is 2018. This issue is very important in setting consistent version. Core team need investigation in this topic. More duplicated issue will be added and more problems it create than resolve.
Possible solutions from Yarn core team to choose:

• Will synchronize two file in every action to give freedom of choose packages (good way)
• Deprecate own lock file in favor of default package-lock.json (best in my opinion)
• Will be only created on demand
• NPM will add possibility to add custom changes in package-lock.json and won't be removed (merge - best for everyone)

NPM won't remove their lock file. Yarn can (pleas don't) do the sam and state will be preserved as is (two lock file) - This issue pleas then set to "wontfix" and add documentation about decision.
Need decision of core team what approach we take.

NPM won't remove their lock file.

$ cat .npmrc
package-lock=false

So far this is best solution that works for me. I tried using package-lock file, but unless https://github.com/npm/npm/issues/17722 is resolved I don't see how npm lock file can be considered even barely usable in any large team. So it makes sense for yarn to consider package-lock.json file and update it when changes to real lock file is made, but I don't see Deprecate own lock file in favor of default package-lock.json (best in my opinion) being possible solution any time soon

On our CI we simply use:

if [ -e 'yarn.lock' ]; then
    yarn install
elif [ -e 'package-lock.json' ]; then
    npm install
fi

This has to be one of the most popular issues ever to be (apparently) completely ignored by the project maintainers - getting on for a year now. For me this is quite a significant blemish on Yarn's otherwise pretty good reputation.

For those using both npm and yarn together in the same project, how are you maintaining the CLI differences in your scripts?

Using multiple package managers in the same project sounds like using multiple version control systems in the same project, or multiple task runners for the same tasks. They do different things in the same use case which is bound to cause undocumented conflicts.

yarn and npm may look very similar in terms of features, but they are very different in implementation.
Giving yarn the freedom to add more features in their yarn.lock without going through npm sounds a lot more simple and future proof to me. Moving over to package-lock.json will cause synchronisation pains in the future when npm decides to change something for their own feature set.

Yes, npm is default right now in NodeJS, but that's a known trade-off when picking any alternative package manager. I think the competition is a good thing; as proven with npm's numerous yarn-inspired features after a long period of stagnation.

At most, if this hasn't been done already, I think yarn should detect other lock files in a project, warn about them when trying to run a command and provide a useful suggestion.

Hey @jahed - we're actually in the process of implementing the very same. Warning when another lockfile is found and giving the ability to import from it. Here's a PR of mine doing the latter as a first step:
https://github.com/yarnpkg/yarn/pull/5745

I'm hoping we'll be able to merge it soon. You can follow more here if you like:
https://github.com/yarnpkg/yarn/issues/5654

Merging into #5654.