Typescript: Add new moduleResolution option: `yarn-pnp`

I think we'd prefer if yarn and npm settled on a common format - since npm is working on tink which does the same thing with a .package-map.json. Regardless, we have a conventional stance right now to not execute external JS as part of a build (maybe that'll change in the future, but right now that's our standing philosophy), so something like this would, minimally, be blocked on https://github.com/yarnpkg/yarn/issues/6388 for us.

Daniel has a much more detailed comment on the pnp rfc here

That sounds perfectly reasonable, thanks @weswigham

yarnpkg/yarn#6388 will likely not be acted upon (edit: it did). People wishing to generate the dependency tree in a specific format can easily do so using a postinstall step, as shown in the sample app. I'd advise against relying on that for tooling integration, though, since it puts an unnecessary burden on the user.

The most generic way for you to do this is to simply expose resolveModuleName within the tsconfig files (https://github.com/Microsoft/TypeScript/issues/18896), just like we did for ts-loader (https://github.com/TypeStrong/ts-loader/pull/862). It would be a useful feature that would not only unlock PnP for tsc users, but also other use cases for other tools authors. I believe it also wouldn't conflict with your default policy of not executing external code, since it would require an explicit acknowledgment from the user.

As I said - we've currently got no plans for a tsconfig.js (and I bring it up often) or a tsconfig managed plugin system. We have no current plans to expose any ways to execute 3rd party code during an invocation of the compiler, as we have a very strong incentive to retain complete control over the compiler's performance profile and the perception thereof (especially when in an editor). I definitely know _how_ I'd like to expose the ability to do something like this, were we to accept it, since I designed an extensive plugin model for the compiler in the past (before it was declined).

Not to say we won't revisit it at some point, but similar suggestions have definitely been declined repeatedly in the past. A static format is much easier to deal with.

I understand, I was simply informing you that waiting on https://github.com/yarnpkg/yarn/issues/6388 would likely not be an option 🙂

Another thing I forgot to mention - in a world where tsc would have built-in support, the resolution name should be pnp, not yarn-pnp. Plug'n'Play is a generic specification designed in such a way that it can be implemented by different vendors, not only by Yarn.

I'd just love if resolveModuleNames was easy to override in tools such as tsc or VS Code, since yarn pnp isn't the only use case for this, and module bundlers being infinitely customizable means the only thing holding usage back is really the editor support (and if you want compatibility with the CLI compiler). The feature is basically "right there" but is hard to use without straight up forking.

There's always going to be a demand for custom ways of resolving modules. The node ecosystem is someone rigid in this regard (because of the node_modules convention being "untouchable"), but the community has done work around it (eg: Webpack custom resolvers and aliases). Now there's also pnp and tink. At HubSpot we have our own bpm. Pretty much all tools make this reasonably easy to work with (Jest, Webpack, stuff like eslint-plugin-imports, and so on. I think Flow now has that feature too?). The TS language service has resolveModuleNames, as mentioned above. If I understand well, there's even a custom resolution hook for ES6 modules in node now?

So the last bastions to make things friction-free are the language service consumers such as tsc and VSCode. If we have that, we're golden.

Another thing that comes to mind: right now because the rest of the toolchain allows customizations but the typescript ecosystem does not, there's additional friction in keeping stuff in sync: whatever I do in my webpack config, I have to replicate in my tsconfig. With better integration of custom resolvers, we could remove this friction by making them behave the same. I could have a custom resolution algorithm and have webpack and VSCode and Jest honor it without changing all the configs one by one. That would be huge. Awkwardly, this is trivially easy to do by forking the pieces involved, but that's not very scalable.

We have no current plans to expose any ways to execute 3rd party code during an invocation of the compiler, as we have a very strong incentive to retain complete control over the compiler's performance profile and the perception thereof (especially when in an editor).

I understand what you're saying but it would be nice if the TypeScript team could take a step back and look at the bigger picture here. It makes sense that you care more about the performance of TypeScript, but many of your users test their applications using CI systems and it matters to them how slow the installation of their application is just as much as the speed of the type-checking/compile process.

For example, installation of the application I'm currently consulting on takes ~2 minutes on CI. I would like to get that number down.

Is there anyway that TypeScript can support pnp without you supporting custom resolvers? If as @arcanis says, it's a "generic specification", I don't see why it shouldn't just be supported out-of-the-box.

I hope we can get this feature some way or other, otherwise it just seems like we are going to be permanently doomed to slower installation times.

We can't support it directly as-is, but as @arcanis points out elsewhere, you can probably write a short postinstall script that converts the dynamic .pnp.js file into, say, the paths tsconfig option. I haven't attempted it myself yet, but I can't think of why that shouldn't work.

I've gone the dynamically generating tsconfig route for some of our stuff (we actually made a VS Code extension for this for our home grown resolution mechanism).

There's some awkward limitations there, such as nested dependencies when you have multiple version of the same package. (eg: package Foo depends on Bar@1 and Baz depends on Bar@2, and your app depends on Foo and Baz, you're out of luck as far as I know. I dont think tsconfig aliases can express this).

@weswigham I'm wondering something - you might be aware that the Node project is working towards standardizing loaders. Doesn't it make your approach impracticable on the medium-long term?

Something to mention regarding the perfs - we actually noticed slightly better perfs in Flow when it uses PnP, which makes sense given that the PnP resolution removes a lot of stat calls.

Some of the current thoughts around loaders involve having ways to properly sandbox them, e.g. not allowing completely arbitrary code but have clear boundaries and potentially allowing to run the hooks out of the main thread. Not saying that will happen but we are trying to take the security implications seriously.

I worked a bit on a script to run at postinstall and write dependency paths to a tsconfig as mentioned by @weswigham . There are issues making this solution very sub-optimal, even if Typescripts resolution is used for the type checker only.

1. The entire dependency tree needs to be flattened to a path list in tsconfig. With only direct dependencies, typescript won't be able to resolve types in nested dependencies. Since there's no way to tell what package is relevant for typescript this list gets very long and seems (I haven't benchmarked in any way, sorry) to slow down type checking in vs-code severely.

2. The tooling experience gets really wonky. Typescript will know, and hint, about all kinds of packages that pnp.js will not let you import since they're not direct dependencies. When auto-importing, typescript/vs-code will also translate module names to Yarn cache paths, which is not wanted.

3. @types/ packages have to be merged down in tsconfig/paths to their "real" package counterparts.
   I don't know what implications this has for the compilation process, but it sure feels like something that could cause problems.

Typescript would probably need a config property for overloading parts of the module tree in "node" resolution mode, in a way that can describe nested dependencies, to make a solution like this work reliably.

@weswigham would you be open to a middle ground solution?

I want to keep the .pnp.js file turing-complete, because it provides a lot of modularity that I suspect we will need. Simultaneously, I understand some of the concerns you have, so let me summarize them to make sure I properly captured the essence of the problem:

• You don't want to provide code hooks into the tsc compiler because any slowdown (that you wouldn't be able to control, by design) would look bad on the tsc team rather than on the people writing a code hook.

• Simultaneously, you don't want to provide code hooks into the tsc compiler because, since it's used by vscode, it could lead to some kind of code execution as soon as a TS file is opened in vscode.

While I don't think those premises are as bad as you think, I don't necessary want to challenge them - it's clear they are a general philosophy of your teams and I can respect this.

Considering your requirements and mines, what would you think about having a package (let's say tsc-pnp), authored by Yarn, that would contain the hook without the data tables? Then this package would read the .pnp.js file (without evaluating it), extract the data tables, and setup the hook using those as source.

Basically, something like this:

.pnp.js

const staticTable = /*table-start*/{
  "lodash": "./cache/lodash-1.0.0",
}/*table-end*/;

exports.resolveName = name => staticTable[name];

tsc-pnp

const pnpJs = readFileSync(pnpJsPath, `utf8`);
const [,table] = pnpJs.match(/\/\*table-start\*\/(.*)\/\*table-end\*\//);
const staticTable = JSON.parse(table);

exports.resolveName = name => staticTable[name];

This would fix the problems mentioned above:

• You would keep control on the resolution performances, and would be able to ensure that we don't put a huge while(true) somewhere by mistake that would affect your teams metrics.

• VSCode wouldn't execute untrusted code by default - you would have the control on when and why you want to upgrade the tsc-pnp package to new versions, and it would go through the classic PR processes, making it traceable.

Now, you might wonder "what's the difference with simply using a json file?" - the difference is that in this case Yarn keeps control over the layout of the .pnp.js file, treating it a bit as an opaque resource. That would give us the latitude we want/need to make changes. I admit even so I find the situation slightly unfortunate (it won't work with different PnP implementations), but it seems the best way to move forward on this.

Would your team be open to discussion on such a solution? I expect some refinement to be needed, of course 🙂

My best idea yet, I think (particularly with regard to VSCode's context): starting from the v2 we'll be able to make bundles with different set of features (since everything will become a plugin on top of the API). In particular, this means we'll be able to make a version of Yarn that just reads a lockfile and generates a .pnp.js file.

So a solution would be, if you don't trust the .pnp.js file in the repository, to simply regenerate it from within VSCode (in this case it would ship with a small "Yarn microkernel"). This would be fast (zero network because it would just consume the existing lockfile data, zero I/O, because with our v2 model we advocate for the package archives to be stored within in the repository), and secure (no untrusted code would run when opening a file).

What do you think? Yarn is starting to move to TS, but we would really like to get TS support for PnP as it makes the things slightly awkward otherwise. Flow has had support for PnP since even before the release 🙂

I thought PnP looked very interesting, and wanted to try it out before I realized VSCode and Typescript would not be able to find d.ts files correctly. Typescript is such an important tool, that I think trying out PnP will be blocked until it's supported in a reasonable manner by TS. 😅

Not only does it block yarn-pnp, but with TypeScript picking up a lot of steam, it's blocking any kind of innovation in the entire ecosystem unless its Microsoft approved, which is really not great.

For what it's worth the PnP runtime is now decoupled into its own package, which should solve one of the main concerns the TS team had (they can now simply read the PnP data without executing any third-party code). I'd be more than happy to finally meet and discuss how to make this happen, but I don't feel like the ball ever was in my camp.

Sorry if this is kind of a "+1" comment, but it looks like there are already solutions proposed and this is in a state where all that's needed is commitment from the TypeScript team.
We've tried to use Yarn PnP in the Jest repo before and post migration from Flow to TypeScript, and this is currently the biggest blocker, the other somewhat big one being ESLint's import rules.
I think this is really essential to helping PnP gain traction, because it enables other large projects that depend on TypeScript for their type checking / build process to use PnP, helping it mature and become more visible as a practical alternative to node_modules 🚀

I just ran in to this. I'm not even _trying_ pnp -- I needed it due to duplicate dependencies in nested_modules directories in a monorepo....

Any movement on this issue would be very much appreciated.

I have created tsc-pnp - a drop-in replacement for tsc and a VSCode extension that adds PnP support to TypeScript language service. Would love to hear feedback!

Oh nice! Fwiw @vlasenko has been working on a similar tool called pnpify that we'll be shipping along with Yarn 2. It will bring native support for tsc, VSCode, and other similar tools that lack builtin support (we emulate a virtual node_modules directory, which is often good enough to fool such tools).

Of course builtin support would still be beneficial to everyone (from a developer experience point of view of course, but because it would enable new features that aren't possible today), but at least we now have some control on the situation. Feel free to ping us on Discord to discuss your extension, @ark120202! Joining our efforts would likely be beneficial to everyone 😃

I very much believe in the principles of Typescript for not opening up the API for integrating plugins and that what makes Typescript tooling way more reliable than many other build tools.

As @weswigham said, Node and package managers should settle on a single standard. Yes, the innovation is great, but the pace of adoption cannot be the same and makes the entire ecosystem such a mess

We have no current plans to expose any ways to execute 3rd party code during an invocation of the compiler, as we have a very strong incentive to retain complete control over the compiler's performance profile and the perception thereof

@weswigham, tsc may not intend to call any hooks, but it does run under whatever Node.js runtime and libraries the user has.

For example, the PnP project could produce a Node.js executable that supports PnP natively by doing something like shimming the entire fs module.

This is in fact exactly how tink exec works. Is that deep (and still arbitrary) manipulation preferable? I would think the answer is "no".

(especially when in an editor)

That's a lost battle entirely. There are commonly so many moving parts that slow things down: other plugins, remote file protocols, etc. Even beyond that, again, TypeScript is merely the guest of whatever IDE the user has chosen, which again could be doing tink-level intrusion.

Personally, I'd look at PnP like that: akin to a file system; a user-chosen, low-level, cross-cutting concern that may impact performance. (But PnP is not literally file system calls because that's a poor way to abstract/implement module resolution. RIP tink.)

Node and package managers should settle on a single standard.

Agreed. Hot take: the only (good) innovation in the Node.js package managment space comes from Yarn.

Policy question aside, @arcanis I believe there is only thing lacked by the PnP API for TypeScript: inferred imports.

When declaration files are generated, the type inference can produce types that are not named in the source file. Formerly, these would simply fail.

Then TypeScript came up an algorithm to attempt to automatically produce a import for these. (Short version: 1. Try node_modules 2. Try relative 3. Fail if the relative path contains "node_modules" in it.)

import { a } from 'a';
export const b = a;

import { a } from 'a';
declare export const b: import('some/module').SomeType;

Either the behavior would go back to what it was before (a regression, but not necessarily a deal breaker), or the PnP API needs a reverseResolve that calculates an import from one file to another (if one exists).

I'm confident tink can be considered obsolete at this point; the maintainer Kat Marchán left the company and the last notable activity was in March.

Given the operational impact (significant decrease of boot times, non-trivial reduction in filesystem operations, reduction in deployment package sizes, ..) of this implementation, we would be more than happy to support this integration where ever we can.

In order to achieve a similar operational effect we currently rely on similar packaging patterns as used in deployment pipelines of frontend code. This effects a significant disparity between the development and production environment, and is definitely not a desirable status quo for us.

I'm confident tink can be considered obsolete at this point

Just a clarifying note: According to the npm roadmap, the tink approach is not dead and scheduled to land in the npm CLI at some point.

But of course I agree that tink/pnpify are only good solutions for a transition period.

https://github.com/entropic-dev/ds is expected to be similar to tink in some ways

According to the npm roadmap, the tink approach is not dead

Maybe, but the repo still hasn't been touched since March and AFAIK no anologous development is happening by npm. Whereas the last PNP master commit on https://github.com/yarnpkg/berry was Saturday.

Node and package managers should settle on a single standard. Yes, the innovation is great, but the pace of adoption cannot be the same and makes the entire ecosystem such a mess

New features in Node.js package management work like this:

1. Yarn RFC
2. Yarn development
3. Eventually npm development

I am unaware of any other process.

For example, with npm v7, features moving to step 3 are workspaces, resolution overrides, and yarn.lock.

PNP has completed step 2. A couple years from now it will probably complete step 3.

not opening up the API for integrating plugins and that what makes Typescript tooling way more reliable than many other build tools.

Nit: typescript is compiler, not a build tool like Make or Ant or Webpack.

In any case, tsc already calls my FUSE code which I might be doing terrible, horrible things for. It already calls pnpify though it's not the ideal solution. It could just call pnpapi too.

Any movement on this?

What's blocking this right now?

1. The last update from the TS team on this issue is Nov 2018 by @weswigham stressing the desire to not run external hooks, for stability and performance.

2. PnP can improve performance due to lower number of stats calls.

3. PnP has introduced a static file format (though the primary interface is JS).

4. The similar npm effort tink has been dead for over a year.

5. pnpify tsc works, by hooking the fs module.

6. PR #35206 was filed in Nov 2019 for native support (currently open).

7. @yarnpkg/plugin-compat automatically applies that patch.