Definitelytyped: Why monorepo?

Created on 13 May 2018  ·  16Comments  ·  Source: DefinitelyTyped/DefinitelyTyped

  1. I want to see .d.ts for @types/X package without cloning repo.
    But when I open types folder on github I see following error, and X folder is not listed.
    Sorry, we had to truncate this directory to 1,000 files. 3,413 entries were omitted from the list.

  2. I want to see what issues already filed for @types/X and file new one.
    But when I open issues tab I see >2k entries for all the packages... I wonder how contributors even find issues filed for their definitions (or they don't?).

Why typings do not live in separate repos? Isn't this monorepo a total mess?

Most helpful comment

Non-monorepo is a nonstarter. Frequently, contributors modifying something in one @types package will need to modify several downstream packages to fix breaks or enable new functionality; this is an absolute dumpster fire with GitHub's workflow as there is no integrated way to merge those PRs in an atomic fashion while still running sensible CI builds on all of them. If you think you looking at 4,000 folders is an issue, imagine us trying to keep GitHub settings synchronized across 4,000 repos.

All 16 comments

  1. You can navigate directly to the folder under types. For example, in order to see the types for jquery, navigate to https://github.com/DefinitelyTyped/DefinitelyTyped/tree/master/types/jquery .
  2. You can search for the name of the library. For example, if you want to search for open issues relating to lodash, then you can go to https://github.com/DefinitelyTyped/DefinitelyTyped/issues?utf8=%E2%9C%93&q=is%3Aissue+is%3Aopen+lodash ; or you can use the search bar, which will redirect you to the appropriate querystring.

Isn't this monorepo a total mess?

screen shot 2018-05-15 at 16 40 12

The easiest way to do that is to use the TypeSearch. You will find out whether the types are in the registry, and if they are, the link in the npm description will point exactly to their subfolder.

image

Non-monorepo is a nonstarter. Frequently, contributors modifying something in one @types package will need to modify several downstream packages to fix breaks or enable new functionality; this is an absolute dumpster fire with GitHub's workflow as there is no integrated way to merge those PRs in an atomic fashion while still running sensible CI builds on all of them. If you think you looking at 4,000 folders is an issue, imagine us trying to keep GitHub settings synchronized across 4,000 repos.

Frequently, contributors modifying something in one @types package will need to modify several downstream packages to fix breaks or enable new functionality

I'm not familiar with writing @types/ packages, so may be I'm missing something...

But isn't it what semver is for?
What is so special about implementing @types/ package, against any other package in npm?

I expect it to work the same: dependency graph where you have eg. package @types/X and downstream package @types/Y which uses @types/X. Both maintained by different people. When @types/X changed, it is published with different version, so it would not immediately affect @types/Y. Contributor of @types/Y later decide to update to newer version of @types/X and fixes issues due to all breaking changes.

Probably on top of that you have DefinitelyTyped meta-repository, which only maintains list of links to type-repos. And then some publisher script which goes though that list and publishes all packages under @types/ npm-namespace.

Or if it is possible, just allow implementers to publish directly under @types namespace. So we dont need DefinitelyTyped meta-repo at all.

https://github.com/npm/npm also 2k issues.
https://github.com/moby/moby even more.

The best solution is the author maintaining definition, which has always been encouraged.

@franklinyu Number of issues is not what's under question. Taking into account this repo aggregates >4k typings, that huge number of issues is ok.

Real question is: what problem DefinitelyTyped trying to solve by aggregating all typings in one repo?

Unfortunately, semver does not work well for typings packages. The major.minor version of the types package needs to match the major.minor of the javascript package, otherwise people will have no idea which @types version to use for any given javascript package version.

For example, right now if you install [email protected], then you would also install @types/[email protected] (right now it's at 4.14.109). However, imagine there is a bug in the lodash types (and trust me this happens very often), and someone fixes it. Now what do we bump the version number to? By definition, any change to the type definitions is an interface breaking change (or at least an interface addition). But we can't bump the version to 4.15.0 because these types are for 4.14, not 4.15. And we certainly don't want to bump the version number to 5.0.0. So instead we bump the version number to 4.14.110, even though there was an interface change, because really the old interface was just inaccurate.

@aj-r Ok, major.minor version of @type/X should equal to major.minor version of X, and if there is a bug in @type/X you use patch version to fix it. That sounds right.

What does that have to do with original question: monorepo vs separate repos?

Sorry, I should have been clear - I was addressing one of your previous comments:

But isn't it what _semver_ is for?

what problem DefinitelyTyped trying to solve by aggregating all typings in one repo?

What problem is splitting up DT into 4,000 separate GitHub repos going to solve?

Every day a TypeScript team member manages the PR backlog, merging or reviewing several dozen PRs. A bot keeps tabs on the huge volume of PRs we get. We run repo-wide lint rules that catch common errors, and we turn on new lint rules regularly as we think of ones that would help. We find and fix definitions with things broken by future compiler versions.

Splitting this out into thousands of subrepos makes all of that harder, and makes nothing else easier. So why would we do it?

What problem is splitting up DT into 4,000 separate GitHub repos going to solve?

  1. As a user you reach sources faster. You don't have to use some additional app like TypeSearch to reach sources. On npm you have a link directly to needed typing-repo.

  2. As a user its easier to see current issues and file new one.

    • You don't have to search and filter on one huge repo with >2k issues. What if reporter forgets to follow issue naming rules (ie. [@types/name-of-package] Issue description)? You never find needed issues.
    • As such it's less possible to file duplicates.
  3. As a contributor you can clearly monitor how many issues you have in your repository.

    • You don't have to (again) filter anything to get issues for your concrete typing. Less possible to miss needed issues.
    • You have a clear motivation to keep issues number low. Issues number is not shared responsibility anymore.
    • As such it's less possible to have forgotten issues hanging there for years.
      Eg. there is 1 issue page from 2013, 5 issue pages from 2014, and so on.
  4. As TypeScript team member you spent more time on improving typescript itself (eg. jsdoc support), instead of maintaining/merging/reviewing thousands of typings for lots of npm packages out there.
    In mature ecosystem that's what community should do.

    • We run repo-wide lint rules that catch common errors

      Well release lint preset with all the recommended rules, and advise all contributors to use latest version of that preset.

    • We find and fix definitions with things broken by future compiler versions.

      That's what community contributors should do. Version of expected ts-compiler should be clearly stated in peerDependency section of package.json of typing-package, so it warns you when you use it in wrong environment.

You're approaching this from the mindset that typings are actually owned by anyone. The reality is that the great majority of files in this repo were written once by someone who ran dts-gen and checked it in with a few fix-ups, then got touched up once a year by three other people. There is not a strong ownership model here, nor does there need to be.

What happens when the ad-hoc self-declared "owner" of one of these repos decides to stop maintaining it? This happens all the time. How do we keep track of which repo is the current "best"? What happens if they start merging changes that result in things we can't publish on NPM?

This also just does make life worse for people who want to submit fixes. Last year we added a type parameter to the React definition file that required downstream changes in several hundred files. Does anyone want to clone several hundred repos (oh, and which repos?) to make a wide-spanning change, then manage several hundred concurrent pull requests? There isn't even tooling for this anywhere.

As TypeScript team member you spent more time on improving typescript itself (eg. jsdoc support), instead of maintaining/merging/reviewing thousands of typings for lots of npm packages out there. In mature ecosystem that's what community should do.

We literally tried this and it didn't work. DT was solely community driven and it resulted in a pull request backlog hundreds deep. Since then the average time to merge a PR has fallen from 2 weeks to 4 days (which is an intentional minimum so people have time to weigh in on changes), even though PR volume has gone from ~200/month to ~1,000/month.

Ok, I think I have the answer on my original question "Why monorepo?":
Because it is just easier for TypeScript team to maintain type definitions (considering type dependencies, atomic updates for downstream packages, CI, linting etc.)

I still think that the model where TypeScript team takes such a major role in maintaining thousands of packages mostly by itself is a bit weird. Especially in the modular, community driven npm world.

But if this model works for you. Then ok :smiley:

@art-in I don't believe the intention is to manage all types from this one repo, it is to manage types that aren't provided by the packages themselves. If a package wants to publish types for its package, it can do so right in the package. DefinitelyTyped is just a place where types can be aggregated without needing consent from the original package maintainer.

Was this page helpful?
0 / 5 - 0 ratings