Ninja: Consider adding a second, "heavy" binary

One half-baked idea would be to make the core ninja code a "real" library, and then keep the regular ninja binary reasonably small and focused, and then add a second binary somewhere (contrib/monster or what have you) that links to the ninja library and adds all these features. That binary could then depend on all kinds of stuff.

This would be cool. FYI, we used ninja in the underlying bsb build system for ReasonML/BuckleScript projects, in general, we enjoy the performance of ninja, but it still feels a bit limited without a persistent mode, I understand it might introduce more deps or make it less cross platform after introducing more features, but it makes sense to export ninja as a library

some configurable frontend thingy with a message-passing interface #1210

That PR doesn't add any runtime dependency at all. What exactly is it that you're concerned about there?

Because if it's Git repository/distribution size or maintainability, having a second binary would hardly help.

I'd like to caution that many projects using Ninja do not expect the users to have ninja pre-installed on their machine, and such a project provides a wrapper script that downloads Ninja binary in the project tree when Ninja is needed. If Ninja is no longer a standalone executable, it MIGHT break those projects.

e.g. download at https://chrome-infra-packages.appspot.com/dl/gn/gn/mac-amd64/+/latest using Python: (python2) zipfile.ZipFile(io.BytesIO(urllib2.urlopen(url).read())).extract("gn", path="path/bin"), (python3) replacing urllib2 with urllib.request.

Therefore, producing a standalone executable is always needed.

I'm interested in pushing this forward. My current thought it so work on making the main() function for ninja very small and pushing all the current functionality inside ninja.cc into a library boundary. I'm going to try sketching out some of these ideas in code to see if we can turn that into a discussion on the various technical tradeoffs.

I'd like to caution that many projects using Ninja do not expect the users to have ninja pre-installed on their machine, and such a project provides a wrapper script that downloads Ninja binary in the project tree when Ninja is needed. If Ninja is no longer a standalone executable, it MIGHT break those projects.

e.g. download at https://chrome-infra-packages.appspot.com/dl/gn/gn/mac-amd64/+/latest using Python: (python2) zipfile.ZipFile(io.BytesIO(urllib2.urlopen(url).read())).extract("gn", path="path/bin"), (python3) replacing urllib2 with urllib.request.

Therefore, producing a standalone executable is always needed.

If I ever see a source tree mutating itself by downloading and executing rogue unsigned binaries, I'll probably not touch that software anymore. An important piece of design is missing here.

@johan-boule no it is not the software mutating itself. A large project (e.g. Chromium) may use a script that handles setup steps for developers, and the setup script has this step to fetch binaries.

@Leedehai I was just reminding that most developers do not accept to loose control of what is being downloaded and executed on their machines. Usually it's made acceptable by using a trusted package manager to do this work. This is the piece of the system design I was alluding to.

I've created an exploratory PR for this effort at https://github.com/ninja-build/ninja/pull/1729. I've gotten quite a few comments from jonesmz, but nobody else. That may be understandable, the PR is quite long and rather meandering as I figured out the structure of ninja. I thought it would be useful for me to summarize my design here for discussion.

1. We need a system for doing logging that allows library clients to capture log messages and emit them their own way. jonesmz suggests a design similar to boost, but constrained to a few points:
   
   
   
   • Multiple backends, so I can dynamically target log files, console output, or a logging server on the network, as I feel like at runtime.
   
   
   • Dynamic variables captured with each log statement, which can include the current functions name, or the current timestamp, or what have you.
   
   
   • Easy to replace the functionality using macros. E.g. I can provide my own logging statements that customize what gets logged, by defining a macro that i call.
   
   
2. Introduction of a ninja namespace. This is just nice when you're a library. Removing 'using namespace std' anywhere it is still used in headers.
3. Separation of a ui layer which is in charge of parsing commandline arguments into various structs for holding configurable behavior. Pass these structs into configurable functions to change behavior.
4. No more logic for "things are going badly, exit now". So no ExitNow function.
5. Creation of an Execution class to hold much of the logic currently in ninja.cc/NinjaMain. This excludes parsing command-line arguments.
6. Removal of global state - some gets pushed into State, which can be reset on rebuild, some gets pushed into the new Execution class (some options, configuration), some become static (debug_flags).
7. Introduction of public header files. They'll live in inc/ninja/*.h. This includes:
   
   
   
   • build_config.h
   
   
   • depfile_parser_options.h
   
   
   • logger.h
   
   
   • execution.h
   
   
   • ui.h
   
   
   • version.h
   
   
   • win32port.h
   
   

I'm going to start a fresh branch re-implementing my changes on top of master trying to make my intent clearer. I'd like to have enough of a discussion with the maintainers on my approach to be reasonably certain that my approach is in the right direction.

@nico @jhasse is this direction acceptable to you? If not, what would you rather see?

Minor correction - point 6, nothing is added to debug flags, rather statics are removed from them and become data members of either State or Execution.

@jhasse or @scivision you seem to be the most active maintainers, is this a topic either of you are interested in?

I'm just a Ninja user. I haven't yet felt the need for this level of rearchitecting of Ninja though I can at least superficially understand the proposed benefits and the need for them.

Thanks for chiming in @scivision , my mistake, I saw that you landed some commits in February and thought it meant you were a maintainer. I appreciate your perspective, though. Looks like I probably should have tagged @nico to get a maintainer.

Given that nobody has engaged in this conversation or shown interest perhaps they agree with you and don't have time to say so.

I also haven't felt the need for this. I'm very interested in #1600 though.

Okay, thanks for the reply. @jhasse do you have a place you'd prefer to have this discussion? The design I'm proposing would allow for creating a "heavy" binary. It would also allow for creating a different front-end, but it would do so rather differently from what was originally proposed by collincross.

Making the appropriate architectural changes to allow the Ninja code base to be consumed as a real library (statically linked, even) would drastically simplify the implementation of #1600 .

Personally I'm very interested in library-fication for the sake of integration into IDEs and other tools (parsing of ninja files, etc) without needing to call an external binary.

This issue is the right place for the discussion about this.

Okay, so at this point what I'm looking for is either:

1 - "Yes, this plan looks good at a high level, proceed and I will review details as they are available to make sure this is something we want to do for ninja."
2 - "No, I don't like this plan, I'd rather see something more like the following: "

I'd like to get this from a maintainer. @jhasse could you elaborate on which of those two answers better fits your opinion? I believe this design supports #1600 which you are very interested in. Do you agree? Would you like to see something else for #1600?

3 - "From what I can tell the plan looks good. But for me there are other issues I'm more interested in and I don't have enough time left to review the changes."

I haven't looked that much into how the changes would help #1600. As I understand it, it would replace it.

It would replace it. #1600 has a small implementation of part of protobufs embedded in it to allow data passing to a separate frontend process. @nico at one point (in #1210) was against adding protobuf to ninja in any way. Nico suggested creating two separate binaries, a light one that is the default ninja in a release and a heavy one that could have things like protobufs and having a configurable frontend output.

The goal of my design here is to make it architecturally feasible to do this. I'm trying to define a core ninja library and its interface. This would allow the ninja project to create a heavy binary and a light binary, both around the same library. It would also allow consumers of the project to link against the library themselves to customize the behavior further.

I originally thought that a PR would be a good way to have a discussion around the overall design. I didn't get any feedback (except for jonesmz, who was awesome). So now I'm trying to have a high-level discussion before I expend more effort on code.

From a maintainers perspective it's easiest if I create all of the code and you only review it when its complete. From my perspective that is a dangerous waste of time, since the answer could be "nope, don't want anything like this". I'm hoping the ninja maintainers can meet me halfway and discuss with me what they'd be willing to accept before I pour more time into the code itself.

Alright, I've given this several weeks and attempted to reach out to maintainers using every means available to me. I'm going to consider this effort as "not interesting enough to be bothered to discuss" from the maintainers perspective.

I'm going to open discussion with the maintainers at the splinter project to see if they are interested in something similar.

If any maintainers come along and want to comment on this definitively, I'd appreciate the clarity.