Ninja: Show subcommand output as it happens when only running one command

I thought we had a bug on this, hrmm... someone in the past had tried hacking something like this in -- when we're only running a single command (and we know, due to being in control of the graph, that we won't start another until that one is done), we ought to show that command's output "live".

I tried to attack this problem at subprocess launch time by not creating pipes, but I’m a fool when it comes to subprocess control and what I did just caused ninja to hang (presumably waiting for some fd to close). But I think my approach was sub-optimal anyway. I think what we want is to always create the pipes, but when we reach a state where everything is waiting for one command to complete, start streaming what’s in its pipe to stdout. @martine, if you’ll drop me a few hints about where/how to approach this, I can take another shot at it.

I'm not sure, but where I'd start is: Subprocess wraps each subprocess and stores its output as it comes into buf_. Then there's a CommandRunner that manages running a bunch of those. See RealCommandRunner::WaitForCommand, which has an array of commands it's waiting for (so you could check if that is of size one) and then calls .DoWork() to read some output from that command. Each time .DoWork returns the subcommand will have either written more output or completed, so that loop is probably the place to stream the output.

Here's a paste of the existing code with a new addition:

  while ((subproc = subprocs_.NextFinished()) == NULL) {
    if (subprocs_.size() == 1) {
      // new code, we know we're waiting for only one process
      // perhaps figure out the one subproc here, then loop calling dowork and tracking what part of the buf we've printed
    }
    bool interrupted = subprocs_.DoWork();
    if (interrupted)
      return false;
  }

It's kinda ugly to special-case the single process there, though, maybe there's some way to refactor this code to have just one flow. And also this function shouldn't be directly printing, it should take a pointer to the caller's BuildStatus object and send its incremental updates through that. (That makes stuff like ninja -v stil work right.)

This also breaks an assumption made on Windows, where we parse a command's output before displaying it. But I think we can worry about that later and maybe just disable this feature when we're in that state.

I was thinking about fixing this as well, haven't got time to give it
priority. More thoughts (Linux):

• If we simply keep using 'pipe', the output of the commands will be
  buffered by libc/OS before it gets to ninja (see e.g.,
  http://www.pixelbeat.org/programming/stdio_buffering/), which may be
  disappointing.
• If we are buffering anyway, we could enhance WaitForComand to allow it
  returning on 'timeout', (passing a timeout to pselect(), adding an extra
  value to ExitStatus.
  Then this code:
• if (!command_runner_->WaitForCommand(&result) ||
  result.status == ExitInterrupted) {
  status_->BuildFinished();
  *err = "interrupted by user";
  *
  could be extended to either print partial output if one process is
  currently running OR update status to print '[XXX] YYY is still running /',
  which could be nice for long-running silent commands, as e.g., linker.

On Fri, Jun 14, 2013 at 8:17 PM, Evan Martin [email protected]:

I'm not sure, but where I'd start is: Subprocess wraps each subprocess
and stores its output as it comes into buf_. Then there's a CommandRunnerthat manages running a bunch of those. See
RealCommandRunner::WaitForCommand, which has an array of commands it's
waiting for (so you could check if that is of size one) and then calls
.DoWork() to read some output from that command. Each time .DoWorkreturns the subcommand will have either written more output or completed,
so that loop is probably the place to stream the output.

Something like:

while ((subproc = subprocs_.NextFinished()) == NULL) {
if (subprocs_.size() == 1) {
// new code, we know we're waiting for only one process
// perhaps figure out the one subproc here, then loop calling dowork and tracking what part of the buf we've printed
}
bool interrupted = subprocs_.DoWork();
if (interrupted)
return false;
}

It's kinda ugly to special-case the single process there, though, maybe
there's some way to refactor this code to have just one flow. And also this
function shouldn't be directly printing, it should take a pointer to the
caller's BuildStatus object and send its incremental updates through
that. (That makes stuff like ninja -v stil work right.)

This also breaks an assumption made on Windows, where we parse a command's
output before displaying it. But I think we can worry about that later and
maybe just disable this feature when we're in that state.

—
Reply to this email directly or view it on GitHubhttps://github.com/martine/ninja/issues/545#issuecomment-19469960
.

Maxim

Hmm, I think I'm a bit out of my depth here, so I think I'd better not sink any more time into this one. Sad to say I'm using make instead of Ninja solely because I can watch the progress of long-running sub-jobs.

It should also be noted that some applications (e.g. Meson) will print ANSI color coded log messages when they are writing directly to a terminal but plain text when their output is forwarded to a pipe or a file.

There are really two cases where you would want to watch the output in real time: regenerating the Ninja file and running the test suite. Neither of these is run concurrently with any other build step. Maybe a new rule variable, let's say unbuffered_output, could be created. When set to true, Ninja would not grab this step's output but rather would let the child process write directly to stdout/stderr.

I am recalling there was a pull request once doing exactly what you are proposing above, which I couldn't find right now. Also, there is a pull request which I've published recently, https://github.com/martine/ninja/pull/629, where I tried to implement this as I though would be right. May be you could try both and leave a feedback. ;)

In addition to the use-cases you are specifying there is another (weak) one, printing long command output in real time (especially when this tool is an "alien" build system building a 3d party library) gives you a chance seeing its output and catching unexpected warning printed by the "tool".

Another use case for this: I have «exe»/debug targets which will build and launch «exe» with a debugger (like gdb). Presently it’s not possible to start an interactive command from ninja so I use application scripting to launch gdb in a new terminal window.

For this use case, it would be ideal if ninja would allow the task to inherit ninja’s stdin/out/err file descriptors, but such solution does conflict with #629, and it doesn’t really allow ninja to detect, that the task is generating output (for that, we would have to let it write to a pipe instead, but then we lose the tty).

@sorbits (re https://github.com/martine/ninja/issues/646#issuecomment-23972234): the reservation is that the current implementation doesn't print ninja status once command running solo starts spewing its output before it finishes.
Imagine running a 'configure' script of a third-party component under ninja. It spews a lot of output, and takes some time to finish. While it is running you can only judge about its own progress, but you don't see ninja's build-status for the total build progress.

But since the configure task runs solo, ninja is not making any further
progress (while waiting for configure to finish), so what output from
ninja are you missing?

On 7 Sep 2013, at 10:19, Maxim Kalaev wrote:

@sorbits (re
https://github.com/martine/ninja/issues/646#issuecomment-23972234):
the reservation is that the current implementation doesn't print ninja
status once command running solo starts spewing its output before it
finishes.
Imagine running a 'configure' script of a third-party component under
ninja. It spews a lot of output, and takes some time to finish. While
it is running you can only judge about its own progress, but you don't
see ninja's build-status for the total build progress.

---

Reply to this email directly or view it on GitHub:
https://github.com/martine/ninja/issues/545#issuecomment-23985051

Well, let's just put it that way: as the author of the patch I am happy this works for you.
As for the appearance - that's subjective. :)

Is there any progress on this issue? I'm just as annoyed by this as Dave. Not seeing the output of the unit tests while they're running really sucks. Especially if one of the unit tests hangs. I work around this issue by going to the build directory and run ctest manually...

Ninja has supports for this since https://github.com/martine/ninja/commit/2832613dc7c1a4a8ff3b9df729954715762a8381
but cmake does not support it yet. You should ask on cmake's mailing list or send a patch.

@nico this issue should be closed.

Uh, good to know. Sorry for the noise then. :)

I have two more use cases for cmake re-running itself:

1) Visual Studio over Ninja: I build two cmake generated outputs, one is the ninja output which performs the build, the other is a Visual studio generated output, with all targets excluded from the solution. I then add one custom target to the Visual Studio output, which re-runs cmake using Ninja. Viola! Visual Studio IDE with Ninja speed!

2) IDL file rule generation: In phase 1, I compile IDL files to determine which outputs they produce, and GLOB that output and write cmake rule files. In phase 2, I perform the actual build, including the produced rule file. ALL depends upon custom target 'go', which reruns cmake --target PHASE1 then cmake --target PHASE2.

Both of these use cases are non-parallel, and buffering the output is not helpful.

As above, if you don't want buffering you should make those tasks use the console pool.