Nunit: Support `dotnet test` in .NET CLI and .NET Core

We were waiting for .NET Core to settle down a bit before we moved forward with #927, that is why it hasn't been updated with a new title and information. I was planning on waiting for any announcements that may come out at //build before moving forward, but we would appreciate any help we can get. My plan is to start providing more complete support for more platforms with the 3.4 release this summer including .NET Core, UWP and Xamarin.

You have provided a well written summary here, so I am going to close #927 and use this issue for tracking going forward. If you want to help with this issue, then we can coordinate, but I would welcome your help.

The first step in this process was creating a PCL driver/agent that can load and execute tests without being tightly bound to a specific version of the NUnit Framework like NUnitLite and the current Xamarin runner. It is still in it's infancy, but the work in progress is in the nunit.portable.agent project.

Any help or advice that @piotrpMSFT or @livarcocc can give would also be appreciated :smile:

@rprouse I have a bug on the CLI to write a better documentation regarding the interactions and requirements between dotnet test and the runner. I will get it done today or tomorrow and I will put a link to it here.

This is the bug tracking it: https://github.com/dotnet/cli/issues/1803.

How is this going? I would love to use this, and the dotnet cli ecosystem seems to be stabilizing.

I am making some progress, but I don't want to do too much until RC2 drops. I am getting sick of trying to keep up with the changes :smile:

.NET Core RC2 has been released and new info is available. Links for reference.

• dotnet-test command
• dotnet-test communication protocol

Can I help any here? This is a blocker to complete some of our ports like StackExchange.Redis.

@NickCraver I could definitely use some help, that first read of the communication protocol didn't make things terribly clear :smile:

First steps are deciding how we are going to approach this. We currently have the portable build that targets .NET Core, but it is limiting. I am considering creating a core build of the framework, but am unsure if I need to or should.

Then I need to look at what it takes to create a test extension for the dotnet command.

Lastly, I would like the nunit engine to be able to launch and communicate with a core nunit-agent so that nunit3-console can run .NET Core unit tests. I've been working on this, but still have a lot of work to do there.

How would you like to help?

@rprouse I think you'll have to have a core build to possibly be used cross-platform, that's a critical feature.

I'm not sure how much time we can put into this vs. investing elsewhere though to be honest, given the amount of work left. For now, I need to get libraries out the door for many downstream people waiting to build their libraries and apps. I hate to say this, but given the lag here vs. what's available today, I think we'll have to finish porting everything over to xUnit, otherwise many hundreds to thousands of developers are waiting on a blocked chain of releases here.

I'll try and re-visit this after we're stable and time allows, but (and correct me if I'm wrong) at the moment we're talking about weeks of work left (going by your post) before I'd be able to use this to run what I hope is finished library ports ready to ship. From an author duty standpoint, I need to unblock people on me first, then help upstream as I can.

@NickCraver rather than spend the effort porting to xUnit, it might be easier to create a command line NUnitLite runner to run your .NET Core tests. It isn't a perfect solution, but possibly the least amount of work. It is a bit out of date, but I blogged about how to do it at Testing .NET Core using NUnit 3.

I am doing it that way for a few personal projects and it is functional.

I'd be happy to help with getting a proper dotnet-test-nunit build out there. I'm sure many folks would benefit. Much of the contract defined in the protocol is already implemented in a helper library. If we have a working xplat nunit runner, this should be little work.

Would it help to break this into two phases - a sort of "quick and dirty, may have grotty hacks, but at least let's get something out there" release and then a more fully-fledged integration later?

(Just being about to build the tests with a TFM of netstandard1.3 would go a long way to unblocking me for Noda Time, for example.)

@jskeet and @NickCraver you can use NUnit today via the NUnitlite "hack" as @rprouse suggested, I've been doing this for Npgsql for a while now and it works well. It's not the perfect solution but it should definitely unblock you guys.

@roji: That worked with dnxcore50 (I've been using it with Noda Time for a while) - but using netstandard1.3, the dependencies on NUnit and NUnitLite are invalid.

@jskeet my test project targets netcoreapp1.0 and depends on Nunit+NUnitLite 3.2.1. I have "imports" : [ "dotnet54" ] and it works like a charm...

@roji: Thanks for that... fingers crossed...

@roji: I'm still having trouble actually running the tests - could you link to where you're doing this in your test project, as an example to follow?

@jskeet sure, here you go: https://github.com/npgsql/npgsql/tree/dev/test/Npgsql.Tests

@jskeet the imports should work for you too. RC2 changes a few things, so I will rewrite my blog post on how to test with NUnit 3 with RC2 in mind and post the code to GitHub. I will add a link here when it is done, hopefully this morning (EST).

@piotrpMSFT can you save me some time and point me to the helper library for creating dotnet-test-nunit? I can search, but I am chasing several things down at the moment so help would be appreciated. I'm sure you are busy too, so if you don't have it at hand, don't spend time on it, I will find it.

I have posted an updated blog post Testing .NET Core RC2 Using NUnit 3 for anyone who is interested.

@jskeet you do need the imports statement in your project.json. It is added by default in the new .NET Core templates, but needs to be added when upgrading.

For project.json in a console runner;

  "frameworks": {
    "netcoreapp1.0": {
      "imports": "dnxcore50"
    }

Or in an assembly;

  "frameworks": {
    "netstandard1.5": {
      "imports": "dnxcore50"
    }

When upgrading to RC2, having old DNX stuff in my path also bit me. Something to check.

@rprouse: Thanks; I'm now able to build, and the tests run on Windows. On Linux I'm seeing an "Object reference not set to an instance of an object." message on dotnet run, which I need to investigate further. Will try your minimal sample to see if that has the same problem...

@jskeet, we are using this approach on both Windows and Linux without issue. See here and here for examples.

@oschwald: Yup, it looks like the problem with the "Object reference not set..." is a very separate issue with dependencies. It's a shame that the error message is so unhelpful (without even a stack trace) - I'm digging into it.

@jskeet There are a few causes, is this yours? https://github.com/dotnet/cli/issues/3075

@NickCraver: Nope - I've done a dotnet restore successfully.

In my case I've got a project.json file that works without a dependency on another project, but fails with it. I need to put some more time into coming up with a complete repro.

@jskeet Gotcha - relevant: https://github.com/dotnet/cli/issues/2469 and the Slack channel is great for project.json live debug, drop in with examples and you can usually get going pretty quickly. For Slack, #general and #dotnet-core channels are pretty active, lots of people going through this and helping the next person.

Hmm... and now it's working. It's possible that it _was_ the same problem @NickCraver reported, and that I was just dotnet restore-ing the wrong project to fix it. Anyway, I'm now running Noda Time tests on Linux, so yay :) Thanks everyone.

@piotrpMSFT re my request for pointers, I have found everything I need and have started work.

I have added an initial PR for the dotnet-test-nunit runner at nunit/dotnet-test-nunit#1

It explores and runs tests, but still needs some work, cleaning up and testing. I will make the NuGet feed for the build artifacts more accessible if anyone wants to play with it.

If anyone has ideas on how I can debug the NuGet package when it gets loaded by Visual Studio, I could use some help. So far, it isn't running in Visual Studio, but that could be a setup problem.

@rprouse: I'd be happy to see how Noda Time copes with it, if that would be useful. Let me know if there's anything in particular you'd like me to check out. Thanks for all your work!

It isn't ready for prime time yet, but I have NUnit .NET Core tests running at the console and in Visual Studio. There are still several issues to resolve before I get an alpha out, but we are now much closer as the hard work is done and it is down to the details.

I will post details on how to test with the CI build after I fix a few issues. I would appreciate it if people could kick the tires and help me find problems.

Here is a screenshot.

@rprouse: Any progress on the CI build? Definitely keen to kick the tires; Noda Time uses a _reasonable_ number of NUnit features, so it should be a good start, anyway. Really looking forward to being able to run tests in VS again. (Hope NCrunch and CodeRush for Roslyn start supporting it once it's out, too...)

@jskeet and others who are interested in kicking the tires and helping me test before I release an alpha, I have updated my demo repository on a dotnet-test-nunit branch to use the new dotnet-test-nunit runner.

Please report issues in the nunit/dotnet-test-nunit repo.

High level instructions are;

Update: dotnet-test-nunit is now available as an alpha on NuGet, select Include prereleases. You no longer need to update your nuget.config file.

dotnet-test-nunit is still under development, so you will need to add a NuGet.Config file to your solution to download NuGet packages from the NUnit CI NuGet feeds.

Your project.json in your test project should look like the following;

project.json

{
    "version": "1.0.0-*",

    "dependencies": {
        "NUnitWithDotNetCoreRC2": "1.0.0-*",
        "NETStandard.Library": "1.5.0-rc2-24027",
        "NUnit": "3.2.1",
        "dotnet-test-nunit": "3.4.0-alpha-1"
    },
    "testRunner": "nunit",

    "frameworks": {
        "netstandard1.5": {
            "imports": [
                "dnxcore50",
                "netcoreapp1.0",
                "portable-net45+win8"
            ]
        }
    },

    "runtimes": {
        "win10-x86": { },
        "win10-x64": { }
    }
}

The lines of interest here are the dependency on dotnet-test-nunit. Feel free to use the newest pre-release version that ends in -CI, that is latest from the master branch. Note that the NUnitWithDotNetCoreRC2 dependency is the project under test.

I have added "testRunner": "nunit" to specify NUnit 3 as the test adapter. I also had to add to the imports for both the test adapter and NUnit to resolve. Lastly, I had to add the runtimes. If anyone can explain why I need to do that, please let me know.

You can now run your tests using the Visual Studio Test Explorer, or by running dotnet test from the command line.

# Restore the NuGet packages
dotnet restore

# Run the unit tests in the current directory
dotnet test

# Run the unit tests in a different directory
dotnet test .\test\NUnitWithDotNetCoreRC2.Test\

Warning

As I said, this is still under development. dotnet-test-nunit version 3.3.0.39-CI listed above has a bug where it will throw an ArgumentException when it tries to save out the TestResult.xml file. That will be fixed shortly. Fixed in 3.3.0.49-CI.

Also note that the dotnet command line swallows blank lines and does not work with color. The NUnit test runner's output is in color, but you won't see it.

The "blank line swallowing" part is a known bug: https://github.com/dotnet/cli/issues/2234

Thanks for the link @jskeet, it looks like color is also a known bug, dotnet/cli#1977

The ArgumentException has been fixed in 3.3.0.49-CI. I will update my instructions above with the fix. One other outstanding issue is that I am not currently supplying the line number for tests to the runner, so clicking on tests in the Visual Studio Test Explorer will not take you to the code.

Am I right in saying there's no command line argument support yet, like --where etc? (I fully understand that it's early days - just trying to check whether this is something I should be able to do.)

I've tried a slightly different project.json to you. I've included it below verbatim, but the important differences are:

• I want to test the desktop runtime as well, so I've got two framework targets
• I'm using netcoreapp1.0 instead of netstandard1.5
• I've included the Microsoft.NETCore.App dependency, with "type"="platform"
• Look ma, no runtime section. (Possibly due to some of the above changes...)

{
  "buildOptions": {
    "keyFile": "../../NodaTime Release.snk",
    "embed": {
      "include":  [   
        "TestData/*"
      ]
    }
  },

  "configurations": {
    "Debug": {
      "buildOptions": {
        "define": [ "DEBUG", "TRACE" ]
      }
    },
    "Release": {
      "buildOptions": {
        "define": [ "RELEASE", "TRACE" ],
        "optimize": true
      }
    }
  },

  "dependencies": {
    "NodaTime": { "target": "project" },
    "NodaTime.Testing": { "target": "project" },
    "NUnit": "3.2.1",
    "dotnet-test-nunit": "3.3.0.49-CI",
    "Microsoft.CSharp": "4.0.1-rc2-24027",
    "System.Dynamic.Runtime": "4.0.11-rc2-24027",
    "System.Reflection.Extensions": "4.0.1-rc2-24027",
    "System.Xml.XDocument": "4.0.11-rc2-24027"
  },

  "testRunner": "nunit",

  "frameworks": {
    "net451": {
      "frameworkAssemblies": {
        "System.Runtime": "",
        "System.Threading.Tasks": "",
        "System.Xml.Linq": ""
      }
    },
    "netcoreapp1.0": {
      "imports" : [ "dnxcore50", "netcoreapp1.0", "portable-net45+win8" ],
      "buildOptions": {
        "define": [ "PCL" ]
      },
      "dependencies": {
        "Microsoft.NETCore.App": { 
          "version": "1.0.0-rc2-3002702",
          "type": "platform"
        },
        "System.Console": "4.0.0-rc2-24027",
      }
    }
  }
}

Result:

Test Count: 15141, Passed: 15141, Failed: 0, Inconclusive: 0, Skipped: 0

Woot. (Using net451 I get a test count of 15646, which is what I'd expect.)

Next up: trying the same thing on a Linux machine (where your project.json presumably wouldn't work without modification as it doesn't mention Linux, but mine should, in theory; having said that, it's an Ubuntu 16.04 machine so its dotnet CLI is somewhat bodged together).

Hmm. Tests on Linux hadn't finished after 14 minutes of eating 100% CPU. Will need to look more closely at what's happening there.

@jskeet

Am I right in saying there's no command line argument support yet, like --where etc?

Poorly tested, but I have wired up --where and most of the common command line parameters. I still have an issue, nunit/dotnet-test-nunit#4 to check that they are all wired up properly and work and add any that may be missing. So far, I haven't tested many, but I have used a few by adding them to the end of the dotnet command. I still need to figure out exactly how it works though. For example, the --debug command line option was working fine from the dotnet-test-nunit solution, but it throws dotnet-test Error: 0 : Microsoft.DotNet.Cli.Utils.CommandUnknownException: No executable found matching command "dotnet-test-" when I run it from my test solution. I also have no idea how to chain the help command line out to ours to show what is supported.

In the meantime, you can take a look at https://github.com/nunit/dotnet-test-nunit/blob/master/src/dotnet-test-nunit/CommandLineOptions.cs to see what command line options I think are supported :smile:

Ah, yes - it looks like dotnet test doesn't have the same facility as dotnet run for separating arguments to the test framework from arguments to dotnet test itself. I'd been trying

dotnet test -- --help

and

dotnet test -- --where=cat!=Slow

Just using

dotnet test --where=cat!=Slow

works fine. Will file a feature request for that - it's nice to be able to be completely unambiguous.

Clicking on tests and navigating to the code is now fixed in 3.3.0.60-CI.

There are still two high priority issues, then I am planning on doing an alpha release at which point I will close this issue and track everything in the new repository. The two issues are;

• [x] nunit/dotnet-test-nunit#22 Add Messages and Stack Traces to Failed Tests
• [x] nunit/dotnet-test-nunit#12 TestContext.WriteLine does not show up in the TestExplorer test summary

Just as an FYI, I ran the Noda Time tests again today on my Ubuntu box, and they completed after about 15 minutes. Looks like they're all running about 5-6x slower on my Linux i5 than on my Win10 i7. Not sure how much that's to do with CPU and how much is to do with JIT though - and it's definitely out of the scope of NUnit :)

@jskeet thanks for the update, that is good news. Did you need mono bindings for the Linux tests as indicated in nunit/dotnet-test-nunit#9?

For anyone who is testing, the latest good NuGet package is 3.3.0.69-CI, please test with that. There are now no outstanding issue that I think are serious enough to block a release. If none are reported this week, I will likely create an alpha release later this week or early next week. Once I do that, I will close this issue and future discussion/issues can go on in the dotnet-test-nunit repo.

Test now or forever hold your peace :smile:

@rprouse: I haven't tried running it on Mono - only .NET Core on Linux. That worked fine with the exact project.json above.

Will give 69-CI a whirl...

One difference I've noticed between this and the console app runner is that the "dotnet test" version seems to include the time spent finding tests in the overall duration, whereas the "dotnet run" command I used to use with NUnitLite didn't. In Noda Time, this means if I run with --where=cat==Foo (there are no such tests in that category), "dotnet test" reports a duration of 14s where as "dotnet run" reports a duration of 0.01s - despite them both taking about the same real elapsed time.

Is this deliberate? I certainly don't mind the behaviour, but It may be worth noting it somewhere to avoid people thinking it's actually slower.

@jskeet that is a good observation, it is indeed the case. Because we don't have the full NUnit engine to run the tests, the code is more simplistic, but I might be able to start the clock on the first test started event rather than wrapping the test run. I will enter an issue.

I have released an alpha version of dotnet-test-nunit on GitHub at https://www.nuget.org/packages/dotnet-test-nunit/3.4.0-alpha-1, so you no longer need to change your NuGet.config file and use CI releases.

Now that I have an alpha out, I am going to close this issue. Please help me test the alpha and report any issues in https://github.com/nunit/dotnet-test-nunit/issues. Final release is planned to go out at the end of the month with the NUnit 3.4 release. I will release updated alphas and betas depending on issues found.

I have updated the documentation in the readme for dotnet-test-nunit. I will keep that up to date.

https://github.com/nunit/dotnet-test-nunit

As I noted before, you don't need to specify the runtimes if you use "type": "platform" for the "Microsoft.NETCore.App" dependency. Given the xUnit docs, I think this is the preferred approach.

I currently have a pull request running through Travis - if that goes green, Noda Time will be depending on this, so that should give it _some_ usage. Nicely done, sir...

One downside I've noticed just now - this means I can no longer easily test with Mono on Linux. I believe that's not an NUnit issue, it's basically a manifestation of https://github.com/dotnet/cli/issues/3073 but for NUnit. I'm still hopeful it might get fixed eventually; for the moment I'm going to disable my mono tests in Travis, with a bit of reluctance. (dotnet test is definitely the future.)