Nunit: Catching AssertionException fails tests since 3.10.

Your test is testing two propositions about how NUnit works:

1. That all assertion failures throw an AssertionException.
2. That catching the exception prevents the error from being reported.

As it happens, the first statement is mostly true but doesn't hold within multiple assert blocks or when issuing warnings. The second is not true at all. By the time you catch the exception, the failure has already been recorded. This is a change in the internal behavior of NUnit, only observable if you are catching these exceptions. I, for one, have been telling people not to rely on that behavior for years, since it's merely an accident of implementation.

I realize that your sample code is only intended to demonstrate the problem you see. Can you provide an example of what you are actually trying to do in catching the exception? I'm pretty sure we can help you to find a better solution.

Thanks for getting back to me on such a short notice. I use the mechanism when things are still flaky. So sometimes the software doesn't work properly. In these cases i report this to the devs. In the meantime we extended the test to workaround this flaky behaviour. So the test passes instead instead of failing on this issue.

We tried using the Warn.If() method for this, however this sets the result to "Test Skipped" in the test explorer. This causes reporting failures as the test is not skipped at all. So we choose to implement the assertion catch instead and trace a warning. I know, not desirable but it worked pretty well. At least till last update.

I am curious whether this change was intentional, in that case I know i have to change things.

Warn.If works fine ... __except__ under the Test Explorer. 😢 The problem is that Test Explorer knows about pass and fail, but not warning. We had to translate our warning results to something they can understand until such time as they implement a warning result status. FWIW we have a similar problem with our Inconclusive result state.

The change we made was intentional and has been in planning for a long time. We don't want to rely on exceptions so much of the time and we want to report the outcome of asserts - including passed ones eventually - in some way that doesn't stop the test. We have made some changes to mitigate the impact of this for users like you, but this one didn't seem to have any mitigation possible.

If you have flaky tests, the standard for many years has been to Ignore them. Unfortunately, many teams "ignore the ignored tests," in which case it's not such a good choice. If you can train your team to take warnings seriously - almost as seriously, in fact, as errors and failures - then you can use warning status to highlight flakiness. Another possibility is to use Ignore with the Until property, so it changes into a failure if not fixed by a certain date.

BTW, when I coach teams, I usually have a continually updated Big Visible Display of ignored tests. On most teams, everyone knows who is responsible for what, so there isn't any need to actually call out the person responsible for the flaky test. They tend to get fixed when everyone sees the problem.

@svanimpelen have you tried using the Retry attribute for your flaky tests? It will allow you to run them several times to get them to pass.

As for warn resulting in skipped tests in the explorer, we've talked to the visual studio team about allowing us to specify more result states than the limited number from MSTest that they currently support. They like the idea, so hopefully we will see improvements there in the future. Until then you'll need to suffer with skipped. That said, skipped results in warnings. I would assume you wanted warnings to be very obvious 😄

So many of our methods document that the API throws AssertionException on failure:

https://github.com/nunit/nunit/blob/d03e93c8f25170a8ff48e80863a3fe6fd294613a/src/NUnitFramework/framework/Assert.That.cs#L40-L43

Doesn't this documentation, plus the longtime behavior of NUnit, constitute a breaking change in 3.10?
Unless I'm missing something, we should at least remove the now-incorrect docs.

I agree that the documentation can be misleading now that we've introduced multiple assertion blocks. That said, the AssertionException is still being thrown in this case, the difference is that the test is now being reported as a failure in Visual Studio even though the exception was caught. I don't see that as a breaking change. Users were relying on undocumented behavior that happened to work. To me, that is no different that users relying on the fact that tests happened to be run in alphabetical order to order their tests.

We should probably update the documentation. We could remove the information about throwing an exception and instead just state that a false condition fails the test. And/Or, we could state that the exception is not thrown in multi-assert blocks? Personally, I think the type of exception that is thrown is an internal detail, so I prefer the first option.

I think that's right . If the use of exceptions is to be documented anywhere, it could be in the wiki section on internals.

When method documentation says an exception will be thrown, I think it's unequivocally saying the exception can be caught at the call site. C# doesn't have checked exceptions and XML docs are how we've been communicating that part of the method contract. Since as far as I'm aware the language "throws X" has always meant the method does not catch it, I am still a bit worried that's how others have taken it. Maybe a breaking change note in the wiki to be on the safe side?

I like the idea of removing all references to AssertionException from the XML docs and of creating a wiki page warning people not to use AssertionException.

💭If catching AssertionException is always a bug, can we consider making that type internal?

@jnm2.

I agree with you on the importance of the XML docs but fact is that those docs are __not__ how we have been communicating part of the contract up to now. I actually think you may be the first person to suggest it and I think it's a good idea.

But I think you have to recognize that starting to think of them that way will involve changing a lot of erroneous comments! I do not think we can treat each of these doc changes as a breaking change. On a case by case basis, we can reasonably treat any underlying behavior change as breaking, but that's different. The docs changes should not be breaking.

This brings up the question of what should be in the list of breaking changes. If I have understood your past comments, I think you tend to want to document anything that might break any user. I prefer to document things we believe would break a lot of users.

Here's how I see the difference - others may disagree... Documenting __everything that might break__ smells like CYA to me, like the kind of CYA that management sometimes indulges in when they focus on who gets blamed for failure rather than preventing the failure itself. IF we document everything, then we can say... "See, we're covered." If we only document what we see as important, then we run the risk of being wrong and having to apologize but we give the average user a much easier to understand set of changes. I prefer the latter.

There are times when it's safe to catch an assertion exception. For example, I made a change a while back so that it would remain safe when running tests directly, without a TestExecutionContext involved. If you make the exception internal, then you break that. Actually, I was happy to break it, but the team agreed we should not so I fixed it. You just have to decide what you are willing to break. That fix did __not__ handle the current issue, which deals with catching the exception inside the test itself, rather than in the code that invokes the method.

Catching AssertionException in a test is what I would call a classic "smell." It's probably wrong, but you have to look at the specific case to know. I would tell users "Catching AssertionException is most likely an error, unless you have detailed knowledge of NUnit internals. To ensure proper behavior, you should not try to change the outcome of the test and should rethrow the exception after doing whatever you want to do with it." However, I still don't think such info belongs in the XML docs or the general user docs. I'd put it as a page in the internals section.

BTW, my past experience has been that documenting something and then telling people not to use it leads to increased usage. 😄

Good points, thanks! I think I am thinking partly in terms of CYA. There's another element too, which is that as a consumer of libraries, I personally would appreciate being told every breaking change so I can look for bad assumptions in my own code. Particularly with changes like this which the compiler doesn't notice. Since I would appreciate that, I project that onto our users to some extent.

Should we start a new issue to track the docs fix?

Why new?

The title looks like something we'd close as an answered question rather than fixing. Would we typically change the title and use this issue to track the doc change?

What I've typically done is review the issue and classify it... in this case as a documentation issue. Admittedly, we have spent a real long time discussing it as a possible bug first. It's a project administration question, really, and outside of my scope.

What I'm saying, is do something logical and consistent. 😄 Closing this as not a bug is an option. That separates the complaint from the solution, however. Writing an explanatory comment as you reclassify it as a docs bug and possibly edit the title keeps the solution with the complaint. In any case, the issue title of Done issues ends up in the release notes, so it should be expressive of what happened.

Can you provide an example of what you are actually trying to do in catching the exception? I'm pretty sure we can help you to find a better solution.

I'm currently facing same problem since my tests started to fail after upgrading to 3.10. Now here's my problem:
I have custom test assertion methods. Those internally calls NUnit.Assert. I test these assertion methods to ensure they actually fail when I expect them to fail.
Currently my tests catches the AssertionException to verify the custom assertion method will cause a test to fail, but after upgrading all those tests starts to fail due to above reasons.

What is the recommended way of testing those with 3.10?

Thanks.

The simplest approach is to use Assert.Throws to catch the exception rather than doing it yourself. Assert.Throws understands NUnit internals and ensures that the failure is not reported.

I saw this in my notifications the other day and didn't think much of it. However, updating a library project of mine, which does "Assert.That" within it's logic (it's a testing helper framework -- specifically for providing an easy way to test EF persistence when coding EF contexts and entities by hand and coding the corresponding database migrations by hand (or with something like FluentMigrator) -- so, in other words, it's supposed to perform assertions on behalf of the caller), I now get tests failing where they shouldn't be -- the one test in particular is asserting that a certain condition should throw test failures -- which it does, but since the "catch" no longer works as it always has, these tests, whilst actually passing, are reported as failing.

So I guess I can't throw proper NUnit assertions, or use Assert.That in library code intended to assist with quicker test development by performing tedious tasks for the user?

I'd agree with prior statements that if the exception can't be consumed by the user, it shouldn't be available for the user to throw; however that doesn't solve the issue where I have Assert.That code in place for the consumer and tests to prove that they do throw fail because the code does what it's supposed to: fail with nunit assertions.

Perhaps we should productize some of the result APIs?

@fluffynuts If I understand you correctly, the catch clause is in your test code, not the code of your user library. (If it's also in the user library, we can talk about that, but the issues are slightly different.)

Of course, NUnit has tests of itself, so we had exactly the same potential problem when this change was made. Naturally, we modified our own tests so they continued to pass. You can do the same thing.

The first line of defense is Assert.Throws. In most cases of testing failure behavior, you can replace try / catch(AssertionException) with Assert.Throws<AssertionException> directly. Internally Assert.Throws creates a throwaway isolated TestExecutionContext so your actual test results are not polluted by the error that you are testing.

In a few cases, you would might need to create your own isolated context. You can see how to do that by examining the code for Assert.Throws.

Much of NUnit's own failure testing, however, actually uses a different approach. We have a separate assembly with tests that are run by our own test programs. We capture the result and can then examine it. This is more work but avoids putting too many implementation details about NUnit into our tests. Most of the methods that do this are found in our test utilities assembly. I think that's what @jnm2 is referring to.

Post any questions you have or contact me offline for further help with this.

Thanks for the info. Yes, the assertions are caught within test code. Assert.Throws would normally do the trick, but the specific test in question runs the library code with "too low" a delta allowed for testing a datetime value that should be put into and they retrieved from a database. Most of the time, the drift experienced for this into localdb is 1ms, but sometimes it drifts higher - and this test just aims to assert that the "sometimes" happens. So, at the moment, I run the code in 4 parallel threads, 10 times and expect at least one failure. Assert.Throws is, unfortunately, too deterministic for this case, so I either need to stop using nunit exceptions (or Assert, within the library code), or I need to learn the fu that you guys use. I'd appreciate any pointers - can happen offline, if you like.

Heh, I just realized that the message pump test failures for my AsyncVoidVerificationScope which I've been staring at are precisely the same issue as @fluffynuts is having. Actually due to the NUnit 3.10 upgrade, not my [assembly: RestoreSynchronizationContext] ITestAction. Serves me right for ignoring them while doing multiple things at once. :D

Anyway, looks like what I want to do is wrap my delegate invocation in using (new TestExecutionContext.IsolatedContext()) just like Assert.Throws does.

Assert.Throws and Assert.ThrowsAsync do not establish isolated contexts before invoking async delegates though. Why is this?

@fluffynuts A long time ago, when I first wrote Assert.Catch it merely caught any exception and didn't generate an error if no exception was thrown. Unfortunately, that changed and we no longer have a "neutral" equivalent of Assert.Throws. You could write one, however. Here's an untested version based on the code in Assert.Throws. It only handles synchronous code but you could extend it easily for async using addional code from Assert.Throws.

```C#
public static void Exception SafelyCatchAnNUnitException(TestDelegate code)
{
Exception caughtException = null;

using (new TestExecutionContext.IsolatedContext())
{
    try
    {
        code();
    }
    catch (Exception ex)
    {
        caughtException = ex;
    }

    return caughtException;
}

}
```

The method will return any exception that is thrown, cleaning up any remnants of the error left behind by NUnit's assert methods. If no exception is thrown, it returns null.

It's also possible to use IsolatedContext() at a higher level in your code so as to retrieve the actual test result from it. That would allow you to test entirely at the level of test outcome, without any reference to exceptions. I've given some thought to writing a library for this type of testing, which I think would be superior to the way NUnit currently does its own tests.