Nunit: Instance-per-test-case feature

Currently, you __can__ share state between parallel tests provided (1) you set the state in the constructor or one-time setup and (2) don't change it in any test. You can even get by in some cases if the state is set to a constant in the test or setup, provided you never set it to any other value. The latter is icky, even though I think we have some NUnit tests that do it.

My thought is that we can't change the default any time soon, if ever. Having mutable state does not currently guarantee a bug. It's only a bug if you actually mutate it and we don't know that unless we do code analysis - which let's not do!

If you don't mutate any instance state within a test or setup or teardown, you won't be broken. If you do, you're already broken so long as your tests can run in parallel with themselves. There's no way you're not broken that I can think of. That's the justification for the Auto option, if we think the resulting complexity for the user is worth the "sensible defaults" outcome.

You have a point but I would continue to advocate that this is a potential extension, which we don't need to decide right now. Let's do this iteratively rather than trying to make all the decisions now.

i'm with @CharliePoole the defaults has to remain as is and can't be removed otherwise it could break people who want that behavior.

maybe a fixture level attribute to trigger instance per thread rather than instance shared for all threads.

@BlythMeister Don't worry! We are not considering any change that would cause existing code to start breaking.

This is interesting. I think instance-per-test-case is more easy to wrap your mind around than instance-per-thread. The ways in which tests are assigned to threads would be unreliable, so I can't think of any benefit that would have over instance-per-test-case.

yeah i'm using the term thread to mean parallel execution....which is factually inaccurate.
my apologies.

In XP terms (and I hope everyone knows that's always my perspective) we have three stories, that build on one another...

1. User may specify instance-per-test-case behavior globally for all fixtures in an assembly.

2. User may specify instance-per-test-case behavior on a fixture.

3. Users may specify instance-per-test-case behavior for fixtures that contain tests which may run in parallel.

I'd like to see (1) completely implemented in this issue. I think we should just put (2) out of our heads - or at least out of the part that writes code - for the moment. I think (3) is a possibility but has some pitfalls we have already encountered in other issues. (A fixture doesn't know if it's going to have test cases run in parallel!) Let's talk about that one JustInTime, which would be after (1) and (2) are complete and merged. FWIW, I think we have a tendency to get tied in knots by trying to think several moves ahead. I mentioned (2) and (3) so that we won't block them in any way, which seems easy enough to avoid.

I think we are all in agreement then. I have a few NUnit priorities before this though, so I'll leave it for others to comment on and possibly pick up.

The beauty of doing this with an attribute rather than a command-line setting is that it will work for any runner.

I agree with @CharliePoole that it should be an attribute. Tests will need to be designed with the instance per test or instance per fixture in mind, so they should always run that way. I also prefer to only do option 1. I think that if you want that style of testing, you should opt-in to it for your entire test suite. Allowing you to opt-in at lower levels introduces complexity that I don't think has a good ROI.

Just curious if there are any imminent plans to implement this feature?

@rprouse This is still marked as requiring design but it looks as if the discussion covered everything already.

@CharliePoole agreed that this can move from design. @pfluegs we haven't started working on this beyond designing it, so there are no plans yet.

Not sure that anybody pays that much attention to it, but the absence of discussion or design labels implies somebody could volunteer. 😄

I've just been badly bitten by this very issue - +1 for fix.

In case we ever want more options, would we want:

```c#
[assembly: FixtureCreation(FixtureCreationOptions.PerTestCase)]

And:
```diff
namespace NUnit.Framework
{
+   public enum FixtureCreationOptions
+   {
+       Singleton, // = default
+       PerTestCase
+   }
}

My own thinking on this has changed a bit, in the context of a new framework at least.

I now think there should be two class-level attributes, one for each behavior. I'd make TestFixture work with an instance per test and Shared Fixture work like NUnit now does. Obvious, that's a breaking change in the context of NUnit, but the general approach might be worth considering.

@CharliePoole [TestFixture(FixtureOptions.PerTestCase)] seems like a nice syntax, but I strongly want an assembly-level attribute as well so I can opt in with every test project. What would we want an assembly-level attribute to look like?

Right, that's why I expressed the reservation WRT what I'd do on a new framework. In the case of NUnit, however, you wouldn't want to change the semantics of TestFixture and I can't think of a really good name for the instance-per-case behavior. If you had two names, all you would have to do to get the desired behavior on all fixtures would be to do a global edit of your source file.

I dislike global settings at the assembly level because they end up confusing some devs, who may not be aware of what is set in the AssemblyInfo file. Currently, we do allow a few, so I suppose that there's no great harm in adding yet another. I have no thoughts about a name for the assembly level attribute except that it should communicate that we are setting an option (default?) for all fixtures in the assembly.

I dislike assembly-level defaults to a certain extent too, for the very reason you mention. Compared to adding a new attribute to each of my currently-attributeless fixture classes, the assembly-level attribute is the route I'd prefer.

@nunit/framework-team What do you think? For the initial introduction of the feature:

• yes/no to assembly-level setting?
• yes/no to fixture-level setting?
• yes/no to method-level setting?
• yes/no to test-case-level setting?

Good point about attributeless fixture classes.

I'd say yes,yes,no,no

I agree with Charlie - with the clarification that ‘fixture-level’ means ‘class-level’. (As opposed to e.g. a test case source method. 😊)

That helps! So the problem with [TestFixture(FixtureCreationOptions.PerTestCase)] is that multiple TestFixture attributes are allowed on the same class and they could then contradict each other. Better and simpler maybe to have a new named attribute applicable to assemblies and classes?

```c#
[assembly: ShareFixtureInstances(false)]

[ShareFixtureInstances(false)]
public class SomeFixture
{
// ...
}

Framework API:

```diff
namespace NUnit.Framework
{
+   [AttributeUsage(AttributeTargets.Assembly | AttributeTargets.Class)]
+   public sealed class ShareFixtureInstancesAttribute : PropertyAttribute
+   {
+       public bool ShareFixtureInstances { get; }

+       public ShareFixtureInstancesAttribute(bool shareFixtureInstances);
+   }
}

Sounds good. What’s the inheritance behaviour of this - if I have the attribute on a base class, with its own tests, and then also inherit from that class? 😊

Presumably, as it’s a ‘class’ attribute - it must be on the actual class the tests in question are being run from?

Also, I’m not totally keen on the name - as a fixture is not always a class in nunit terms, I don’t think (e.g. all the tests in a testcasesource - unless ‘suite’ and ‘fixture’ have a stronger semantic meaning than I think, in Nunit terms!)

How about something along the lines of InstancePerTest...but better named...

Agreed about Assembly and Class level but no deeper. I also prefer something along the lines of @ChrisMaddock's suggestion ClassInstancePerTest(bool). I also assume that the attribute at a class level would override the assembly level and I would expect it to be inherited. I think inheritance is important because you would usually want an instance per test based on the members of a class combined with parallelism. You would therefore want that to be automatically applied to derived classes.

I was once genuinely surprised NUnit does not create an instance per test when a test fixture type is constructible. I always thought of unit tests like, I don't know, unit test: you set up scaffolding, run the test, and the scaffolding's gone. No inadvertent transfer of state between tests.

Having learned that. I, like @jnm2, instantiate tests manually. But when the tests e. g. create and delete files, case runners become truly cumbersome: they must be IDisposable; No magical [TearDown] for ya any more. :( No consistent way to tell test errors from scaffolding errors either. This is a really painful use case.

If I may barge in on the design, I am with @CharliePoole on yes, yes, no, no.

The name PerTestInstance, InstancePerTest or something akin to that sounds much better than ShareFixtureInstances. I mildly dislike the word Class in the name of the attribute, as when I am testing F# code, I do not think of type instances being classes. But sure thing I can live with that! :)

Oh boy, can't wait for this feature!

@kkm000 I agree about the use of "class". NUnit has typically used names that are distinct from the implementation. Even then, people make assumptions, but it's better to be consistent in avoiding such things.

Examples of what I'm talking about: Test, TestFixture, SetUpFixture, Worker (not thread), etc. In principal, a class __could__ represent a single test, not a fixture... I'm not saying we do that or even should, but __we could__.

@kkm000

Thanks for the comment! You're not barging in at all. We have a design goal of being language-agnostic where possible.
Besides the focus on C#, one other thing about the word class: what if one day we support fixtures that aren't classes?

[InstancePerTestCase] would sidestep the issue, but it might lose intuitive clarity.

Today fixtures and classes don't diverge much, but if they did diverge, I'm pretty sure this attribute is concerned with fixtures as opposed to classes.

Suites are a generic organizational concept, while fixtures are things which hold setup/teardown logic and test state and are passed to test methods (currently as instance methods’ this argument). Fixtures do not have to be part of the organizational hierarchy. Let's imagine fixtures diverging from the concept of classes and diverging from the concept of test suites:

```c#
public static class SomeTestSuite // Not a fixture: no setup/teardown logic and no test state
{
[Test] // Single test, not a parameterized suite;
// fixture is injected via a parameter instead of via the implicit this parameter
public static void SomeTest(SomeFixture fixture)
{
fixture.SomeService.SomeProperty = true;
fixture.SomeAction();
fixture.AssertSomething();
}
}

[TestFixture]
public struct SomeFixture : IDisposable
{
public SomeService SomeService { get; } // State

public SomeFixture() // This was almost legal syntax in C# 6, but I'm making a point 😁
{
    // Setup
}

public void Dispose()
{
    // Teardown
}

public void SomeAction()
{
    // Helper method
}

public void AssertSomething()
{
    // Helper method
}

}
```

In the above example, [FixtureInstancePerTestCase] or [SharedFixtureInstances(false)] is not focused on classes and is not focused on hierarchy (suites): it's focused on fixtures, and fixtures are the reusable setup/teardown logic and state. Specifically, it's focused on whether a new SomeFixture is created for each test case or whether the same one is passed to each test.

Since this attribute will control whether different ITests will share ITest.Fixture instances, and the thing being controlled is whatever we decorate with [TestFixture] (another attribute with Fixture in the name), it seems like a _consistent_ option to use the word Fixture.

(I'm bringing points that seem important to discuss, but I'm not super invested in what the name ends up being.)

@jnm2 You're quite right that the concept of a suite and of a fixture are separable. NUnit, however, followed the example of all xunit test frameworks existing at the time it was created in combining the two. It deviated from other frameworks in making the fixture shared.

To my knowledge, xunit.net was the first (only?) framework to make the fixture a separate entity from the test inheritance hierarchy.

When I created parameterized tests, I could have introduced TestCaseSource as a separate "fixture". I didn't do that and I think the changes needed to redo it would be breaking now.

I think that the reason folks would occasionally (before parallelism) ask about instance-per-case behavior is that it's what other frameworks do. For a similar reason they __didn't__ ask for separable fixtures.

So, summing up, here's what I think...

1. We need instance per test case behavior.

2. It can't be the default because of backward compatibility.

3. A separable fixture is a cool idea, but is also a separate issue.

4. Separable fixtures look at least as complex to implement as per instance fixtures. Of course you wouldn't have to deal with backward compatibility in this case, which makes it easier.

Just to confirm about points 3 and 4, I intended my code sample not as a suggestion for new features but as a thought experiment intended to help us land on the right name for this attribute. The thought experiment tells me that we're really focused on the idea of a fixture, not so much a class or a suite, even though they happen to coincide right now.

Whatever form they take, fixtures are reusable definitions of setup/teardown/state. I believe that is the meaning of the word. Fixture instances (occurrences of setup/teardown/state) are what our new attribute decides to share or not share between test cases. I think FixtureInstances somewhere in the name would bring maximum clarity.

Does this seem right to everyone? If not, what other points can we consider?

Ah! Sorry, I thought you had hit on a very good point, usually overlooked, but I was also afraid we might be digressing. :slightly_frowning_face:

I think the full name (whatever you type) should include both Fixture and Instance. How it's done depends on whether you're talking about an attribute name or an enum used in constructing it. Not sure if you have settled on one or the other.

Yes, good point.

If we do an enum, it makes it longer to type. This matters more for class-targeted attributes than assembly-targeted ones. We should also stay away from adding an enum as a parameter to TestFixtureAttribute because multiple TestFixtureAttributes are allowed on the same class and the parameter of one might contradict the parameter of the other. We can handle this via test-time error, but it's usually nice to make invalid states unrepresentable.

Let's get some options up and we can tweak from there.
We're looking for something that balances intuitiveness, consistency with NUnit's style, and the right amount of flexibility. (For the most part, these things should already overlap. 😄)

Proposal A

```c#
[assembly: FixtureOptions(InstancePerTestCase = true)]

[FixtureOptions(InstancePerTestCase = false)]
class SomeFixture
{
}

Pro: extending later if there's ever more options for fixture handling (doubtful?)
Pro: two names so there's less stuffed into a single name
Con: `[FixtureOptions]` would be legal but highly misunderstandable syntax. (Does it reset the assembly-wide setting to the default, `false`, or does it have no effect, e.g. `null`?)


### Proposal B

```c#
[assembly: FixtureOptions(FixtureCreationOptions.PerTestCase)]

[FixtureOptions(FixtureCreationOptions.Singleton)]
class SomeFixture
{
}

public enum FixtureCreationOptions
{
    Singleton, // = default
    PerTestCase,
    // Doubtful there will be other options?
    // Maybe Pool, where we reset and reuse between nonconcurrent tests?    
}

Pro: invalid states are unrepresentable
Pro: extending later if there are ever more options for fixture handling (doubtful?)
Con: redundancy in typing both the attribute name and the enum name

Proposal C

```c#
[assembly: FixtureInstancePerTestCase]

[FixtureInstancePerTestCase(false)]
class SomeFixture
{
}
```

Pro: invalid states are unrepresentable
Pro: hard to misunderstand (but I have a blind spot here because I'm not a new user)
Con: long name
Maybe con: Very specific name that can specify both the option and the value. (But we also have [NonParallelizable] which I like.)

(Alternate proposals welcome!)

There's something to be said for having two attributes: one for the assembly and one for the class itself.

Pro: Easier to implement two simple attributes rather than one attribute with different behaviors in context. (Lesson learned from ParallelizableAttribute)
Pro: Likely to be easier for users to understand, since each can be named appropriately. For example, assembly level can include the word "Default".
Con: Two attributes to remember.

Approach using each of your alternatives:

```C#
[assembly: DefaultFixtureOptions(InstancePerTestCase = true)]

[FixtureOptions(InstancePerTestCase = false)]
class SomeFixture
{
}

[assembly: DefaultFixtureOptions(FixtureCreationOptions.PerTestCase)]

[FixtureOptions(FixtureCreationOptions.Singleton)]
class SomeFixture
{
}

[assembly: DefaultFixtureInstancePerTestCase]

[FixtureInstancePerTestCase(false)]
class SomeFixture
{
}
```

Other comments:

• I like A and B better than C
• You shouldn't call shared fixture behavior "Singleton", 'cause it isn't one. :smile: If you inherit, you'll get multiple instances of the base fixture.
• If you use separate attributes at assembly and fixture level, you can actually drop fixture from the name at the fixture level.
• We have not specified how SetUpFixtures fit into this at all. They are problematic and may need a run-time error if the attribute is used on them. In fact, one advantage of an entirely separate attribute is that we don't have to deal with this. (It's also an advantage to a property of TestFixture, BTW)
• We need to warn people that they instance-per-testcase doesn't apply to static classesd. It's obvious to all of us but there are folks who will not understand it immediately.

BTW, I've often wished we had gone with DefaultTestCaseTimeOut for the assembly and fixture levels and kept Timeout to methods only.

Good, thanks for bringing up this possibility.

Something bugging me right now is that this could theoretically make sense, even though we aren't going this direction:

c# [DefaultFixtureOptions(InstancePerTestCase = true)] class SomeFixture { [FixtureOptions(InstancePerTestCase = false)] public void SomeTest() { } }

Because the level on which this actually plays out is not the fixture level itself but rather the test case level.
As an example: a derived fixture may override the base fixture's setting, but all it's really doing is affecting the setting for each test case produced by the derived class.

Or:

```c#
[assembly: DefaultFixtureOptions(InstancePerTestCase = true)]

// Why shouldn't this be considered a default, too?
[DefaultFixtureOptions(InstancePerTestCase = true)]
public abstract class BaseFixture
{
}

public sealed class SomeFixture : BaseFixture
{
}
```

How do we come up with a rationale for when and when not to use Default?

I think implementing this attribute on each test case would introduce a great deal of complexity with only small benefit.

For "normal" test fixtures, we will continue to instantiate the instance as part of the command chain associated with the test fixture. For instance-per-test fixtures, we will not instantiate it there but will have to implement a new command as part of the test chain, which will do pretty much the same thing as what is now done in the fixture chain.

If a fixture may contain both types of test, then we will have to execute it twice, once for each model.

The benefit seems small, since the user can easily achieve the same thing by splitting tests into separate fixtures. Fixtures, after all, are an NUnit artifact and it's reasonable to require users to put tests that need the same fixture behavior into the same fixture.

Regarding when to use "Default"... it seems to me that you would __not__ use it on any TestFixxture and you would use it only on the assembly level. Base classes are not an exception because the attribute is actually found on the derived class by virtue of inheritance. So setting the particular behavior on the base is not a default. Rather it's the setting for that class and any that derive from it. Contradictory specifications should either cause an error or be ignored, as we do in all cases of inherited attributes currently. That is, placing attribute A on the base class and B on the derived class is not distinguished from placing A and B directly on the base class. We designed it that way because it would be surprising to the user to do anything else.

WRT setup fixtures, btw, I think it should generate a runtime error to specify either instantiation option there. SetUpFixtures are not TestFixtures.

That is, placing attribute A on the base class and B on the derived class is not distinguished from placing A and B directly on the base class. We designed it that way because it would be surprising to the user to do anything else.

Oh my, I would have expected [Apartment] or [Parallelizable] to replace the base class' or base method's setting!

This all makes sense to me.

Can we rule out wanting an enum such as FixtureCreationOptions? PerTestCase, PerFixture, hypothetical ReusedPerTestCase?

I'm leaning towards the simplicity of:

```c#
[assembly: DefaultFixtureOptions(InstancePerTestCase = true)]

[FixtureOptions(InstancePerTestCase = false)]
class SomeFixture
{
}

Hypothetical other creation options would then be added this way:
```c#
[FixtureOptions(InstancePerTestCase = false, ReuseInstances = true)]

@nunit/framework-team What are your preferences?

Regarding

[assembly: DefaultFixtureOptions(InstancePerTestCase = true)]

Maybe just me, but generally I understand attributes as indicating that the marked object possesses a certain trait, not otherwise present; arguments to the attribute, if any, are refining said trait. Take, for example, Serializable--the object is serializable, but if undecorated, it is not; InternalsVisibleTo--internals visible to something specific, and the arguments refine this, otherwise internal are not visible to anything, etc. Some attributes are not following the pattern, but these are usually low level, e, g, CompilationRepresentation, MethodImpl. These usually come with required constructor arguments.

Now consider
c# [assembly: DefaultFixtureOptions]
This is a well-formed C#, since the assignment to InstancePerTestCase is optional. What does it express semantically?

To me, [assembly: DefaultFixtureInstancePerTestCase] seems more natural (with an exception for its sesquipedalian name, but that's only an opinion, not justifiable save aesthetically). One attribute, one trait. Perhaps this one does not need any parameters or optionally settable properties.

Also, as a user, I find it surprising that there are two attributes with different names, one applicable only at the assembly level (DefaultSomething) and another only at a type level (just Something). Naturally, I do not understand what was the "learned lesson" that @CharliePoole mentioned regarding the Parallelizable attribute, but from this side of the isle, a pair of attributes to learn where one would do (in this user's superficial thinking) seems perplexing.

@kkm000 I agree with your comment about an attribute that expresses a single trait.

WRT two different attributes for the "same" trait at different levels, I __would__ agree if I thought they were actually the same. But I don't think they are. I'll use Timeout for an example.

If Timeout is placed on a test method it means "This test method will time out if it takes more than the number of specified milliseconds to run." However, when Timeout is placed on a fixture or assembly it has a different meaning: "The default timeout for all test cases contained in this item is at the specified number of milliseconds." That default may, of course, be overridden at lower levels. Two rather different meanings, which by the way have two separate implementations. The first applies the timeout to the test on which it appears. The second stores the default in the test context where it will be found and used as a default when contained test cases are encountered. ParallelizableAttribute works similarly.

In this particular case, placed on a fixture, the attribute means "Instantiate this fixture once per test case." At the assembly level it means "Instantiate Test Fixtures in this assembly once per fixture unless otherwise specified."

In this particular case, placed on a fixture, the attribute means "Instantiate this fixture once per test case." At the assembly level it means "Instantiate Test Fixtures in this assembly once per fixture unless otherwise specified."

Assuming you misspoke in the last sentence--was _"Instantiate Test Fixtures in this assembly once per ~fixture~ test case unless otherwise specified."_ intended? Or am I seriously misunderstanding the scope and intent?

Assuming I am not, to me, as a user, the semantics does not look different at all: “instantiate a class per test within the scope marked by the attribute,” maybe clarified with “unless its inner scope is marked differently, if there is indeed such an inner scope,” be it a fixture or the assembly. Just my opinion, but two attributes do look a bit confusing. Maybe it's just my ways of thinking, I dunno really.

two separate implementations

In the gedankenworld of unicorns and rainbows implementation does not affect the user-facing side. But we (both you the NUnit designer and I one of its users) are all engineers, and engineering is all about compromises. If using one attribute for different scopes would in fact lead to internal klugity and maintenance burden, I wouldn't gritch at all. :)

In the end, to me this is a minor issue, and I do not feel strongly for the same attribute and can live with the attributes arranged either way.

@kkm000 Yes, I intended "once per test case."

I agree that it all makes perfect sense if you think of it as a series of nested scopes. However, experience shows that users either (a) don't always think that way or (b) don't get the definition of scope right.

As an example of the last, consider nested classes and base classes. Neither of them is treated by NUnit as nested "scopes" for test purposes but they may seem that way to users.

I'll grant that the issue I mentioned with Timeout / DefaultTimeout is a bit more serious, since Timeout on a fixture might easily mean that the timeout applied to the whole fixture, which of course it doesn't. For that matter, Timeout on the assembly might apply to the whole assembly, which it doesn't.

In this case, there seems to be less ambiguity, because we are using "per test case" in the name so I could easily go either way. I'm not working on it anyway. :smile:

Come to think of it, if Timeout had been called TestCaseTimeout that would have been better!

I'm happy with a single [FixtureInstancePerTestCase]. Seems the least mental burden on users.

@nunit/framework-team Can you help us settle on a direction, skimming from https://github.com/nunit/nunit/issues/2574#issuecomment-426347735 down?

[FixtureInstancePerTestCase] is also my least-disliked option. 😄 The lack of extensibility is the only downside.

I don't like the [FixtureOptions] problem - I think that's too confusing. I'd personally prefer a single attribute for both assembly and class - I appreciate that's caused us issues before for Timeout and Paralellizable - however when this one has "Fixture" in the name, it seems less ambiguous to me.

I personally quote liked the enum option...except I can't think how to name it appropriately to make it more succinct!

Is there any update on this?

Any updates will be posted to this issue.

Someone can resolve this issue please??
Currently we must wrap all our test code like this:
csharp [Test] public void TestExample(){ using(var tc = new MyTestContext()){ //code goes here } }
Using this feature, we can have the test context initialized on set up and disposed on tear down and still being thread safe.

Hi, we are running into this issue at our company as well. We are migrating from a framework that supports this feature (Py.Test), so this is a bit of a gotcha. We would appreciate it greatly if you could show us the source code that controls this instantiation so we can fork it and make the necessary changes.

@MChiciak
If this issue won't be fixed soon, we are considering migration to xUnit, since in xUnit this is the default behaviour: "xUnit.net creates a new instance of the test class for every test that is run" https://xunit.github.io/docs/shared-context

@LirazShay, @MChiciak I ran into this issue as well but agree with @CharliePoole's points. So I made my test-specific fields that are initialized at SetUp [ThreadStatic] and it works fine.

@dfal
Thanks for the nice workaround!

I would love this. I currently have an existing test suite with thousands of tests running on nunit, that can't be parallelized easily because of shared state. If this option were provided, we'd be able to parallelize our test suite much more easily.

As a thought experiment, imagine we simply changed how TestFixture worked. What would this break?

1. Any tests that use ordering and rely on state changes made by one test to influence other tests. This is a very bad practice in the case of (supposedly) independent unit tests. We've always warned people not to do this and I wouldn't mind breaking any tests (with a message, of course) that relied on having a single instance in this way.

2. Depending on what is done within the one-time setup, it's possible that the individual fixture instances would not be able to run in parallel. This is ironic, since the motivation for the change is to allow the fixtures to run in parallel! This is not a breaking change for people who need the feature, since they are already unable to run in parallel, but it is a significant breaking change for those who already rely on running the tests in parallel __after__ a non-parallelizable `OneTimeSetUp'.

3. Any fixture-level OneTimeSetUp would now be run multiple times. This would eliminate any efficiency gain from the one-time setup and might, in some cases, change the semantics. In particular, if the initialization affected global state in some way (e.g. setting up a database) it might lead to errors. This one seems like an unacceptable breakage to me.

I found this thought-experiment useful in thinking about how to implement per-test-case instantiation of fixtures. It clearly needs to be a new non-default feature if introduced in NUnit 3. For NUnit 4 or any other new framework (e.g. TestCentric) the new behavior could easily be the default.

as a note, I very simply replaced some MSTest's TestClass and TestMethod with TestFixture and Test respectively, and discovered that NUnit and MSTest have different semantics here. MSTest creates new instances for each test where, as I'm sure everyone reading this is aware, NUnit re-uses the TestFixture instances.

It would be nice if i could write TestFixture(lifeCycle = OneTestPerInstance)] or similar, to ease this migration.

_edit:_ Actually, having just ported the tests to use a Setup method, it actually turns out the existing code initializer/before/teardown code was fairly tortured under MSTest, and is simple property-initializer constructors now, which is much nicer and more completed, so I'm glad I did the translation. Still, as a matter of compatibility, it would be nice to have.

@CharliePoole

Any tests that use ordering and rely on state changes made by one test to influence other tests.

Aah, yes, break, break these tests, pretty please!!! 🤣 That's the worst abuse of a unit testing framework I can think of.

Any fixture-level OneTimeSetUp

Uh, I have always thought this is run once per process... I always wondered why is not it static.

But yeah, performance is also a very valid point.

This feature has been in discussions for over 2 years now, I don't mean to sound offensive, but are there any concrete and realistic plans to actually implement it in the nearest future?

I would like to know, if my team should plan for migration to another framework or will this be resolved in the coming months, because this issue is a showstopper for us.

This is a major change to how tests run and would need to consider the full impact so as to not completely break those who are dependant on the current functionality.

The best way to "resolve" this is to make your test class thread safe. Just like any good multi-threaded application, thread safety should be paramount.
If you are using for example fields that you setup in a setup method and then used in many test methods, your class is not thread-safe. that's not a flaw of NUnit. You have written a non thread safe singleton.

By createing a subclass inside your TestFixture which holds you contextual information which you create as part of each test you are then making sure that your singleton TestFixture is threadsafe...thus...you "resolve" the issue.
So rather than spending time "migrating to another framework" might i suggest that you spent that time making your singleton TestFixtures threadsafe....you will probably find that to be a much quicker exercise.

For example, when i first came across the fact that an NUnit TestFixture is a singleton i converted my 100+ test fixtures with over 500 tests to be threadsafe (and had to move from RhinoMocks to Moq as RhinoMocks is also not threadsafe) and this took me around 4 weeks (to be honest, Rhino -> Moq took most of that time!)

You're 100% right in principle, but when you have an existing test suite of ~7000+ unit tests, just rewriting them to be thread safe is a huge investment.

Not sure if it's more of an investment than moving to another framework (there may be some tool to automate that) but I don't think this feature request should be discarded on principle, IMO

I'm not advocating that this feature should not be looked at.

But such a big change to the way nunit works, in my opinion should be considered for either backwards compatibility (which may be quite tough) or new major version (also not a small task)

I'm suggesting that making the code a user of nunit writes threadsafe to run in a multi thread environment is an easier form of action. You could possibly even script that.

I am not a contributor to this repo, so I don't have any context about how it's implemented. This may be incredibly naive because of that, but wouldn't it be possible to implement this as an optional switch? So that this breaks nothing?

This whole issue is discussing various ways that an optional switch could be implemented...there are many ways

This is an open source project staffed entirely by volunteers. Once a feature is accepted, as this has been, it waits for someone on the team or outside to pick it up and do it. There is no "boss" to assign work on it to anyone. It's actually a good thing that nobody has done that if they don't have the time to devote to getting it done. That way it's available for whomever is capable and interested in doing it.

PRs are always welcome but this is probably not a first-time contribution for someone, since it has the potential to break many other things in NUnit. My opinion anyway.

I'm no longer contributing so actively as in the past so my comments as to how I think it should be done were given as advice for whomever picks it up.

I do agree that it has been around for a while and I'd love to see it done myself.

@BlythMeister I think a switch oversimplifies it a bit - as you probably know but others may not.

Sure, it's a switch, but it's not just turning on a light bulb. It's more like switching your radio from AM to FM... entirely new circuits come into play.

In this case, what is involved is an entire new life cycle in the execution of a test, so a fair amount of code has to be replaced. Not a small thing. The risk will be in making sure that the new "circuitry" doesn't impinge on the old in any way.

Sorry your right, my explanation was vague in they why a "switch" is not simple!

@BlythMeister I actually did exactly what you describe - created a disposable object to track state during a test and then dispose of it after the test is complete - to drive my Selenium test harness. One major problem with that approach is that the state does not persist into teardown. As a result it's not possible to execute actions conditionally for failed tests (grab screenshots, logs or other pieces of state) because the test state won't resolve inside the test method itself and can only be evaluated during teardown. So while a state object is a powerful way to make tests thread-safe, it only works when you don't need to do things with the test result during teardown.

It's useful to keep I mind that doing things with the test result isn't what TearDown us designed for. Of course, I know many people use it that way and I understand the motivation but there are lots of tricky corner cases that essentially come down to the fact that TearDown is part of the test and it isn't over till its over.

With NUnit 3 we hoped to see more users examining results outside the framework using engine extensions but working within the test itself seems to be more familiar and accessible.

@CharliePoole

1. Any fixture-level OneTimeSetUp would now be run multiple times. This would eliminate any efficiency gain from the one-time setup and might, in some cases, change the semantics. In particular, if the initialization affected global state in some way (e.g. setting up a database) it might lead to errors. This one seems like an unacceptable breakage to me.

Why we can't implement Instance-per-test-case feature and still run the OneTimeSetUp only once (keep the main thread remember to call it or something)?

Because it would break any existing OneTimeSetUp that initialized the instance, which is very common.

I see this as a problem in any switch-based solution. It's less of a problem if we define an entire new alternative to test fixture, an approach I find quite tempting.

Because it would break any existing OneTimeSetUp that initialized the instance, which is very common.

How about only allowing static OneTimeSetUp and OneTimeTearDown methods when the "instance per test case" feature is enabled? That would force members initialized within those methods to be static, thus obviously being shared among fixture instances.

While the main motivation of this Github issue is parallelization, I'd like to throw in _principle of least surprise_ and _avoiding errors caused by tests influencing each other_ as motivations as well. (Apologies if I missed that someone else has already mentioned this.)

In case you didn't read back far enough in this rather old thread, my comments don't argue against the feature, but against a particular implementation that was suggested. I prefer replacing TestFixture with an entirely new type of fixture, which works differently.

I think I the principal of least surprise works well here. It's unsurprising if a new fixture works differently.

If, on the other hand, we create a flag that controls the existing fixture, I can think of five or six different places in the current implementation where we would need to branch on that flag.

That's not to say that the two fixture types might not share certain code. But that would be outside of the view of users.

@fschmied BTW, if we were doing a new framework (or maybe even NUnit 4) then I'd argue differently... I think we should have made it much harder for tests to interfere with one another in NUnit 2 and 3. However, so long as we are making changes in NUnit 3, I think compatibility rules.

I prefer replacing TestFixture with an entirely new type of fixture, which works differently.
[...]
I think I the principal of least surprise works well here. It's unsurprising if a new fixture works differently.

You're right of course. What I meant is that for the original kind of test fixture, the behavior will still be surprising to most people. (It happened to surprise a co-worker two weeks ago, when he found a case of a test influencing another one. He has been using NUnit since 2013, which was why I looked up this issue.)

But of course, it's a tradeoff between backwards compatibility, implementation complexity, and the wish to correct an original sin [1].

Back to the issue of OneTimeSetUp: I think it's still important to have a OneTimeSetUp per test fixture, even if the test fixture class is instantiated per test. To avoid confusion in case the OneTimeSetUp manipulates instance state, I'd propose enforcing OneTimeSetUp (and OneTimeTearDown) to be static with the instance-per-test feature. (No matter if it's implemented as a switch or as a new kind of test fixture.)

[1] James Newkirk wrote this:

I think one of the biggest screw-ups that was made when we wrote NUnit V2.0 was to not create a new instance of the test fixture class for each contained test method. I say "we" but I think this one was my fault. I did not quite understand the reasoning in JUnit for creating a new instance of the test fixture for each test method.

He also writes this, though:

[...] it would be difficult to change the way that NUnit works now, too many people would complain [...]

:)

Yes, Jim and I have argued about this one for years. :smile: One thing we agree on is the last sentence. Jim went off to do a new framework before implementing his preferred approach. I'm basically doing the same and SetUp/TearDown is one thing I'm likely to do differently.

I keep being surprised at how many fairly experienced NUnit users are not aware of the use of the same instance for all test cases. It may be that it was more widely known when NUnit first came out because it was an obvious difference from JUnit. Now that's all forgotten and new people come to NUnit with expectations that NUnit doesn't match. Something to think about.

so.. is this up for grabs? or still in the design phase?

right now in .NET land its either XUnit without TestContext or NUnit, with boilerplate parallel tests

The feature is accepted and assigned a normal priority. Nobody is assigned to it and it's not listed on a milestone. That means somebody has to decide they want to work on it. That could be a committer or a contributor.

We don't usually do a design phase - if we want one, the design label is applied - but even so, whoever does this would be well advised to post a comment describing how they plan to implement it, particularly how it will look to users. If you were to start out by coding it, you might end up in design discussions anyway, so it's probably best to get that over with first.

So, would you like to work on it?

sure, i just started work at
https://github.com/avilv/nunit/tree/instance-per-test

im new to this codebase so im trying to stick to the original code style as much as i can
so far i introduced an InstancePerTestCase attribute and plan on having it supported on assembly and class level

let me know if im way off base here

I added comments on some implementation details on your last commit.

As far as the design, I think an attribute derived from IApplyToContext makes sense. I think I would prefer something more general than just InstancePerTestCase, maybe

C# [FixtureLifeCycle(LifeCycle.InstancePerTestCase]

That would allow you to set it at the assembly level and reset it on individual fixtures.

CharliePoole you are too much , thanks a lot for the feedback and help

i think i'm getting how things are built / what does what etc. slowly but surely

here is my proposal for the LifeCycle enum (ive also updated my branch with your suggestion)

    /// <summary>
    /// Specifies the lifecycle for a fixture.
    /// </summary>
    public enum LifeCycle
    {
        /// <summary>
        /// A single instance is created and for all test cases.
        /// </summary>
        SingleInstance,

        /// <summary>
        /// A new instance is created for each test case for fixtures that are marked with <see cref="ParallelizableAttribute"/>.
        /// </summary>
        InstancePerTestCaseForParallelFixtures,

        /// <summary>
        /// A new instance is created for each test case.
        /// </summary>
        InstancePerTestCase
    }

@jnm2 What do you think of this approach - you're kind of the "sponsor" of this feature in the first place. I'm a bit uncertain about InstancePerTestCaseForParallelFixtures, which might be a bit too much for an initial implementation. As usual, once implemented, we're stuck with it.

If we __do__ accept the tie to parallelism, it has to be context-based, not attribute-based. That is, any test may be run in parallel, so it's more like `InstancePerTestCaseWhenCasesAreParallel. Parallel fixtures don't particularly require this feature. And maybe it's better left to the user to decide.

I also agree that it should not be coupled in any way with parallelism. parallelism was the reason to add this feature but once we agree that this feature is useful, in my opinion, it should not be reflected in the code

i agree, i understand the use case for using it only in parallel scenarios since thats where NUnit's current default model will bite you, but hopefully this will be the default life cycle in the future (nunit 4?) since i think it really should have been this way from the start

perhaps we should leave it at SingleInsance/InstancePerTestCase

another question is should we handle the IDisposable pattern here, that would pretty much make SetUp/TearDown redundant but would feel much more natural ala XUnit style.

We already dispose any Test Fixture that implements IDisposable. You just need to make sure that continues to work.

great, will do.

ill probably start adding test cases today, but since its such a core mechanism, it might add quite bit of cases which just make sure the same thing works when InstancePerTestCase is on, correct me if im wrong

The DisposeFixtureCommand takes care of calling Dispose.

IMO your main problem is ensuring that the entire command stream for the test fixture is invoked __for each test__. The changes I see so far correctly create a command stream that seems to do the job, but the key question is how to ensure that it is invoked for each test case. (Note that the command structure for fixtures is distinct from that for test cases.) I think that compositeworkitem has to be sensitive to the context and invoke the command correctly (as you have done) but I'm not sure it will get a chance to do so for __every__ test case. (I'm only reading the code, so I could be wrong.)

@CharliePoole looks like you are right, i did a little graphing over what exactly executes a test method:

currently i create the test object instance in the RunChildren loop, but that wont be enough for Retry/Repeat commands, not to mention if there are other wrappers i missed / someone needs new ones.
its going to have to be much closer to TestMethodCommand

i updated the latest changes on
https://github.com/avilv/nunit/tree/instance-per-test-2

removed InstancePerTestCaseWhenCasesAreParallel
implemented FixtureLifeCycle via adding Construct/Dispose fixture inside MakeTestCommand in SimpleWorkItem, and skipping it in CompositeWorkItem MakeOneTimeSetUpCommand

it seems to work well, also makes more sense with how the framework is built
now i just need to add a bunch of unit tests to make sure its all working correctly

any chance this would get reviewed before next version ? im still up for improvements/changes of course

This approach sounds good to me and I'd love to use it in the next release if we can do that!

any chance we could get this reviewed? feels like were getting really close

Curious if there has been any movement on this feature? Seems like @avilv has gotten it pretty close. Have been anticipating a feature like this for a long time, and excited to use it!

Thanks all!

Is this feature working now ?

@CharliePoole could you please review @avilv latest version? It seems to be fully implemented. I think a lot of people are waiting for this feature.

@janissimsons Sorry, but I'm not on this project any longer so my review wouldn't move it forward. @nunit-framework-team Can't someone review this?

Appreciate all the work that was done to get #3427 merged in. @rprouse any idea when we can expect this in a release?

How do we use the new feature? Could someone write a short description about the new attributes and how to properly use them?

I am planning on releasing in the next couple of weeks. I wanted to wait until. NET 5 is final and do a bit of final testing with it before release. I doubt there will be any problems, but it is so close that I thought it would be better to wait.

@rprouse Any estimates when you're going to release the new version? I'd like to start using this feature.

@janissimsons very soon. We're working out some issues with our .NET 5.0 builds and a couple of other issues, then I will release. You can track the status on this project board, https://github.com/nunit/nunit/projects/4

This is wonderful news!

Do I understand correctly that this will make the following possible in NUnit?

Combine all of these:

• Tests are parameterized
• Run tests in parallel.
• Be able to run parameterized variations of the same test in parallel.
• Tests can be data driven

Tried out some things in the past here https://github.com/jawn/dotnet-parallel-parameterized-tests

Will the new release still be compatible with .NET Full Framework 4.7.2 or at least 4.8?

Best regards,
D.R.

@jawn I believe all that will work, yes. I've tested all of those things, but NUnit has many features so there may be edge cases with some combinations of features. Feel free to pull our nightly NuGet package and give it a test! We'd love more testing and feedback before we release.

@drauch yes, the 3.13 release will still support back to .NET 3.5.

@avilv Great work!!
by the way using which tool did you created that nice flow diagram ?
Thanks

@rprouse Could you also please update documentation on how to use this new feature? Thanks!

@janissimsons I am planning on updating the docs for the release. Short version though is to add [FixtureLifeCycle(LifeCycle.InstancePerTestCase)] to your test class, probably along with [Parallelizable(ParallelScope.All)] and an instance of your test class will get instantiated for each test that is run within the class. Doing that allows you to use member variables in the class in parallel test runs without worrying about other tests changing the variables.

@LirazShay The diagram in the screenshot was created in Visual Studio Enterprise. You can right-click a selection in Solution Explorer and choose "Show on Code Map," for example. https://docs.microsoft.com/en-us/visualstudio/modeling/map-dependencies-across-your-solutions ReSharper has a similar feature.

@jnm2 Thanks a lot!!!
I have VS enterprise and didn't know about this feature that can be very helpful
Thank you so much for the tip

Am I correct that this has been released and is part of the 3.13 release?

Am I correct that this has been released and is part of the 3.13 release?

I believe there has been a couple of problems observed and fixed, especially regarding assembly-level attribute application in #3720 and we're waiting for 3.13.1 now..

NUnit 3.13.1 has been released.
Any more issues remaining?

No that's why it was closed @andrewlaser