Nunit: Empty result from `TestCaseSource` generator should fail the test
When using [TestCaseSource(nameof(GeneratorMethod))], it would make sense to fail the test (or mark it as skipped) if the generator produces no inputs. At least don't show the test as passing :/
fluffynuts
All 3 comments
If memory serves, it works as it does due to a bug report some years back, which convinced us it should pass in this case.
CharliePoole
on 3 Jun 2020
😢
I just wrote a test with (currently) one item in the source, and, being a new test in a TDD cycle, I expected it to fail, only to find it passed... so I had to track down my faulty generator.
Perhaps failure is a bit to draconian (my preference, but may be annoying to the reporter of the prior issue). How about marking the test as skipped?
fluffynuts
on 3 Jun 2020
Some points to consider...
There can be multiple sources, so we need to be clear about whether the discussion applies to __each__ source or to the aggregate result of all sources.
Sources generate tests, not just data. If no tests are generated, then there is nothing to fail or skip. In the distant past (V2) we generated a fake failing test, that ended up being both messy in the code and confusing to the users.
A test method with no cases is a bit like a method with no code. Also like a fixture with no tests. We consider those passing.
If you call your test a theory, you will get a failure if there is no passing case.
I'd vote for leaving the current behavior but giving the user some way to ask for a warning message if no cases are generated.
CharliePoole
on 3 Jun 2020
Related issues
Thaina
·
5Comments
xplicit
·
5Comments
ChrisMaddock
·
3Comments
ffMathy
·
3Comments
Prodigio
·
3Comments
Most helpful comment
Some points to consider...
There can be multiple sources, so we need to be clear about whether the discussion applies to __each__ source or to the aggregate result of all sources.
Sources generate tests, not just data. If no tests are generated, then there is nothing to fail or skip. In the distant past (V2) we generated a fake failing test, that ended up being both messy in the code and confusing to the users.
A test method with no cases is a bit like a method with no code. Also like a fixture with no tests. We consider those passing.
If you call your test a theory, you will get a failure if there is no passing case.
I'd vote for leaving the current behavior but giving the user some way to ask for a warning message if no cases are generated.