Testng: @Test(enabled=false) annotation at class level should disable all the methods in the class

Hi Nicolas,

You should see this if you don't use any @Test method on your methods, but as soon as you do (like you do in your example above with dataProvider) then the enabled attribute gets overridden, and its default is true, hence the behavior you're seeing.

Does this make sense?

Yes, Thanks Cedric.

I make the wrong assumption that the annotation @test(enabled=false) on class exclude entirely the class from execution

I agree it's a bit counter intuitive, but it's too late to change now, sadly...

Hi, Cedric!
What about adding a new parameter to @Test annotation and call it something like "enabledClass" and make it true by default? It should be used only on a class-level (enabledClass = false) to disable the whole class regardless of the method-level annotations and be ignored, if added for the method.
In this way we would preserve backward compatibility and provide a pretty useful functionality at the same time.
If you'd like, I can implement it by myself and provide a pull request.
Thanks in advance!

@andronix83,

I'm afraid this makes things more confusing now that you need to explain the subtle difference between enabled and enabledClass.

The current enabled behavior is not the most intuitive but it doesn't seem to be an issue in general, if I judge by the number of times this issue has been brought up (hardly ever).

Hi Cédric,

I've actually run into this issue as recently as last week. I am working on a web server that relies on the Facebook Graph API. I had a test class that had some test methods relying on code hitting the Graph API, with other test methods not needing to rely on that service at all. Then, there was a global outage on Facebook as a whole for more than an hour. This resulted in some of the tests in this class failing.

What I was hoping to do, in order to keep the rest of my development team unblocked, is just disable all of the tests on this class immediately, via a @Test(enabled=false) annotation on the class level. Of course, that didn't work. Instead, I had to disable each failing test method one-by-one, taking me more time than desired, or even necessary to solve this problem.

Ideally, TestNG supports some sort of functionality similar to JUnit's @Ignore annotation: http://junit.sourceforge.net/javadoc/org/junit/Ignore.html. Long story short, developers are indeed desiring this functionality.

Thank you.

@ecbrodie This is already supported (try it!) but there is a caveat: every individual method that has a @Test annotation will re-enable the test.

@Test(enabled = false)
public class T {
  void f() {} // will not run
}

@Test(enabled = false)
public class T {
  @Test
  void f() {} // WILL run!
}

This is due to the semantics that attributes at the method level override attributes defined at the class level, so the method level @Test annotation is basically saying @Test(enabled = true) since it's the default value...

Makes sense?

Hi @cbeust , thanks for that example, but that was already the understanding of the problem, so I think you may have misunderstood me. That point that I was trying to make was, given the current semantics of @Test(enabled=true/false) and the desire to have a way to indeed specify an override of enabled on the class level to ignore whatever is set at the method level, perhaps it is time to reconsider your stance against adding in some sort of enabledClass annotation value.

You seem to indicate some disdain towards the original approach you took with @Test(enabled) semantics. If that is indeed the case, then why not bring its semantics to something that you feel is more intuitive?

Agreed, I think a @Test(enabledClass = false) would make sense and it's the only way to implement your suggestion without breaking backward compatibility. Should be pretty easy too.

Maybe you or @juherr would be interested in submitting a PR?

I'm not feeling it good. I don't like the idea to add a new attribute which only be used on class.
Then, I think enable=false is a bad practice when used in source because you'll have to modify sources when you want to run the tests.
IMO, if you want to unselect a class, testng.xml should be used instead.
@ecbrodie How do you run your tests?

BTW, what I propose instead is to provide a IAnnotationTransformer implementation which will override the default behavior of enable=false on class.
IAnnotationTransformer has just to enable=false every tests if their class has enable=false.
It won't break the backward compatibility and won't add a specific parameter.
@cbeust What do you think?

@juherr Cool PR, it's indeed very simple to do with an IAnnotationTransformer.

My only concern is that for the user, it's a bit more arcane to use, as opposed to an enabledClass attribute.

Note that we already have a few attributes that are only applicable at the class level (suiteName, testName).

As I said, I don't like the use case. And I suppose it will be enough for the only 2 persons who asked it in the 4 last years :)

@juherr Fair enough.

@ecbrodie What do you think of the PR?

https://github.com/cbeust/testng/pull/816

@cbeust
Just curious. What if we provided a built in AnnotationTransformer which can do this ?

Yes, it is what #816 proposes

@juherr
Thanks for sharing that PR information. Any reason why this is waiting to be merged ?

What If I have a scenario as below:

@Test(groups = { "regression", "smoke" }, dependsOnGroups = { "Creation" })
public class EditName {

    @Test(dataProvider="SomeTestData",dataProviderClass=SearchData.class)
    public void TC_1(String Msg) throws Exception{
        System.out.println(Msg);
    }

    @Test(dataProvider="SomeTestData",dataProviderClass=SearchData.class, dependsOnMethods = { "TC_1" }))
    public void TC_2(String Msg) throws Exception{
        System.out.println(Msg);
    }
}

1. Will TC_1 and TC_2 belong to both, regression and smoke groups ?
2. Will both TC_1 and TC_2 depend on group "Creation"?
3. Will TC_2 be dependent on group "Creation" as well as TC_1, or just TC_1?

@Rameshwar-Juptimath What is the relation between you sample and this issue?

Due to these non intuitive behavior of @Test at class and method level.
What should be recommend as a best practice?
Should we avoid using both in tests?
Sine we are more or less using it at method level to have different dataprovider for each case, we should avoid having it on the class?
Thoughts?

@aliciatang The behavior is the same for all the annotation attributes: values on the method will override the values on the class.

Since 6.13 you can use @Ignore which the behavior you expect. Documentation for @Ignore can be found here