Testng: @Test methods from super class are re-executed for every sub class

@vlad230

P.S. Editing code in github issues is incredibly bad...please move to something else.

You might want to spend sometime getting yourself familiarized with Markdown. That would make it very easy to add code snippets along with formatting.

Am not quite sure I understand your issue here. Ultimately the child classes would get the base class methods as well. And if they are annotated with TestNG annotations they are going to get executed for every child class. That is the behavior of TestNG. So I would say that TestNG is working as designed.

@juherr - Your thoughts ?

@krmahadevan thanks for the formatting :)

Am not quite sure I understand your issue here. Ultimately the child classes would get the base class methods as well. And if they are annotated with TestNG annotations they are going to get executed for every child class. That is the behavior of TestNG. So I would say that TestNG is working as designed.

If you ask me, it wouldn't make any sense to execute the tests in the super class multiple times.
If this is the expected behaviour of TestNG right now, maybe we could add this as in improvement.

Working as designed.
Why don't you move baseMethod test in its own class?

@vlad230

If you ask me, it wouldn't make any sense to execute the tests in the super class multiple times.

What exactly are you trying to achieve? It would be great if you could please help elaborate or explain your scenario.

If this is the expected behaviour of TestNG right now, maybe we could add this as in improvement.

Am not sure that would be taken up as an enhancement because its counter intuitive . If you basically want the base class method to be executed exactly once, no matter how many child classes are present, then you could do it today in your test project by doing the following

1. Have your base class implement org.testng.IHookable
2. Within its run() method, add an edit check that would check if the base class method is already executed and run it if not executed.

Here's a sample

Marker interface that indicates that a base class method should run only once

import static java.lang.annotation.ElementType.METHOD;
import static java.lang.annotation.ElementType.TYPE;

import java.lang.annotation.Retention;
import java.lang.annotation.Target;

@Retention(java.lang.annotation.RetentionPolicy.RUNTIME)
@Target({METHOD, TYPE})
public @interface RunOnce {
}

Here's the state tracker singleton

import java.lang.reflect.Method;
import java.util.Map;
import java.util.concurrent.ConcurrentHashMap;

public class StateTracker {

  private static final StateTracker instance = new StateTracker();

  private final Map<String, Object> runOnceData = new ConcurrentHashMap<>();

  private StateTracker() {}

  public static StateTracker getInstance() {
    return instance;
  }

  boolean canExecute(Method method) {
    RunOnce runOnce = method.getAnnotation(RunOnce.class);
    String methodName = method.getName();
    boolean execute = true;
    if (runOnce != null) {
      if (runOnceData.containsKey(methodName)) {
        execute = false;
      } else {
        runOnceData.put(methodName, new Object());
      }
    }
    return execute;
  }
}

Here's how the base class would look like

import org.testng.IHookCallBack;
import org.testng.IHookable;
import org.testng.ITestResult;
import org.testng.annotations.Test;

public class BaseTest implements IHookable {

  @Test
  @RunOnce
  public void baseclassTestMethod() {
    System.err.println("Base class test method executed");
  }

  @Override
  public void run(IHookCallBack callBack, ITestResult testResult) {
    boolean execute =
        StateTracker.getInstance()
            .canExecute(testResult.getMethod().getConstructorOrMethod().getMethod());
    if (execute) {
      callBack.runTestMethod(testResult);
    } else {
      testResult.setStatus(ITestResult.SKIP);
      testResult.setThrowable(new RuntimeException("Intentionally skipping"));
    }
  }
}

Here's how the child classes would look like

import org.testng.annotations.Test;

public class ChildClass1 extends BaseTest {

  @Test
  public void childTest() {
    System.err.println(getClass().getName() + ".childTest() executed");
  }
}

import org.testng.annotations.Test;

public class ChildClass2 extends BaseTest {

  @Test
  public void childTest() {
    System.err.println(getClass().getName() + ".childTest() executed");
  }
}

You would need to add some filtering mechanism that prunes the test results and removes skipped tests which are annotated using @RunOnce

@krmahadevan
What exactly are you trying to achieve? It would be great if you could please help elaborate or explain your scenario.

I'm trying to separate the tests in multiple classes (that are related to a feature). For example you have a "base class" that has the setup part for that feature with @Before & @After methods along side other helper methods (not tests) that support the tests from the base class along side the tests in the "child classes". So, this way you would just add new tests to your child classes and not duplicate the code in the child classes.
The base class would have some test methods that test more general parts of that feature and the child classes will have more specific tests.
Since I'm trying to optimize the test run time by running classes in parallel, I had to split my original "base class" which had 50+ test methods into multiple smaller "child classes" such that these smaller classes are distributed to other threads/executors.

@krmahadevan thanks a lot for the detailed solution but it looks a bit cumbersome to me. Since TestNG can be customized, I would rather see this as the default behaviour.
I don't see a need of constantly re-executing the tests from the base class. If it's a matter of setup/teardown one could always use @Before/@After that would be re-executed every time before a child class from the base class.

Yes, I think the behavior could be configurable.

In fact, I don't find a use case where we want to run the parent test for each child instance.
Unless you have a direct dependency to test (dependsOnMethods) because the test state is often modified like that in integration tests.

@krmahadevan WDYT?

@vlad230

thanks a lot for the detailed solution but it looks a bit cumbersome to me

Your use case also kind of looks a bit different from the regular. Moreover I have given you a working example of how it can be done. All you need to do is, just adopt this and build upon this.

Since TestNG can be customized, I would rather see this as the default behaviour.

No. That cannot be done in my opinion. Tests are meant to be executed all the time. Just because they reside in a base class doesn't mean that they have to be selectively executed. TestNG can be customised. But that doesn't mean TestNG needs to support all customisations. It merely needs to provide for ways in which a user should be able to customise it to their needs and work with it. The sample I shared is an example of applying those customisations without TestNG having to undergo changes.

I don't see a need of constantly re-executing the tests from the base class. If it's a matter of setup/teardown one could always use @Before/@after that would be re-executed every time before a child class from the base class.

To me that is just one way of looking at things. If a particular method shouldn't be executed every time, then you should not be using the @Test annotation. You could very well use the configuration annotations such as @BeforeTest (which gets executed only once per <test> tag) or @BeforeSuite (which gets executed only once per <suite> tag) no? Wouldn't they work for you.

I am not sure, if your test methods in your base class are really test methods. To me they sound more like configurations which establish the setup conditions to be satisfied for the test to run through.

@juherr - I am not quite sure I agree with the use case here. A test irrespective of where it is found is meant to be executed. If there is scenario wherein we don't want it to be executed for all child classes, then we should be using the configuration annotations and not the @Test annotation.

On the implementation side, I think this will add a lot of chaos to the codebase especially when people start resorting to using groups as a means for execution.

Moreover, I feel that there is already a way of doing this currently within TestNG (which I have shared as an example). Why not just leverage that? This seems to be a one off use case, and doesn't seem to fit into the regular way of how TestNG is used.

@krmahadevan I'm sorry, but I think you didn't understand what I'm trying to do.
In my parent class (BaseTest - e.g. DashboardTest) I have 15 tests now and I have 3 child classes (e.g. DashboardFilterTest, DashboardTreeTest, DashboardEditorTest) that each have tests inside. Using the approach I described in my post, each time I run the tests, those 15 tests get executed 3 times over (so, 3x15 = 45 tests being executed) bringing no value and extending the run time uselessly, as opposed to just executing the 15 tests once.

I still think it doesn't make sense for tests that are located in a parent class to be re-executed each time tests form a child class are executed. Would prefer the default behaviour of TestNG to execute test methods from a parent class once, irrespective of the number of child classes that are extending it.

If this cannot be done, I understand. Thanks anyway.

@vlad230

Thanks for adding additional context.

In my parent class (BaseTest - e.g. DashboardTest) I have 15 tests now and I have 3 child classes (e.g. DashboardFilterTest, DashboardTreeTest, DashboardEditorTest) that each have tests inside. Using the approach I described in my post, each time I run the tests, those 15 tests get executed 3 times over (so, 3x15 = 45 tests being executed) bringing no value and extending the run time uselessly, as opposed to just executing the 15 tests once.

In that case, why would DashboardFilterTest, DashboardTreeTest, DashboardEditorTest be extending DashboardTest ? It looks like DashboardTest tests need to reside in their own class instead of you leveraging an inheritance approach here no? The base class should be refactored to remove out all the common test methods and be housed in a separate test class, and DashboardTest should be refactored to just house common non @Test methods no ?

If this cannot be done, I understand. Thanks anyway.

It's not about if they can be done or not (the implementation challenges is a separate topic), but I am still having a hard time trying to understand the validity of the use case itself.

@krmahadevan

In that case, why would DashboardFilterTest, DashboardTreeTest, DashboardEditorTest be extending DashboardTest ? It looks like DashboardTest tests need to reside in their own class instead of you leveraging an inheritance approach here no? The base class should be refactored to remove out all the common test methods and be housed in a separate test class, and DashboardTest should be refactored to just house common non @Test methods no ?

'DashboardTest' contains helper methods (e.g. openDashboard(), createDashboard() etc.) and setup/teardown/cleanup (@Before/@After) methods that are needed for the child classes as well. So, I don't need to replicate the code in the child classes.
Yes, some of the tests in 'DashboardTest' could be moved to some other specific classes but I wanted to keep them here since they are more general (e.g. checking the overall look of the dashboard, basic CRUD tests or even some tests that didn't fit in a specific category or are a mix).

Sure, I could have a non-@Test test class (although it would be weird and someone could add tests here by mistake) but it would be just to avoid this issue, right?

@vlad230

'DashboardTest' contains helper methods (e.g. openDashboard(), createDashboard() etc.) and setup/teardown/cleanup (@Before/@after) methods that are needed for the child classes as well. So, I don't need to replicate the code in the child classes.

This makes sense. and yes they can very well reside within the base class so that the configuration methods are made available to the child classes as well.

Yes, some of the tests in 'DashboardTest' could be moved to some other specific classes but I wanted to keep them here since they are more general (e.g. checking the overall look of the dashboard, basic CRUD tests or even some tests that didn't fit in a specific category or are a mix).

You could still create a class called GenericDashboardTest extends DashboardTest which houses all the generic @Test methods in it.

Sure, I could have a non-@test test class (although it would be weird and someone could add tests here by mistake) but it would be just to avoid this issue, right?

Well, isn't that what code reviews are there for :) To prevent such slippages. The base class (from what you explained) doesn't need to contain the common generic @Test methods. It can very well house just the helpers and the configurations and you can have one more test class which extends the base class and houses the generic test methods. That should resolve the confusion at hand.

Closing this issue with resolution as _working as designed_