Testng: @BeforeClass and @AfterClass in superclass ignore groups when running parallel in classes

Actually, the behavior above is on 6.14.3. On 7.0.0 it's even weirder. I even added timestamps to be sure.

Output on 7.0.0:

Ran before class A on 15 thread, 23:43:10.723
Ran before class A on 14 thread, 23:43:10.723
Ran before class B on 15 thread, 23:43:10.830
Ran before class B on 14 thread, 23:43:10.830
Ran after class B on 15 thread, 23:43:10.952
Ran after class A on 14 thread, 23:43:10.952
Ran before class A on 14 thread, 23:43:11.054
Ran before class B on 15 thread, 23:43:11.054
Test in class A with group A ran on 14 thread, 23:43:11.265
Test in class B with group B ran on 15 thread, 23:43:11.265
Ran before class A on 14 thread, 23:43:11.465
Ran after class B on 15 thread, 23:43:11.465
Ran before class B on 15 thread, 23:43:11.569
Ran after class A on 14 thread, 23:43:11.569
Ran after class A on 14 thread, 23:43:11.677
Ran after class A on 15 thread, 23:43:11.677
Ran after class B on 14 thread, 23:43:11.777
Ran after class B on 15 thread, 23:43:11.778

@MindaugasMateika - The problem is in your suite xml file. You are not running by groups.For you to be able to run by groups you should be using the <groups> tag in your suite file and select the groups in question. Without that, group execution will not be triggered (irrespective of whether your tests/configurations belong to a group or not)

Closing this issue.
Please try that and comment if the issue still continues to exist.

Heya @krmahadevan , I've added groups to xml but I'm experiencing the same behavior.

testng.xml

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd" >
<suite name="All tests" verbose="1" parallel="classes" thread-count="2" >
    <test name="Tests">
        <groups>
            <run>
                <include name="a" />
                <include name="b" />
            </run>
        </groups>
        <classes>
            <class name="test.ClassA" />
            <class name="test.ClassB" />
        </classes>
    </test>
</suite>

Ran before class A on 15 thread, 10:51:48.794
Ran before class A on 14 thread, 10:51:48.794
Ran before class B on 14 thread, 10:51:48.902
Ran before class B on 15 thread, 10:51:48.902
Ran after class A on 14 thread, 10:51:49.014
Ran after class B on 15 thread, 10:51:49.017
Ran before class A on 14 thread, 10:51:49.116
Ran before class B on 15 thread, 10:51:49.117
Test in class A with group A ran on 14 thread, 10:51:49.444
Test in class B with group B ran on 15 thread, 10:51:49.444
Ran after class B on 15 thread, 10:51:49.624
Ran before class A on 14 thread, 10:51:49.624
Ran after class A on 14 thread, 10:51:49.725
Ran before class B on 15 thread, 10:51:49.726
Ran after class A on 14 thread, 10:51:49.828
Ran after class A on 15 thread, 10:51:49.828
Ran after class B on 14 thread, 10:51:49.931
Ran after class B on 15 thread, 10:51:49.931



md5-37e721bbba1e94aae099dbc7071b38c3



<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd" >
<suite name="All tests" verbose="1" parallel="classes" thread-count="2" >
    <test name="Tests">
        <groups>
            <run>
                <include name="a" />
            </run>
        </groups>
        <classes>
            <class name="test.ClassA" />
        </classes>
    </test>
    <test name="Tests B">
        <groups>
            <run>
                <include name="b" />
            </run>
        </groups>
        <classes>
            <class name="test.ClassB" />
        </classes>
    </test>
</suite>



md5-38a48c328d261b22db36e9db68bb81ec



Ran before class A on 14 thread, 10:55:06.758
Ran after class A on 14 thread, 10:55:06.875
Ran before class A on 14 thread, 10:55:06.975
Test in class A with group A ran on 14 thread, 10:55:07.278
Ran before class A on 14 thread, 10:55:07.522
Ran after class A on 14 thread, 10:55:07.625
Ran after class A on 14 thread, 10:55:07.728
Ran before class B on 15 thread, 10:55:07.840
Ran after class B on 15 thread, 10:55:07.941
Ran before class B on 15 thread, 10:55:08.044
Test in class B with group B ran on 15 thread, 10:55:08.151
Ran after class B on 15 thread, 10:55:08.254
Ran before class B on 15 thread, 10:55:08.356
Ran after class B on 15 thread, 10:55:08.458

@MindaugasMateika - There are a couple of things here..

To get your issue resolved, please remove class level annotation and instead use test method level annotation when specifying groups.. That should get your issue resolved.

With respect to the timestamp.. The only way to ascertain this would be to remove all of the other methods, just have two test methods and try it once again. TestNG internally uses ExecutorService to manage concurrency. AFAIK submitting a task to ExecutorService doesn't mean that it would get executed immediately. So there's not much that can be done there. But the tests are running in parallel for sure else you would see the same thread ids.

Now for the weird behavior..
In your case, you are including two groups a and b and you are running two test classes. So for each of the test class, TestNG is going to execute both the @BeforeClass methods in the base class, because the unit of selection is "groups". TestNG uses groups ONLY to find out "what to run". Once it has determined that, after that, groups wouldn't have any relevance. Since your base class has "two" configuration methods at @BeforeClass level, for every test class, both of them are getting executed. I dont see that as a problem per se.

Also remember, parallel execution is applicable to Test methods and NOT CONFIGURATION methods.

All said and done, the only thing pending that needs to be fixed here, is that when groups are included at the class level and test method level, and when group selection is employed to run tests, something doesn't work.

_Note to myself_ : Could be related to something around https://github.com/cbeust/testng/pull/1992

Sample to reproduce

import java.lang.reflect.Method;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;
import org.testng.ITestResult;
import org.testng.Reporter;
import org.testng.annotations.AfterClass;
import org.testng.annotations.BeforeClass;
import org.testng.annotations.Test;

public class BaseTest {
  public static List<String> logs = new ArrayList<>();

  @BeforeClass(groups = {"a"})
  public void beforeClassA() {
    printer();
  }

  @BeforeClass(groups = {"b"})
  public void beforeClassB() {
    printer();
  }

  @AfterClass(groups = {"a"})
  public void afterClassA() {
    printer();
  }

  @AfterClass(groups = {"b"})
  public void afterClassB() {
    printer();
  }

  void printer() {
    ITestResult result = Reporter.getCurrentTestResult();
    String className = "";
    String methodName = result.getMethod().getMethodName();
    Method m = result.getMethod().getConstructorOrMethod().getMethod();
    String annotation = "";
    String groups = "";
    BeforeClass beforeClass = m.getAnnotation(BeforeClass.class);
    if (beforeClass != null) {
      annotation = "@BeforeClass";
      groups = Arrays.toString(beforeClass.groups());
    }
    AfterClass afterClass = m.getAnnotation(AfterClass.class);
    if (afterClass != null) {
      annotation = "@AfterClass";
      groups = Arrays.toString(afterClass.groups());
    }
    Test test = m.getAnnotation(Test.class);
    if (test != null) {
      annotation = "@Test";
      groups = Arrays.toString(test.groups());
    }
    String tid = Long.toString(Thread.currentThread().getId());
    String instanceId = result.getInstance().toString();
    String msg =
        String.format(
            "Running %s annotated %s.%s() [ belongs to groups %s] on Thread [%s] aind instanceId = %s",
            annotation, className, methodName, groups, tid, instanceId);
    System.err.println(msg);
    log(msg);
  }

  private static synchronized void log(String msg) {
    logs.add(msg);
  }
}

import org.testng.annotations.Test;

@Test(groups = "a")
public class ClassA extends BaseTest {
  @Test
//  @Test(groups = "a")
  public void someTest() {
    printer();
  }
}

import org.testng.annotations.Test;

@Test(groups = "b")
public class ClassB extends BaseTest {
  @Test
//  @Test(groups = "b")
  public void someTest() {
    printer();
  }
}

import java.util.Arrays;
import java.util.Collections;
import java.util.List;
import org.testng.Assert;
import org.testng.TestNG;
import org.testng.xml.XmlClass;
import org.testng.xml.XmlGroups;
import org.testng.xml.XmlRun;
import org.testng.xml.XmlSuite;
import org.testng.xml.XmlSuite.ParallelMode;
import org.testng.xml.XmlTest;

public class TestRunner {
  public static void main(String[] args) {
    XmlSuite xmlSuite = new XmlSuite();
    xmlSuite.setName("test_suite");
    XmlTest xmlTest = new XmlTest(xmlSuite);
    xmlTest.setName("sample_test");
    XmlGroups xmlGroups = new XmlGroups();
    XmlRun xmlRun = new XmlRun();
    xmlRun.onInclude("a");
    xmlRun.onInclude("b");
    xmlGroups.setRun(xmlRun);
    xmlTest.setGroups(xmlGroups);
    List<XmlClass> xmlClasses =
        Arrays.asList(new XmlClass(ClassA.class), new XmlClass(ClassB.class));
    xmlTest.setXmlClasses(xmlClasses);
    xmlSuite.setParallel(ParallelMode.CLASSES);
    xmlSuite.setThreadCount(2);
    xmlSuite.setVerbose(2);
    TestNG testng = new TestNG();
    testng.setXmlSuites(Collections.singletonList(xmlSuite));
    testng.run();
    System.err.println(BaseTest.logs.size());
    Assert.assertEquals(BaseTest.logs.size(), 10);
  }
}

Since your base class has "two" configuration methods at @BeforeClass level, for every test class, both of them are getting executed. I dont see that as a problem per se.

Two configuration methods but with different groups. I thought that was one of groups purposes - to be able to have different setup and tear down methods for different test classes with those groups. :/

@MindaugasMateika

Two configuration methods but with different groups. I thought that was one of groups purposes - to be able to have different setup and tear down methods for different test classes with those groups.

Yes. But you are choosing both the groups to be included for execution no?

@juherr @cbeust WDYT ?

Yes, I'm including both groups because I want to run all tests in both groups but I don't think it should affect any configuration method execution. I still think that in this case before/after methods with groups should run before/after classes with those groups. ¯_(ツ)_/¯

@MindaugasMateika - I have done my best to help provide an explanation. Since I see that you aren't convinced, let me step aside and wait for either @cbeust or @juherr to provide you with an answer.

The problem related to https://github.com/cbeust/testng/issues/2135#issuecomment-524169438 is due to the fact that when we include group information via the @Test method at the class level, then configuration methods end up inheriting the groups as well. This is the behavior of TestNG.

it can be turned off by using the attribute inheritGroups=false via the @BeforeXXX configuration annotation.

Now we have also changed the behavior of TestNG such that TestNG's behavior wherein it ends up inheriting groups info from class level @Test annotation onto its configuration annotations, is turned on only when group filtering is chosen via the <groups> tag. That change set is part of the PR https://github.com/cbeust/testng/pull/2167

@juherr @cbeust - All the outstanding queries are resolved in this issue. The only thing that's pending is for one of you to please help chime in and provide a resolution to @MindaugasMateika question (For some reason my explanations haven't been found to be satisfactory)

From code perspective there is nothing else that is required to be addressed in this issue.

@krmahadevan naah, that's fine. If I'm the only one with this issue and if everything works here as expected then I'll make it work in some other way on my own. I think we can close this 🔐

Closing this based on the above comment

@Proryanator Could you translate descriptions into samples? It will be easier to understand.

i.e.:

I have a 'BaseTest' class with a @BeforeMethod defined in it, with no @test methods in there. I'm just using @BeforeMethod with dependsOnMethods = {"importantParentBeforeMethod"} to prevent @BeforeMethod execution if this one important method fails.

public abstract class BaseTest {

  @BeforeMethod(dependsOnMethods = {"importantParentBeforeMethod"})
  public void setUp() {
    ...
  }
}

@Proryanator Could you translate descriptions into samples? It will be easier to understand.

i.e.:

I have a 'BaseTest' class with a @BeforeMethod defined in it, with no @test methods in there. I'm just using @BeforeMethod with dependsOnMethods = {"importantParentBeforeMethod"} to prevent @BeforeMethod execution if this one important method fails.

public abstract class BaseTest {

  @BeforeMethod(dependsOnMethods = {"importantParentBeforeMethod"})
  public void setUp() {
    ...
  }
}

I thought about doing this hehe, no problem!

I'm using TestNG 7.0.0 right now.

I'm primarily trying to understand how to run a _@BeforeMethod_ no matter the _group_, but still depend on another config method's success, i.e. the use of both _alwaysRun = true_ and _dependsOnMethods_.

So we've got a top-most parent class, which sets up a mobile driver. If something goes wrong, it does an assert fail.

public abstract class AbstractMobileBaseTest{
   @BeforeMethod(alwaysRun = true)
   public void mobileDriverCreation(){
      if (couldNotCreateDriver()){
         Assert.fail("Could not create the driver at all...");
      }
   }
}

A child class that all our test classes inherit from, which is supposed to always run 'createSharedObjects' no matter the group, but should not run if the method from the parent class failed.

public abstract class MobileAppBaseTest extends AbstractMobileBaseTest{
   @BeforeMethod(alwaysRun = true, dependsOnMethods = {"mobileDriverCreation"})
   public void makeMostPages(){
      // does some setup stuff here
   }
}

There are also other _@BeforeMethods_ in the child classes too which are not important to show here but, they are also using both _alwaysRun = true_ and _dependsOnMethods = {"mobileDriverCreation"}_, so they should also not run if something goes wrong in the other method.

I'm executing just one test right now to test how this works like this:

mvn verify ... -Dit.test=TestClassName -Dgroups=groupName

Here's the results. When something goes wrong in the mobileDriverCreation() @BeforeMethod I expect any child @BeforeMethods to not run, however I see the following:

There's an assertion failure in mobileDriverCreation(), yet all the other @BeforeMethods run (which throw their own exceptions and errors cause they depend on the first method). You can see that the methods still ran since they have dropdowns, and a ms execution time.

I did a followup test, where I directly specified in the child @BeforeMethods called _makeMostPages_ and _digitalServicesMobileSetup_ to use (groups = {"groupToRun"}, dependsOnMethods = {"mobileDriverCreation"}), kind of confirming that this is a weird thing with _alwaysRun = true_ only and not the group tags. You can see in the screenshot below that those two mentioned methods show 'broken' and did not run, because they were skipped due to the method it depends on failing, which is what we're going for!

It seems that _alwaysRun = true_ takes precedence over _dependsOnMethods_ and almost ignores it, or perhaps is utilized/checked last. I think we ought to add _dependsOnMethods_ as a later step.

I know that just using _dependsOnMethods_ works as I'm expecting so long as I don't use any _groups_ but, we use _groups_ regularly so I can't do that 😄

@juherr @krmahadevan

Just to be thorough too, our project is not using suite files (thank goodness). We run it using tags in the maven-failsafe-plugin like the following, where includes is _**.java_ by default, if this could have any impact on how _alwaysRun = true_ and _dependsOnMethods_ is interpreted:

I've even tried injecting using the tag, and not passing it via the command line, but I still see the above behavior.

<plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-failsafe-plugin</artifactId>
        <version>${failsafe.plugin.version}</version>
        <executions>
          <execution>
            <goals>
              <goal>integration-test</goal>
              <goal>verify</goal>
            </goals>
            <configuration>
              <!-- This reads in any other system variables that you provide via -DparamName=value -->
              <systemPropertyVariables>
                <mobile>${mobile}</mobile>
                <url>${url}</url>
                <platform>${platform}</platform>
                <remote>${remote}</remote>
                <!-- Thread count is set by default variable at the top, if you'd like to change it pass in 'threadCount' as a system variable -->
                <threadCount>${threadCount}</threadCount>
                <retryCount>${retryCount}</retryCount>
                <iosVersion>${iosVersion}</iosVersion>
              </systemPropertyVariables>
              <argLine>
                -javaagent:${settings.localRepository}/org/aspectj/aspectjweaver/${aspectj.version}/aspectjweaver-${aspectj.version}.jar
              </argLine>
              <!-- Cannot run parallel using methods yet, use classes -->
              <parallel>classes</parallel>
              <threadCount>${threadCount}</threadCount>
              <skip>false</skip>
              <encoding>${project.build.sourceEncoding}</encoding>
              <includes>
                <include>${include}</include>
              </includes>
              <workingDirectory>target/${mobile}</workingDirectory>
            </configuration>
          </execution>
        </executions>
      </plugin>

@Proryanator - I am not able to reproduce your issue using TestNG 7.3.0. Please try using the attached sample.

issue_2135.zip

If the sample is not what you are trying and if you are able to tweak the sample so that I can reproduce the problem, then please feel free to open a new issue and attach the sample to it.

@krmahadevan I love the bear analogy in your sample 😆

I downloaded your sample and made a few minor modifications, I just always failed the 'GrandpaBear' @BeforeMethod to make it a bit simpler, since we are concerned with when it fails:

Here's the top-most parent class:

public class GrandpaBearGitHubTest {
  @BeforeMethod(alwaysRun = true)
  public void mobileDriverCreation() {
    Assert.fail("Could not create the driver at all...");
  }
}

So now this @BeforeMethod will always throw an assertion error.

The first class below this one, with the _alwaysRun=true_ and _dependsOnMethods_, where it should not run if the @BeforeMethod called _mobileDriverCreation_ fails:

public class PapaBearGithubTest extends GrandpaBearGitHubTest {

  @BeforeMethod(alwaysRun = true, dependsOnMethods = {"mobileDriverCreation"})
  public void makeMostPages() {
    System.out.println("Ran makeMostPages");
  }
}

Here's the simple @Test class:

public class GrandsonBearGithubTest extends PapaBearGithubTest {

  @Test(groups = "yogi-bear")
  public void testMethod() {
    System.err.println("Running test method");
  }
}

Here's the results of running both with/without -Dgroups=yogi-bear. Both times the child class @BeforeMethod still runs (as you can see when it prints 'Ran makeMostPages', even though the method it depends on has failed:

[Run with -Dgroups]

[Run without -Dgroups]

I also ran it in IntelliJ to be sure it's about TestNG and not about the way that maven is handling groups, the results were the same:

My guess is that _alwaysRun=true_ might cause a short-circuit or, might just be a priority over _dependsOnMethods_ where I believe they should work together in conjunction with each other.

An even further exploration/test, I added a second @BeforeMethod to the _PapaBearGithubTest_ to use the specific group itself instead of _alwaysRun=true_ and that @BeforeMethod did not run, indicating that the behavior issue revolves around _alwaysRun=true_:

This now only runs this @BeforeMethod when you specify -Dgroups=yogi-bear which is expected, however it will now not run the new @BeforeMethod because it respects the _dependsOnMethods_:

public class PapaBearGithubTest extends GrandpaBearGitHubTest {

  @BeforeMethod(alwaysRun = true, dependsOnMethods = {"mobileDriverCreation"})
  public void makeMostPages() {
    System.out.println("Ran makeMostPages");
  }

  @BeforeMethod(groups = {"yogi-bear"}, dependsOnMethods = {"mobileDriverCreation"})
  public void makeOtherPages() {
    System.out.println("Ran makeOtherPages");
  }
}

If the new @BeforeMethod would have run we would see 'Ran makeOtherPages' but we see no such output, just the 'Ran makeMostPages' from the first one:

@krmahadevan I've attached my updated sample, if you're also able to reproduce it I'll go ahead and open an issue.
updated-issue_2135.zip

I don't mind working on the fix, we could use this at my organization, but I'm not sure where to look in the source code for when it comes to how the parameters of @BeforeMethod are analyzed, you could point me in the right direction 😄

@Proryanator Could you open a new issue? It will be better than this old closed issue :)

@Proryanator Could you open a new issue? It will be better than this old closed issue :)

You can find the newly opened issue here: #2448