Testng: Inherited test methods do not get expected group behavior

Created on 1 Mar 2012 · 8Comments · Source: cbeust/testng

Consider the following test classes:

@Test
public abstract class ParentTest {
    public void parentTestMethod() {
    }
}

@Test(groups = "myGroup")
public class ChildTest extends ParentTest {
    public void childTestMethod() {
    }
}

When running this test (ChildTest), I would expect that both test methods (childTestMethod and parentTestMethod) belong to group 'myGroup'. Apparently this is not the actual behavior (tested with version 6.4).

I put together some code to investigate this issue:

TestNGMethodFinder testNGMethodFinder = new TestNGMethodFinder(new RunInfo(), new JDK15AnnotationFinder(new DefaultAnnotationTransformer()));
ITestNGMethod[] testMethods = testNGMethodFinder.getTestMethods(ChildTest.class, new XmlTest());
for (ITestNGMethod testMethod : testMethods) {
    String[] groups = testMethod.getGroups();
    System.out.println(testMethod + " groups:" + Arrays.toString(groups));
}

Output from running this code:

ChildTest.childTestMethod()[pri:0, instance:null] groups:[myGroup]
ParentTest.parentTestMethod()[pri:0, instance:null] groups:[]

I looked into the TestNG source code, and it appears that this behavior originates from the method org.testng.internal.BaseTestMethod#initGroups(Class). To be exact, I think the following statement is the cause of this rather odd behavior:

ITestOrConfiguration classAnnotation = (ITestOrConfiguration) getAnnotationFinder().findAnnotation(getMethod().getDeclaringClass(), annotationClass);

In order to fix this issue, this statement could be modified to:

ITestOrConfiguration classAnnotation = (ITestOrConfiguration) getAnnotationFinder().findAnnotation(getInstance().getClass(), annotationClass);

but I am not sure if this would be the correct way to resolve the issue. Nor eventual implications....

groups inheritance

Source

erikgb

All 8 comments

I see the problem, and your suggestion doesn't quite work, unfortunately. It's a bit tricky to solve because annotations are resolved in the init phase, which means that when the parent's @Test annotation is parsed, it obviously knows nothing about the fact that the child annotation will add more groups to it...

cbeust on 3 Mar 2012

Cedric, thank you for your prompt feedback on this issue. You say "it is a bit tricky to solve", does that mean that you do not think that this issue will have a chance of being resolved in the near future? Can you think about any possible workarounds?

erikgb on 6 Mar 2012

What I meant was: when the group resolution is performed, no test class has been instantiated yet, so when I look at parentTestMethod(), I don't have an instance of ChildTest that would allow me to put that method in this group.

Group resolution is purely static, while the example above can only be resolved dynamically.

Does this make sense?

cbeust on 6 Mar 2012

I have seen this behavior as well. When I run my tests without a groups filter, everything passes, but when running with groups, things fail. In some cases it is due to my test class inheriting from some other test class, but in other cases I have POJO test classes and it seems like my @BeforeMethod and @AfterMethod methods are not invoked properly.

mumrah on 21 Jun 2013

👍1

BTW, my solution is to use excludedgroups which of course won't work in all situations. @cbeust, has there been any work on a proper solution for this?

mumrah on 22 Jun 2013

👍1

More than a year later and this still does not work.
@cbeust IMHO this does not make any sense: since the parent class is abstract, its group resolution should be based in a concrete instance of a test class.

Controlix on 15 Jul 2014

👍1

FWIW I just ran into this as well.

ken-p on 23 Oct 2014

I created a custom method selector, that uses the groups from subclass Test annoation

/* *****************************************************************************
 * Copyright 2017 VMware, Inc.  All rights reserved. VMware Confidential
 * ****************************************************************************/
package com.vmware.cloud.systemtests.util;

import java.util.Arrays;
import java.util.List;

import org.apache.commons.lang3.ArrayUtils;
import org.testng.IMethodSelector;
import org.testng.IMethodSelectorContext;
import org.testng.ITestNGMethod;
import org.testng.annotations.Test;
import org.testng.xml.XmlTest;

/**
 * Selector that uses groups in subclass @Test annotations to determine if methods should be included
 */
public class SubclassAwareXmlMethodsSelector implements IMethodSelector {
    @Override
    public boolean includeMethod(IMethodSelectorContext context, ITestNGMethod method, boolean isTestMethod) {
        XmlTest xmlTest = method.getXmlTest();
        if (xmlTest == null) {
            return method.isAlwaysRun();
        }
        Class realTestClass = isTestMethod ? method.getTestClass().getRealClass() : method.getRealClass();
        Test testClassAnnotation = (Test) realTestClass.getAnnotation(Test.class);
        String[] classAnnotationGroups = testClassAnnotation != null ? testClassAnnotation.groups() : new String[0];
        List<String> groups = Arrays.asList(ArrayUtils.addAll(classAnnotationGroups, method.getGroups()));

        context.setStopped(true);
        if (groups.size() == 0 && xmlTest.getIncludedGroups().isEmpty()) {
            return true;
        } else if (groups.size() > 0 && xmlTest.getExcludedGroups().stream().anyMatch(groups::contains)) {
            return false;
        } else if (groups.size() > 0 && xmlTest.getIncludedGroups().stream().anyMatch(groups::contains)) {
            return true;
        } else {
            return method.isAlwaysRun();
        }
    }

    @Override
    public void setTestMethods(List<ITestNGMethod> list) {
    }
}