Testng: DataProvider not running tests In Parallel when using Factory

Created on 13 Nov 2018  ·  15Comments  ·  Source: cbeust/testng

TestNG Version

7.0.0-beta1

Expected behavior

I have a class that contains a DataProvider with 5 data sets including browser name, version, and platform as you can see here:

public class DataProviderManager {
    @DataProvider(parallel = true)
    public static Object[][] defaultWebDataProvider() {
        return new Object[][] {
                new Object[]{"chrome", "70", "Windows 10"},
                new Object[]{"internet explorer", "11", "Windows 10"},
                new Object[]{"safari", "12.0", "Mac 10.13"},
                new Object[]{"chrome", "70", "Mac 10.13"},
                new Object[]{"firefox", "63", "Mac 10.13"}
        };
    }
}

In my test class I am using Factory to pass data as such:

public class MSGWebExampleTest extends BaseTest {
    // Declare page objects
    private MSGHome msgHome;
    // Declare test data

    @Factory(dataProvider = StaticProps.DEFAULT_WEB_DATA_PROVIDER, dataProviderClass = DataProviderManager.class)
    public MSGWebExampleTest(String browser, String browserVersion, String platform) {
        super.setRunParams(browser, browserVersion, platform);
    }

    @BeforeMethod(alwaysRun = true)
    public void setupTest() {
        msgHome = new MSGHome();
    }

    @Features({GroupProps.WEB})
    @Test(groups = {GroupProps.DEBUG})
    @Parameters({ParamProps.WEB, ParamProps.MOBILE_WEB})
    public void openSiteTest() {
        new WebInteract(null,null).pause(1000).openUrl(URLBuilder.buildUrl());
    }
}

testing.xml:

<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd" >
<suite name="Test Engineering Core Suite" parallel="methods" thread-count="2" data-provider-thread-count="4">

If I set thread-count =2 and data-provider-thread-count =4 and parallel=true my test should run 8 times in parallel.

Actual behavior

Tests run sequentially regardless of how many threads I try to set as if parallel=true is ignored. I tried numerous attempts to get this to run in parallel using the testing.xml above, by passing -Dthreadcount=2 -Ddataproviderthreacount=4 from commandline and even setting these values in the config tag for the surefire plugin and nothing is working.

Is the issue reproducible on runner?

  • [x] Shell
  • [x] Maven
  • [ ] Gradle
  • [ ] Ant
  • [ ] Eclipse
  • [x] IntelliJ
  • [ ] NetBeans

Test case sample

Please, share the test case (as small as possible) which shows the issue

Question
Source
picture VinceBarresi

All 15 comments

@VinceBarresi - I am not quite sure how you are saying that TestNG is running tests sequentially.

I altered your test class to print out the thread ids and here's how it looks like

import org.testng.annotations.DataProvider;
import org.testng.annotations.Factory;
import org.testng.annotations.Test;

public class MSGWebExampleTest {

  private String browser, browserVersion, platform;

  @Factory(dataProvider = "dp")
  public MSGWebExampleTest(String browser, String browserVersion, String platform) {
    this.browser = browser;
    this.browserVersion = browserVersion;
    this.platform = platform;
  }

  @Test
  public void openSiteTest() {
    System.err.println("Running " + this + " on thread [" + Thread.currentThread().getId() + "]");
  }

  @Override
  public String toString() {
    return String.format("[%s, %s, %s]", browser, browserVersion, platform);
  }

  @DataProvider(parallel = true, name = "dp")
  public static Object[][] defaultWebDataProvider() {
    return new Object[][]{
        new Object[]{"chrome", "70", "Windows 10"},
        new Object[]{"internet explorer", "11", "Windows 10"},
        new Object[]{"safari", "12.0", "Mac 10.13"},
        new Object[]{"chrome", "70", "Mac 10.13"},
        new Object[]{"firefox", "63", "Mac 10.13"}
    };
  }
}

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="Test Engineering Core Suite"
  parallel="methods"
  data-provider-thread-count="4"
  thread-count="2"
  verbose="2">
  <test name="Test Engineering Core_test">
    <classes>
      <class name="com.rationaleemotions.github.issue1951.MSGWebExampleTest"/>
    </classes>
  </test>
</suite>

The output

...
... TestNG 7.0.0-beta1 by Cédric Beust ([email protected])
...
Running [chrome, 70, Mac 10.13] on thread [11]
Running [chrome, 70, Windows 10] on thread [12]
Running [internet explorer, 11, Windows 10] on thread [12]
Running [firefox, 63, Mac 10.13] on thread [11]
Running [safari, 12.0, Mac 10.13] on thread [12]
PASSED: openSiteTest on [chrome, 70, Mac 10.13]
PASSED: openSiteTest on [chrome, 70, Windows 10]
PASSED: openSiteTest on [internet explorer, 11, Windows 10]
PASSED: openSiteTest on [firefox, 63, Mac 10.13]
PASSED: openSiteTest on [safari, 12.0, Mac 10.13]

===============================================
    Test Engineering Core_test
    Tests run: 5, Failures: 0, Skips: 0
===============================================

===============================================
Test Engineering Core Suite
Total tests run: 5, Passes: 5, Failures: 0, Skips: 0
===============================================

As you can see from the above output, the thread ids are different (two threads to be specific since our thread-count is 2)

Your data provider in your sample is tied to a factory method and NOT to a @Test method. So it won't have any role to play i the number of threads being executed.

Only when the following happens -

  • a data provider is coupled with a @Test method
  • the parallel attribute of @DataProvider is set to true
  • the attribute parallel is set to methods in the <suite> (or) <test> tag
  • data-provider-thread-count value is set with a value higher than 1

Parallelism will be enabled by TestNG and the number of @Test methods powered by the data provider would be equal to the number of threads set via the attribute data-provider-thread-count.

Note: thread-count attribute in the suite xml file is applicable ONLY to REGULAR methods and data-provider-thread-count attribute in the suite xml file is applicable ONLY to a Data Provider powered test method

Please let me know if that explanation makes sense and if this issue can be closed.

krmahadevan picture krmahadevan on 14 Nov 2018

@krmahadevan Thank you for your detailed response. I'd like to run myself to test this out when I get a chance. I do have a couple of questions,

1: Does the data provider (in this case - "dp) have to be in the same class as the @Test we are passing the data provider to in order to still achieve parallelism by data-provider-thread-count and parallel = true?

2: Did you execute tests at the suite level? If so, does this even matter in terms of parallelism? For example, if I kick of my tests of group DEBUG via surefire using mvn test -Dgroups=DEBUG, do I need to pass the dataprovider threadcount via a parameter in this command?

VinceBarresi picture VinceBarresi on 14 Nov 2018

1: Does the data provider (in this case - "dp) have to be in the same class as the @Test we are passing the data provider to in order to still achieve parallelism by data-provider-thread-count and parallel = true?

Not necessary. It doesnt matter where the data provider resides.

2: Did you execute tests at the suite level? If so, does this even matter in terms of parallelism? For example, if I kick of my tests of group DEBUG via surefire using mvn test -Dgroups=DEBUG, do I need to pass the dataprovider threadcount via a parameter in this command?

Yes I did execute the tests at the suite level. It shouldn't matter. As
long as the data provider thread count is made available in the suite file
(default value if its not provided is 10 I believe), it should work
seamlessly.

krmahadevan picture krmahadevan on 15 Nov 2018

@krmahadevan So I had a chance to test this morning using your suggested changes and have a couple observations.

  1. If I keep the DataProvider in the test class and pass the DataProvider name to the test annotation, it appears the test is trying to run in parallel but Im seeing the following errors:
...
... TestNG 7.0.0-beta1 by Cédric Beust ([email protected])
...

org.testng.internal.reflect.MethodMatcherException: 
Data provider mismatch
Method: openSiteTest([])
Arguments: [(java.lang.String) chrome,(java.lang.String) 70,(java.lang.String) Windows 10]

    at org.testng.internal.reflect.DataProviderMethodMatcher.getConformingArguments(DataProviderMethodMatcher.java:40)
    at org.testng.internal.Parameters.injectParameters(Parameters.java:936)
    at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1057)
    at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:140)
    at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:122)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)


org.testng.internal.reflect.MethodMatcherException: 
Data provider mismatch
Method: openSiteTest([])
Arguments: [(java.lang.String) chrome,(java.lang.String) 70,(java.lang.String) Windows 10]

    at org.testng.internal.reflect.DataProviderMethodMatcher.getConformingArguments(DataProviderMethodMatcher.java:40)
    at org.testng.internal.Parameters.injectParameters(Parameters.java:936)
    at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1057)
    at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:140)
    at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:122)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)


org.testng.internal.reflect.MethodMatcherException: 
Data provider mismatch
Method: openSiteTest([])
Arguments: [(java.lang.String) chrome,(java.lang.String) 70,(java.lang.String) Windows 10]

    at org.testng.internal.reflect.DataProviderMethodMatcher.getConformingArguments(DataProviderMethodMatcher.java:40)
    at org.testng.internal.Parameters.injectParameters(Parameters.java:936)
    at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1057)
    at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:140)
    at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:122)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)


org.testng.internal.reflect.MethodMatcherException: 
Data provider mismatch
Method: openSiteTest([])
Arguments: [(java.lang.String) chrome,(java.lang.String) 70,(java.lang.String) Windows 10]

    at org.testng.internal.reflect.DataProviderMethodMatcher.getConformingArguments(DataProviderMethodMatcher.java:40)
    at org.testng.internal.Parameters.injectParameters(Parameters.java:936)
    at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1057)
    at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:140)
    at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:122)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)


org.testng.internal.reflect.MethodMatcherException: 
Data provider mismatch
Method: openSiteTest([])
Arguments: [(java.lang.String) chrome,(java.lang.String) 70,(java.lang.String) Windows 10]

    at org.testng.internal.reflect.DataProviderMethodMatcher.getConformingArguments(DataProviderMethodMatcher.java:40)
    at org.testng.internal.Parameters.injectParameters(Parameters.java:936)
    at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1057)
    at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:140)
    at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:122)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)


Test ignored.

===============================================
Test Engineering Core Suite
Total tests run: 5, Passes: 0, Failures: 5, Skips: 0
===============================================


Process finished with exit code 0
  1. If I move the DataProvider back to my DataProviderManager.java class and try to run with the DataProvider passed to the test annotation, it again appears to be trying to run in parallel but I see the following which is a different error:
...
... TestNG 7.0.0-beta1 by Cédric Beust ([email protected])
...
[Utils] [ERROR] [Error] org.testng.TestNGException: 
Method public void com.msg.test.coretest.uitests.msgdotcom.MSGWebExampleTest.openSiteTest() requires a @DataProvider named : defaultWebDataProvider
    at org.testng.internal.Parameters.findDataProvider(Parameters.java:614)
    at org.testng.internal.Parameters.handleParameters(Parameters.java:808)
    at org.testng.internal.Parameters.handleParameters(Parameters.java:775)
    at org.testng.internal.ParameterHandler.handleParameters(ParameterHandler.java:60)
    at org.testng.internal.ParameterHandler.createParameters(ParameterHandler.java:39)
    at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1024)
    at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:140)
    at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:122)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)

[Utils] [ERROR] [Error] org.testng.TestNGException: 
Method public void com.msg.test.coretest.uitests.msgdotcom.MSGWebExampleTest.openSiteTest() requires a @DataProvider named : defaultWebDataProvider
    at org.testng.internal.Parameters.findDataProvider(Parameters.java:614)
    at org.testng.internal.Parameters.handleParameters(Parameters.java:808)
    at org.testng.internal.Parameters.handleParameters(Parameters.java:775)
    at org.testng.internal.ParameterHandler.handleParameters(ParameterHandler.java:60)
    at org.testng.internal.ParameterHandler.createParameters(ParameterHandler.java:39)
    at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1024)
    at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:140)
    at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:122)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)

[Utils] [ERROR] [Error] org.testng.TestNGException: 
Method public void com.msg.test.coretest.uitests.msgdotcom.MSGWebExampleTest.openSiteTest() requires a @DataProvider named : defaultWebDataProvider
    at org.testng.internal.Parameters.findDataProvider(Parameters.java:614)
    at org.testng.internal.Parameters.handleParameters(Parameters.java:808)
    at org.testng.internal.Parameters.handleParameters(Parameters.java:775)
    at org.testng.internal.ParameterHandler.handleParameters(ParameterHandler.java:60)
    at org.testng.internal.ParameterHandler.createParameters(ParameterHandler.java:39)
    at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1024)
    at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:140)
    at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:122)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)

[Utils] [ERROR] [Error] org.testng.TestNGException: 
Method public void com.msg.test.coretest.uitests.msgdotcom.MSGWebExampleTest.openSiteTest() requires a @DataProvider named : defaultWebDataProvider
    at org.testng.internal.Parameters.findDataProvider(Parameters.java:614)
    at org.testng.internal.Parameters.handleParameters(Parameters.java:808)
    at org.testng.internal.Parameters.handleParameters(Parameters.java:775)
    at org.testng.internal.ParameterHandler.handleParameters(ParameterHandler.java:60)
    at org.testng.internal.ParameterHandler.createParameters(ParameterHandler.java:39)
    at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1024)
    at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:140)
    at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:122)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)

[Utils] [ERROR] [Error] org.testng.TestNGException: 
Method public void com.msg.test.coretest.uitests.msgdotcom.MSGWebExampleTest.openSiteTest() requires a @DataProvider named : defaultWebDataProvider
    at org.testng.internal.Parameters.findDataProvider(Parameters.java:614)
    at org.testng.internal.Parameters.handleParameters(Parameters.java:808)
    at org.testng.internal.Parameters.handleParameters(Parameters.java:775)
    at org.testng.internal.ParameterHandler.handleParameters(ParameterHandler.java:60)
    at org.testng.internal.ParameterHandler.createParameters(ParameterHandler.java:39)
    at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1024)
    at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:140)
    at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:122)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)


org.testng.TestNGException: 
Method public void com.msg.test.coretest.uitests.msgdotcom.MSGWebExampleTest.openSiteTest() requires a @DataProvider named : defaultWebDataProvider

    at org.testng.internal.Parameters.findDataProvider(Parameters.java:614)
    at org.testng.internal.Parameters.handleParameters(Parameters.java:808)
    at org.testng.internal.Parameters.handleParameters(Parameters.java:775)
    at org.testng.internal.ParameterHandler.handleParameters(ParameterHandler.java:60)
    at org.testng.internal.ParameterHandler.createParameters(ParameterHandler.java:39)
    at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1024)
    at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:140)
    at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:122)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)


org.testng.TestNGException: 
Method public void com.msg.test.coretest.uitests.msgdotcom.MSGWebExampleTest.openSiteTest() requires a @DataProvider named : defaultWebDataProvider

    at org.testng.internal.Parameters.findDataProvider(Parameters.java:614)
    at org.testng.internal.Parameters.handleParameters(Parameters.java:808)
    at org.testng.internal.Parameters.handleParameters(Parameters.java:775)
    at org.testng.internal.ParameterHandler.handleParameters(ParameterHandler.java:60)
    at org.testng.internal.ParameterHandler.createParameters(ParameterHandler.java:39)
    at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1024)
    at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:140)
    at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:122)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)


org.testng.TestNGException: 
Method public void com.msg.test.coretest.uitests.msgdotcom.MSGWebExampleTest.openSiteTest() requires a @DataProvider named : defaultWebDataProvider

    at org.testng.internal.Parameters.findDataProvider(Parameters.java:614)
    at org.testng.internal.Parameters.handleParameters(Parameters.java:808)
    at org.testng.internal.Parameters.handleParameters(Parameters.java:775)
    at org.testng.internal.ParameterHandler.handleParameters(ParameterHandler.java:60)
    at org.testng.internal.ParameterHandler.createParameters(ParameterHandler.java:39)
    at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1024)
    at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:140)
    at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:122)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)


org.testng.TestNGException: 
Method public void com.msg.test.coretest.uitests.msgdotcom.MSGWebExampleTest.openSiteTest() requires a @DataProvider named : defaultWebDataProvider

    at org.testng.internal.Parameters.findDataProvider(Parameters.java:614)
    at org.testng.internal.Parameters.handleParameters(Parameters.java:808)
    at org.testng.internal.Parameters.handleParameters(Parameters.java:775)
    at org.testng.internal.ParameterHandler.handleParameters(ParameterHandler.java:60)
    at org.testng.internal.ParameterHandler.createParameters(ParameterHandler.java:39)
    at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1024)
    at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:140)
    at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:122)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)


org.testng.TestNGException: 
Method public void com.msg.test.coretest.uitests.msgdotcom.MSGWebExampleTest.openSiteTest() requires a @DataProvider named : defaultWebDataProvider

    at org.testng.internal.Parameters.findDataProvider(Parameters.java:614)
    at org.testng.internal.Parameters.handleParameters(Parameters.java:808)
    at org.testng.internal.Parameters.handleParameters(Parameters.java:775)
    at org.testng.internal.ParameterHandler.handleParameters(ParameterHandler.java:60)
    at org.testng.internal.ParameterHandler.createParameters(ParameterHandler.java:39)
    at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1024)
    at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:140)
    at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:122)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)


Test ignored.

===============================================
Test Engineering Core Suite
Total tests run: 5, Passes: 0, Failures: 5, Skips: 0
===============================================

Any thoughts on this?

VinceBarresi picture VinceBarresi on 15 Nov 2018

Can you please share the test code you used to trigger the error ?

krmahadevan picture krmahadevan on 15 Nov 2018

Im using the exact testing.xml file you pasted above (obviously with the class pointing to my test class in my package). Here is my DataProviderManager.java

public class DataProviderManager {
    @DataProvider(parallel = true, name = "defaultWebDataProvider")
    public static Object[][] defaultWebDataProvider() {
        return new Object[][]{
                new Object[]{"chrome", "70", "Windows 10"},
                new Object[]{"internet explorer", "11", "Windows 10"},
                new Object[]{"safari", "12.0", "Mac 10.13"},
                new Object[]{"chrome", "70", "Mac 10.13"},
                new Object[]{"firefox", "63", "Mac 10.13"}
        };
    }
}

here is my test class

public class MSGWebExampleTest extends BaseTest {

    // Declare page objects
    private MSGHome msgHome;

    @Factory(dataProvider = StaticProps.DEFAULT_WEB_DATA_PROVIDER, dataProviderClass = DataProviderManager.class)
    //  @Factory(dataProvider = StaticProps.DEFAULT_WEB_DATA_PROVIDER)
    public MSGWebExampleTest(String browser, String browserVersion, String platform) {
        super.setRunParams(browser, browserVersion, platform);
    }

    @BeforeMethod(alwaysRun = true)
    public void setupTest() {
        msgHome = new MSGHome();
    }

    @TestCaseId("")
    @Features({GroupProps.WEB})
    @Test(groups = {GroupProps.DEBUG}, dataProvider = StaticProps.DEFAULT_WEB_DATA_PROVIDER)
    @Parameters({ParamProps.WEB, ParamProps.MOBILE_WEB})
      public void openSiteTest() {
        Logger.logMessage(Thread.currentThread().getId());
        new WebInteract(null, null).pause(1000).openUrl(URLBuilder.buildUrl());
    }
//    @DataProvider(parallel = true, name = StaticProps.DEFAULT_WEB_DATA_PROVIDER)
//    public static Object[][] defaultWebDataProvider() {
//        return new Object[][] {
//                new Object[]{"chrome", "70", "Windows 10"},
//                new Object[]{"internet explorer", "11", "Windows 10"},
//                new Object[]{"safari", "12.0", "Mac 10.13"},
//                new Object[]{"chrome", "70", "Mac 10.13"},
//                new Object[]{"firefox", "63", "Mac 10.13"}
//        };
//    }

}

If you want to test scenario one then uncomment whats in the test. For scenario 2 you can run as is

VinceBarresi picture VinceBarresi on 15 Nov 2018

@VinceBarresi - The problem in both the cases is due to your test code. I don't think TestNG is at fault here.

Scenario 1

The error org.testng.internal.reflect.MethodMatcherException: Data provider mismatch is coming up because you are tying the test method public void openSiteTest() with a data provider which returns 3 parameters defaultWebDataProvider() but your test method doesn't have any method arguments at all. You need to ensure that the test method and the data provider with which it is tied to, match in the parameters in terms of length, type and order of parameters. Please fix your test code.

Scenario 2

The error Method public void com.msg.test.coretest.uitests.msgdotcom.MSGWebExampleTest.openSiteTest() requires a @DataProvider named : defaultWebDataProvider is again due to an issue in your test code. You moved your data provider to a different class named DataProviderManager. But in your @Test method you are only referring to the data provider by name. By default if you dont provide the class name wherein your data provider resides, TestNG will always look for a data provider in the current class. Since its not found TestNG complains. Again you need to fix your test code so that you refer to a data provider properly.

Since the original problem stands resolved, I am closing this issue with resolution as Question answered

krmahadevan picture krmahadevan on 16 Nov 2018

@krmahadevan This is a misunderstanding of DataProvider on my part - Since I don't need the parameters passed from it in the test itself ( I need them in my threadlocal getters and setters to pass to selenium capabilities ) I didn't realize I needed to pass them to the actual test method. I really appreciate you clearing this up!

I do have have one more question on parallelism if you have a moment - How do I set data provider thread count and/or thread count so that all 5 data sets from

@DataProvider(parallel = true, name = "dp") public static Object[][] defaultWebDataProvider() { return new Object[][]{ new Object[]{"chrome", "70", "Windows 10"}, new Object[]{"internet explorer", "11", "Windows 10"}, new Object[]{"safari", "12.0", "Mac 10.13"}, new Object[]{"chrome", "70", "Mac 10.13"}, new Object[]{"firefox", "63", "Mac 10.13"} }; }
run at the exact same time for each @test ? Currently I can run in parallel but its executing the same set (e.g "chrome", "70", "Windows 10") n number of thread times instead of running each set at the same time.

VinceBarresi picture VinceBarresi on 16 Nov 2018

You need to be using a factory (@Factory annotation) with a data provider.

The factory uses the data provider to basically create test class instances (a class that houses one or more @Test methods), and then when they get run, TestNG would ensure that it would run all the test methods in an instance (use the attribute parallel=instances)

krmahadevan picture krmahadevan on 17 Nov 2018

@krmahadevan I'm using @Factory which is sets up capabilities including browser, version, platform from my DataProvider and initializes thread safe instances of WebDriver. If I set (parallel=instances) the test does run in parallel but is running multiple instances with the same browser, version, platform from my DataProvider as you can see below:

screen shot 2018-11-19 at 10 20 21 am

Instead of running multiple threads each of them on Safari, 12, Mac 10.13, I'm trying to run the test with on multiple threads in parallel with -

new Object[]{"chrome", "70", "Windows 10"}, THIS SHOULD BE THREAD 1
new Object[]{"internet explorer", "11", "Windows 10"}, THIS SHOULD BE THREAD 2
new Object[]{"safari", "12.0", "Mac 10.13"}, THIS SHOULD BE THREAD 3
new Object[]{"chrome", "70", "Mac 10.13"}, THIS SHOULD BE THREAD 4
new Object[]{"firefox", "63", "Mac 10.13"} THIS SHOULD BE THREAD 5

VinceBarresi picture VinceBarresi on 19 Nov 2018

@VinceBarresi - I have added a detailed answer to your question on stack overflow.

To wrap up in a nutshell, when you use a @Factory with a @DataProvider to produce test class instances, TestNG instantiates test class instances in parallel, but runs each of the @Test methods in each of the test classes in sequence.

krmahadevan picture krmahadevan on 19 Nov 2018
👍1

@krmahadevan your reply on StackOverflow was very enlightening, thank you!

One question: Let's say you use the same data provider for every test. Is there a way we can avoid duplicating the factory code in every class?

Thanks!

borbamartin picture borbamartin on 9 Mar 2020

@borbamartin - You can have all your @Factory powered factories share the same data provider using the dataProviderClass attribute in the @Factory annotation.

your reply on StackOverflow was very enlightening, thank you!

Glad to know it helped you 👍

krmahadevan picture krmahadevan on 10 Mar 2020

@krmahadevan yes, let me rephrase.

Let's say I have 50 feature classes which contain multiple tests each. This means I have to write the following 50 times (in each class)

@Factory(dataProvider = "browserMatrix")
public LoginFeature(BrowserConfig browserConfig) {
    super.browserConfig = browserConfig;
}

Question is, does TestNG provide something I can use to avoid writing this in every test class?

Thanks once again!

borbamartin picture borbamartin on 10 Mar 2020

@borbamartin - To the best of my knowledge, there's nothing in TestNG that will automatically do this for you. You would need to do it yourself.

krmahadevan picture krmahadevan on 12 Mar 2020
👍1
Was this page helpful?
0 / 5 - 0 ratings

Related issues

baflQA picture baflQA  ·  7Comments
RocketRaccoon picture RocketRaccoon  ·  4Comments
shaburov picture shaburov  ·  10Comments
blackduck-joe picture blackduck-joe  ·  13Comments
vlad230 picture vlad230  ·  12Comments