Webdriverio: Bail config not working with Jasmine in v5

Created on 24 May 2019  ·  3Comments  ·  Source: webdriverio/webdriverio

Describe your question with as much detail as possible
I recently upgraded wdio from v3 to v5. All is working great except for the bail option. Previously in v3, it worked with setting bail: 1 and stopOnSpecFailure: true in jasmineNodeOpts. However, it is not working as such in v5.

Q: Did something change in the functionality here?

Follow up Q: I would also like to use the functionality of both bail and specFileRetries in order to retry the test up to 3 times on the first failure. Will that be achievable? As I remember reading something where specFileRetries and bail setup can conflict with one another.

If it's about a specific piece of code, try and include some of it to support your question.
[...]

Environment (please complete the following information):

  • WebdriverIO version: ^5.0.0 [e.g. 4.13.2]
  • Mode: WDIO Testrunner [Standalone mode or WDIO Testrunner]
  • If WDIO Testrunner, running sync/async: sync [e.g. sync/async]
  • Node.js version: 10.15.3 [e.g. 8.11.2]
  • NPM version: 6.4.1 [e.g. 5.8.0]
  • Browser name and version: Chrome 74 [e.g. Chrome 68]
  • Platform name and version: Mac OS X and Linux [e.g. Windows 10]
  • Additional wdio packages used (if applicable): @wdio/jasmine-framework, @wdio/spec reporter, @wdio/local-runner, @wdio/cli, @wdio/sync

Config of WebdriverIO

// ====================
// Runner Configuration
// ====================
//
// WebdriverIO allows it to run your tests in arbitrary locations (e.g. locally or
// on a remote machine).
runner: 'local',
//
// ==================
// Specify Test Files
// ==================
//
suites: suites,
// Specs to include/exclude.
// specs: ,
// exclude: ,
//
// ============
// Capabilities
// ============
//
maxInstances: 10,
//
capabilities: [
    {
        maxInstances: 5,
        //
        browserName: 'chrome',
        'goog:chromeOptions': {
            // to run chrome headless the following flags are required
            // (see https://developers.google.com/web/updates/2017/04/headless-chrome)
            // args: ['--headless', '--disable-gpu'],
        }
        // If outputDir is provided WebdriverIO can capture driver session logs
        // it is possible to configure which logTypes to include/exclude.
        // excludeDriverLogs: ['*'], // pass '*' to exclude all driver session logs
        // excludeDriverLogs: ['bugreport', 'server'],
    }
],
//
// ===================
// Test Configurations
// ===================
// Define all options that are relevant for the WebdriverIO instance here
//
// Level of logging verbosity: trace | debug | info | warn | error | silent
logLevel: 'trace',
//
// Set directory to store all logs into
outputDir: `${path.output}`,
// If you only want to run your tests until a specific amount of tests have failed use
// bail (default is 0 - don't bail, run all tests).
bail: 1,
//
baseUrl: url.base,
//
// Default timeout for all waitFor* commands.
waitforTimeout: waitTimeout,
//
// Default timeout in milliseconds for request
// if Selenium Grid doesn't send response
connectionRetryTimeout: 90000,
//
// Default request retries count
connectionRetryCount: 3,
//
// Make sure you have the wdio adapter package for the specific framework installed
// before running any tests.
framework: 'jasmine',
//
// The number of times to retry the entire specfile when it fails as a whole
specFileRetries: 0,
//
reporters: ['spec'],
//
// Options to be passed to Jasmine.
jasmineNodeOpts: {
    //
    // Jasmine default timeout
    defaultTimeoutInterval: parameter.testTimeout,
    //
    stopOnSpecFailure: true,
    // The Jasmine framework allows interception of each assertion in order to log the state of the application
    // or website depending on the result. For example, it is pretty handy to take a screenshot every time
    // an assertion fails.
    expectationResultHandler: function(passed, assertion) {
        /**
         * only take screenshot if assertion failed
         */
        if (passed) {
            return;
        }
    }
},

Additional context
Context on follow up question: As some spec files are big, it would be great if I could use bail along to save time. Bail option is not working both with and without specFileRetries.

Bug🐛 good first pick help wanted

Most helpful comment

I'm also experiencing the same issue upgrading from wdio v4->v5

I have tried permutations of these options:

        bail: 1,
        stopSpecOnExpectationFailure: true,
        stopOnSpecFailure: true,

With a single test, it is retried ~ 8 times (95s) where the first failure happens around 20s. This can save us a lot of time when debugging our tests.

All 3 comments

I'm also experiencing the same issue upgrading from wdio v4->v5

I have tried permutations of these options:

        bail: 1,
        stopSpecOnExpectationFailure: true,
        stopOnSpecFailure: true,

With a single test, it is retried ~ 8 times (95s) where the first failure happens around 20s. This can save us a lot of time when debugging our tests.

For people still facing this issue out there, my workaround was to set failFast: true in JasmineNodeOpts, in addition to bail: 1. It now stops execution on first failure and retries as well with specFileRetries.

As some spec files are big, it would be great if I could use bail along to save time. Bail option is not working both with and without specFileRetries.

Ah, I understand the issue now. The WebdriverIO bail option is only looking at wdio spec files. As stated in the documentation:

If you only want to run your tests until a specific amount of tests have failed use bail (default is 0 - don't bail, run all tests). Note: Please be aware that when using a third party test runner such as Mocha, additional configuration might be required.

If you want to bail within spec files you need to use framework specific options.

Was this page helpful?
0 / 5 - 0 ratings