使用 TestNG + Gradle 设置 Selenium 测试的状态不起作用答案

【问题标题】：Setting status of Selenium test using TestNG + Gradle not working使用 TestNG + Gradle 设置 Selenium 测试的状态不起作用
【发布时间】：2014-01-01 21:02:17
【问题描述】：

我添加了一个侦听器，用于截屏、将页面源转储到文件中以及将某些失败的测试状态设置为 SKIP。屏幕截图和 DOM 转储操作有效，但我的测试仍然在报告中列为失败。

我正在使用运行 Selenium 2.x 和 TestNG 的 Gradle。

这是我的监听代码：

package com.xmatters.testng;

import org.testng.ITestResult;
import org.testng.Reporter;
import org.testng.TestListenerAdapter;
import com.xmatters.webdriver.tests.BaseTest;

public class Listener extends TestListenerAdapter {

    @Override
    public void onConfigurationFailure(ITestResult itr) {
        BaseTest.writeSource(itr.getName()); //writes source to a text file (useful for hidden stack traces)
        BaseTest.takeScreenshot(itr.getName()); //captures and saves a screenshot
        System.out.println("DEBUG: Failed test result getName() = " + itr.getName());
        if (itr.getName().contains("createTestData")) {
            System.out.println("DEBUG: Skip condition met - createTestData");
            itr.setStatus(ITestResult.SKIP); //sets failed configurations to skipped, re-run logic will be added
            Reporter.setCurrentTestResult(itr);
        }
    }

    @Override
    public void onTestFailure(ITestResult tr) {
        BaseTest.writeSource(tr.getName()); //writes source to a text file (useful for hidden stack traces)
        BaseTest.takeScreenshot(tr.getName()); //captures and saves a screenshot
        System.out.println("DEBUG: Failed test result getName() = " + tr.getName());
        if (tr.getName().contains("createTestData")) {
            System.out.println("DEBUG: Skip condition met - createTestData");
            tr.setStatus(ITestResult.SKIP); //sets failed configurations to skipped, re-run logic will be added
            Reporter.setCurrentTestResult(tr);
        }
    }
}

这是我的 build.gradle：

apply plugin: 'java'
apply plugin: 'idea'
apply plugin: 'eclipse'

repositories {
    mavenCentral()
}

dependencies {
    compile "org.seleniumhq.selenium:selenium-java:2.38.0"
    compile "org.testng:testng:5.14.10"
    testCompile('org.uncommons:reportng:1.1.2') {
        exclude group: 'org.testng'
    }
    testCompile "junit:junit:4.8.2"
    compile "com.jayway.restassured:rest-assured:1.8.1"
}

//initialize thread count variable for parallel testing and default to 1
def threadCount = System.getProperty("MAXTHREADS", "1")

tasks.withType(Test) {
    maxParallelForks = 1
    forkEvery = 1000
    ignoreFailures = false

    // Pass all system properties to the tests
    systemProperties = System.getProperties()

    // Makes the standard streams (err and out) visible at console when running tests
    testLogging.showStandardStreams = true

    exclude '**/tasks/'

    classpath += configurations.testCompile
}

task firefox(type: Test) {
    maxParallelForks = Integer.valueOf(threadCount) //default is 1 if not specified
    testLogging.events "started"
    testLogging {
        events "started", "passed", "skipped", "failed", "standardOut", "standardError"
        exceptionFormat "full" // default is "short"
    }
    useTestNG() {
        excludeGroups 'chrome', 'broken'
        useDefaultListeners = false
        listeners << 'org.uncommons.reportng.HTMLReporter'
        listeners << 'org.uncommons.reportng.JUnitXMLReporter'
        listeners << 'com.xmatters.testng.Listener'
    }

    testResultsDir = file("${buildDir}/test-results/firefox")
    testReportDir = file("${reporting.baseDir}/firefox")

    systemProperties.BROWSER = System.getProperty('BROWSER', 'firefox')

    exclude '**/selenium/'
    exclude '**/setupscripts/'
}

task chrome(type: Test) {
    maxParallelForks = Integer.valueOf(threadCount) //default is 1 if not specified
    testLogging {
        events "started", "passed", "skipped", "failed", "standardOut", "standardError"
        exceptionFormat "full"
    }
    useTestNG() {
        excludeGroups 'broken'
        useDefaultListeners = false;
        listeners << 'org.uncommons.reportng.HTMLReporter'
        listeners << 'org.uncommons.reportng.JUnitXMLReporter'
        listeners << 'com.xmatters.testng.Listener'
    }

    testResultsDir = file("${buildDir}/test-results/chrome")
    testReportDir = file("${reporting.baseDir}/chrome")

    systemProperties.BROWSER = System.getProperty('BROWSER', 'chrome')

    exclude '**/selenium/'
    exclude '**/setupscripts/'
}

task internetExplorer(type: Test) {
    useTestNG() {
        excludeGroups 'chrome', 'broken'
        useDefaultListeners = false;
        listeners << 'org.uncommons.reportng.HTMLReporter'
        listeners << 'org.uncommons.reportng.JUnitXMLReporter'
        listeners << 'com.xmatters.testng.Listener'
    }
    testReportDir = file("${reporting.baseDir}/internetExplorer")
    testResultsDir = file("${buildDir}/test-results/internetExplorer")

    systemProperties.BROWSER = System.getProperty('BROWSER', 'internetExplorer')

    exclude '**/selenium/'
    exclude '**/setupscripts/'
}

task applylicense(type: Test) {
    useTestNG() {
        useDefaultListeners = false;
        listeners << 'org.uncommons.reportng.HTMLReporter'
        listeners << 'org.uncommons.reportng.JUnitXMLReporter'
        listeners << 'com.xmatters.testng.Listener'
    }

    testResultsDir = file("${buildDir}/test-results/applyLicense")
    testReportDir = file("${reporting.baseDir}/applyLicense")

    scanForTestClasses = false
    include '**/setupscripts/ApplyLicense.class'
}

task uatsetup(type: Test) {
    useTestNG() {
        useDefaultListeners = false;
        listeners << 'org.uncommons.reportng.HTMLReporter'
        listeners << 'org.uncommons.reportng.JUnitXMLReporter'
        listeners << 'com.xmatters.testng.Listener'
    }

    testResultsDir = file("${buildDir}/test-results/uatSetup")
    testReportDir = file("${reporting.baseDir}/uatSetup")

    scanForTestClasses = false
    include '**/setupscripts/UatSetup.class'
}

task restsetup(type: Test) {
    useTestNG() {
        useDefaultListeners = false;
        listeners << 'org.uncommons.reportng.HTMLReporter'
        listeners << 'org.uncommons.reportng.JUnitXMLReporter'
        listeners << 'com.xmatters.testng.Listener'
    }

    testResultsDir = file("${buildDir}/test-results/restSetup")
    testReportDir = file("${reporting.baseDir}/restSetup")

    scanForTestClasses = false
    include '**/setupscripts/RestSetup.class'
}

task soapsetup(type: Test) {
    useTestNG() {
        useDefaultListeners = false;
        listeners << 'org.uncommons.reportng.HTMLReporter'
        listeners << 'org.uncommons.reportng.JUnitXMLReporter'
        listeners << 'com.xmatters.testng.Listener'
    }

    testResultsDir = file("${buildDir}/test-results/soapSetup")
    testReportDir = file("${reporting.baseDir}/soapSetup")

    scanForTestClasses = false
    include '**/setupscripts/SoapSetup.class'
}

task selenium(type: Test) {
    testResultsDir = file("${buildDir}/test-results/selenium")
    testReportDir = file("${reporting.baseDir}/selenium")

    include '**/selenium/'
    exclude '**/webdriver/'
    exclude '**/*$*', '**/Abstract*'
    exclude '**/messagepanel/'
    exclude '**/propertylibrary/'
}

javadoc() {
    title = "xMod Page Objects API"
}

task wrapper(type: Wrapper) {
    gradleVersion = '1.7'
}

这是我想要标记为 SKIP 的测试的 Jenkins 控制台输出：

com.xmatters.webdriver.tests.FormPermissionsTest.createTestData STARTED

com.xmatters.webdriver.tests.FormPermissionsTest.createTestData FAILED
    org.openqa.selenium.TimeoutException: Timed out loading CompanyAdministratorsChangeWebLoginScreen
    Timed out after 60 seconds waiting for visibility of element located by By.selector: form#webLoginDetailsForm
    Build info: version: '2.39.0', revision: '14fa800511cc5d66d426e08b0b2ab926c7ed7398', time: '2013-12-16 13:18:38'
    System info: host: 'vic-apod-build', ip: '127.0.0.1', os.name: 'Linux', os.arch: 'amd64', os.version: '3.2.0-57-generic', java.version: '1.6.0_18'
    Driver info: driver.version: unknown
    Build info: version: '2.39.0', revision: '14fa800511cc5d66d426e08b0b2ab926c7ed7398', time: '2013-12-16 13:18:38'
    System info: host: 'vic-apod-build', ip: '127.0.0.1', os.name: 'Linux', os.arch: 'amd64', os.version: '3.2.0-57-generic', java.version: '1.6.0_18'
    Driver info: driver.version: unknown
        at org.openqa.selenium.support.ui.FluentWait.timeoutException(FluentWait.java:259)
        at org.openqa.selenium.support.ui.FluentWait.until(FluentWait.java:228)
        at com.xmatters.webui.pages.AbstractPageObject.waitForPageToLoad(AbstractPageObject.java:42)
        at com.xmatters.webui.pages.AbstractPageObject.<init>(AbstractPageObject.java:31)
        at com.xmatters.webui.pages.AbstractPageObject.<init>(AbstractPageObject.java:23)
        at com.xmatters.webui.pages.home.admin.companies.CompanyAdministratorsChangeWebLoginScreen.<init>(CompanyAdministratorsChangeWebLoginScreen.java:15)
        at com.xmatters.webui.pages.home.admin.companies.CompanyAdministratorsDetailsScreen.clickSave(CompanyAdministratorsDetailsScreen.java:67)
        at com.xmatters.webdriver.tasks.Users.createCompanyAdmin(Users.java:50)
        at com.xmatters.webdriver.tests.FormPermissionsTest.createTestData(FormPermissionsTest.java:62)

Gradle test STANDARD_OUT
    DEBUG: Failed test result getName() = createTestData
    DEBUG: Skip condition met - createTestData

任何见解将不胜感激！

【问题讨论】：

标签： java selenium jenkins gradle testng

【解决方案1】：

尝试使用以下代码：

if (tr.getName().contains("createTestData")) {
        tr.setStatus(ITestResult.SKIP); //sets failed configurations to skipped, re-run logic will be added
        Reporter.setCurrentTestResult(tr);
}

我的例子：

@Override
public void onTestFailure(ITestResult result) {
    result.setStatus(ITestResult.SUCCESS); // make all FAILED tests a SUCCESS
    Reporter.setCurrentTestResult(result);
}

使用此代码，测试运行结果如下

PASSED: shouldShowHelpPageForNewQueryPage
PASSED: shouldShowHelpPageForViewPage
PASSED: shouldShowInactiveCreateNewQuery

===============================================
Default test
Tests run: 3, Failures: 0, Skips: 0
===============================================


===============================================
Default suite
Total tests run: 3, Failures: 0, Skips: 0
===============================================

如果我从onTestFailure()中删除代码，结果如下：

FAILED: shouldShowHelpPageForNewQueryPage
java.lang.NullPointerException
...

FAILED: shouldShowHelpPageForViewPage
java.lang.NullPointerException
...

FAILED: shouldShowInactiveCreateNewQuery
java.lang.NullPointerException
...


===============================================
    Default test
    Tests run: 3, Failures: 3, Skips: 0
===============================================


===============================================
Default suite
Total tests run: 3, Failures: 3, Skips: 0
===============================================

【讨论】：

您好弗拉德，感谢您的推荐。我在result 的行中遇到了一个未解决的符号问题：Reporter.setCurrentTestResult(result); 这是否意味着传入 ITestResult 对象？
@Darwin 忘记更改变量名...更新了响应中的代码
酷，这就是我正在尝试的。使用旧版本的 TestNG 似乎是我的问题，并且更新它会导致问题。我会解决这个问题，然后再试一次。
好的，我在测试中添加了一些调试代码，Gradle 似乎仍在将测试解释和记录为失败，结果没有改变。逻辑似乎是事后运行，对实际报告没有影响。在我上面的问题中添加了更多信息。
如果我实现 ITestListener 的 onTestFailure() 方法，这段代码对我来说很好。