Robotium 对于测试活动和片段的启动速度是否可靠？

Question

我正在尝试编写黑盒自动化测试来断言诸如“确保登录页面出现在启动应用程序的 500 毫秒内”和“确保登录时间少于 2 秒”之类的内容。我想通过驱动真实应用的 UI 来做到这一点，以尽可能地模拟真实用户。

我正在使用 Robotium 5.0.1 进行黑盒 UI 测试，我希望添加一些简单的计时代码会很简单。但是，测试似乎在不同的地方间歇性地失败，即使在不发出网络请求的地方也是如此。在模拟器中本地运行多个测试时，似乎偶尔会出现约 2 秒的延迟（我们还使用 CloudBees 在云中的 Jenkins 上运行测试，尽管我还没有尝试过那里的测试）。

Robotium 是否适合用于此类测试？您对进行此类测试的最佳方法有什么建议吗？

这是我的测试：

public void testLogin() {
    AppData.getAppData().clear();

    startTimer();
    launchActivity();
    assertTrue(solo.waitForFragmentByTag("landingfragment", 3000));
    stopTimer();
    assertWasQuickerThan(500);

    startTimer();
    solo.clickOnButton("Log In");
    assertTrue(solo.waitForFragmentByTag("loginfragment", 3000));
    stopTimer();
    assertWasQuickerThan(500);

    solo.enterText(0, TestUtils.EXISTING_USER_EMAIL);
    solo.enterText(1, TestUtils.EXISTING_USER_PASSWORD);

    startTimer();
    solo.clickOnButton("Next");
    assertTrue(solo.waitForActivity(LaunchActivity.class, 3000));
    stopTimer();
    assertWasQuickerThan(2000);
}

这是 logcat（这显示登录页面在 16 毫秒内出现，但在按下登录按钮后，登录页面出现需要 2079 毫秒）：

03-12 14:46:11.535      386-571/system_process I/ActivityManager﹕ START u0 {cmp=com.example/com.example.ui.LaunchActivity} from pid 1180
03-12 14:46:11.555    1180-1193/com.example D/MyApp﹕ LoginTest: Step took 16ms to complete,
03-12 14:46:12.035    1180-1180/com.example D/dalvikvm﹕ GC_FOR_ALLOC freed 1470K, 47% free 3456K/6424K, paused 96ms, total 98ms
03-12 14:46:12.045    1180-1180/com.example I/dalvikvm-heap﹕ Grow heap (frag case) to 4.842MB for 1463056-byte allocation
03-12 14:46:12.145    1180-1281/com.example D/dalvikvm﹕ GC_FOR_ALLOC freed 5K, 25% free 4880K/6424K, paused 87ms, total 102ms
03-12 14:46:12.405      386-400/system_process I/ActivityManager﹕ Displayed com.example/com.example.ui.LaunchActivity: +848ms
03-12 14:46:13.115    1180-1180/com.example I/MyApp﹕ USER: LandingFragment: Sign in pressed,
03-12 14:46:13.315    1180-1180/com.example D/dalvikvm﹕ GC_FOR_ALLOC freed 1478K, 46% free 3508K/6424K, paused 21ms, total 23ms
03-12 14:46:13.315    1180-1180/com.example I/dalvikvm-heap﹕ Grow heap (frag case) to 4.761MB for 1324816-byte allocation
03-12 14:46:13.345    1180-1180/com.example D/dalvikvm﹕ GC_FOR_ALLOC freed 3K, 26% free 4798K/6424K, paused 23ms, total 23ms
03-12 14:46:13.395    1180-1180/com.example D/dalvikvm﹕ GC_FOR_ALLOC freed <1K, 26% free 4798K/6424K, paused 31ms, total 31ms
03-12 14:46:13.405    1180-1180/com.example I/dalvikvm-heap﹕ Grow heap (frag case) to 6.153MB for 1463056-byte allocation
03-12 14:46:13.425    1180-1281/com.example D/dalvikvm﹕ GC_FOR_ALLOC freed 0K, 21% free 6227K/7856K, paused 26ms, total 26ms
03-12 14:46:13.445    1180-1180/com.example I/Choreographer﹕ Skipped 47 frames!  The application may be doing too much work on its main thread.
03-12 14:46:13.635    1180-1193/com.example D/MyApp﹕ LoginTest: Step took 2079ms to complete,
03-12 14:46:14.695    1180-1180/com.example I/Choreographer﹕ Skipped 52 frames!  The application may be doing too much work on its main thread.
03-12 14:46:15.325    1180-1193/com.example I/TestRunner﹕ failed: testLogin(com.example.blackbox.LoginTest)
03-12 14:46:15.335    1180-1193/com.example I/TestRunner﹕ ----- begin exception -----
03-12 14:46:15.335    1180-1193/com.example I/TestRunner﹕ junit.framework.AssertionFailedError: Step was too slow, expected 500ms but took 2079ms
            at junit.framework.Assert.fail(Assert.java:50)
            at junit.framework.Assert.assertTrue(Assert.java:20)
            at com.example.blackbox.BaseBlackBoxTest.assertWasQuickerThan(BaseBlackBoxTest.java:57)
            at com.example.blackbox.LoginTest.testLogin(LoginTest.java:52)

...这是我的测试类扩展的BaseBlackBoxTest 类：

abstract class BaseBlackBoxTest<T extends android.app.Activity>
        extends ActivityInstrumentationTestCase2<T> {

    protected Solo solo;

    protected long mStartTime;
    protected long mStopTime;

    @Before
    public void setUp() throws Exception {
        solo = new Solo(getInstrumentation(), getActivity());
    }

    @After
    public void tearDown() throws Exception {
        solo.finishOpenedActivities();
    }

    public BaseBlackBoxTest(Class clazz) {
        super(clazz);
    }

    protected void launchActivity() {
        Activity activity = getActivity();
        Intent intent = new Intent(activity, activity.getClass());
        activity.startActivity(intent);

        solo.assertCurrentActivity("Expecting " + activity.getClass(), activity.getClass());
    }

    // TIMING UTILITIES //

    protected void startTimer() {
        mStartTime = System.nanoTime();
    }

    protected void stopTimer() {
        mStopTime = System.nanoTime();
    }

    protected void assertWasQuickerThan(long maxDurationMillis) {
        long durationMillis = (mStopTime - mStartTime) / 1000000;
        LogIt.d(this, "Step took " + durationMillis + "ms to complete");
        assertTrue("Step was too slow, expected " + maxDurationMillis + "ms but took " + durationMillis + "ms",
                durationMillis < maxDurationMillis);
    }
}

Answer 1

恕我直言，我认为您不应该进行这样的自动化性能测试，因为您无法控制模拟器的性能。您应该专注于测试您的应用显示的内容，而不是它的运行速度。

如果您需要检查 Web 服务的可用性，您可以进行单独的测试，调用 HTTP 请求并在您指定的超时后失败。