MapReduce - 用 Mockito 模拟

Question

我有一个要编写测试用例的 reducer 类：

减少类：

public class MyReducer extends Reducer<Text, Text, NullWritable, Text> {
    private static final Logger LOG = LogManager.getLogger(MyReducer.class);
    public static List<String> l1 = new ArrayList<String>();
    String id = null;
    private MultipleOutputs<NullWritable, Text> mos;

    @Override
    public void setup(final Context context) throws IOException, InterruptedException {
        mos = new MultipleOutputs<NullWritable, Text>(context);

         final Path[] uris = DistributedCache.getLocalCacheFiles(context.getConfiguration());

        try {
            final BufferedReader readBuffer1 = new BufferedReader(new FileReader(uris[0].toString()));
            String line;
            while ((line = readBuffer1.readLine()) != null) {
                l1.add(line);
            }
            readBuffer1.close();
        } catch (Exception e) {
            LOG.error(e);
        }
    }

    public void reduce(final Text key, final Iterable<Text> values, final Context context)
            throws IOException, InterruptedException {

        final String[] key1 = key.toString().split("-");
        final String keyA = key1[10];
        final String date = key1[1];

/* Some condition check */ 

           mos.write(NullWritable.get(), new Text(inputEventValue), keyA + "//date=" +
                    date.substring(0, 4) + "-" + date.substring(4, 6));

       }

    @Override
    public void cleanup(final Context context) throws IOException, InterruptedException {
        mos.close();
    }

}

测试用例看起来像：

@RunWith(MockitoJUnitRunner.class)
public class MyTest {

   @Mock
    private MyReducer.Context mockContext;

    MyReducer reducer;
    MultipleOutputs<NullWritable, Text> mos;

    @Before
    public void setUp() {
        reducer = new MyReducer();
    }


   @Test
    public void myReducerTest() throws Exception {

        MyReducer spy = PowerMockito.spy(new MyReducer());
        doNothing().when(spy).setup(mockContext);

        mos = new MultipleOutputs<NullWritable, Text>(mockContext);
        List<Text> sline = new ArrayList<>() ;
        List<String> l1 = new ArrayList<String>();
        l1.add(“1234”);
        sline.add(new Text(“xyz”));
        Whitebox.setInternalState(MyReducer.class,”l1", l1);
        Whitebox.setInternalState(MyReducer.class,"mos",mos);
        reducer.reduce(new Text(“xyz-20200101-1234),sline,mockContext);

    }

    @After
    public void tearDown() throws Exception {
        /*
         * this will do the clean up part
         */
        verifyNoMoreInteractions(mockContext);
    }

在 Debug 模式下运行时，它会转到 reducer 的 reduce 方法，并在 mos write 语句所在的位置出现 NullPointerException 失败？

完整的堆栈跟踪：

java.lang.NullPointerException
    at org.apache.hadoop.mapreduce.lib.output.MultipleOutputs.getNamedOutputsList(MultipleOutputs.java:196)
    at org.apache.hadoop.mapreduce.lib.output.MultipleOutputs.<init>(MultipleOutputs.java:324)
    at MyTest.myeducerTest
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.junit.internal.runners.TestMethod.invoke(TestMethod.java:66)
    at org.powermock.modules.junit4.internal.impl.PowerMockJUnit44RunnerDelegateImpl$PowerMockJUnit44MethodRunner.runTestMethod(PowerMockJUnit44RunnerDelegateImpl.java:310)
    at org.junit.internal.runners.MethodRoadie$2.run(MethodRoadie.java:86)
    at org.junit.internal.runners.MethodRoadie.runBeforesThenTestThenAfters(MethodRoadie.java:94)
    at org.powermock.modules.junit4.internal.impl.PowerMockJUnit44RunnerDelegateImpl$PowerMockJUnit44MethodRunner.executeTest(PowerMockJUnit44RunnerDelegateImpl.java:294)
    at org.powermock.modules.junit4.internal.impl.PowerMockJUnit47RunnerDelegateImpl$PowerMockJUnit47MethodRunner.executeTestInSuper(PowerMockJUnit47RunnerDelegateImpl.java:127)
    at org.powermock.modules.junit4.internal.impl.PowerMockJUnit47RunnerDelegateImpl$PowerMockJUnit47MethodRunner.executeTest(PowerMockJUnit47RunnerDelegateImpl.java:82)
    at org.powermock.modules.junit4.internal.impl.PowerMockJUnit44RunnerDelegateImpl$PowerMockJUnit44MethodRunner.runBeforesThenTestThenAfters(PowerMockJUnit44RunnerDelegateImpl.java:282)
    at org.junit.internal.runners.MethodRoadie.runTest(MethodRoadie.java:84)
    at org.junit.internal.runners.MethodRoadie.run(MethodRoadie.java:49)
    at org.powermock.modules.junit4.internal.impl.PowerMockJUnit44RunnerDelegateImpl.invokeTestMethod(PowerMockJUnit44RunnerDelegateImpl.java:207)
    at org.powermock.modules.junit4.internal.impl.PowerMockJUnit44RunnerDelegateImpl.runMethods(PowerMockJUnit44RunnerDelegateImpl.java:146)
    at org.powermock.modules.junit4.internal.impl.PowerMockJUnit44RunnerDelegateImpl$1.run(PowerMockJUnit44RunnerDelegateImpl.java:120)
    at org.junit.internal.runners.ClassRoadie.runUnprotected(ClassRoadie.java:34)
    at org.junit.internal.runners.ClassRoadie.runProtected(ClassRoadie.java:44)
    at org.powermock.modules.junit4.internal.impl.PowerMockJUnit44RunnerDelegateImpl.run(PowerMockJUnit44RunnerDelegateImpl.java:118)
    at org.powermock.modules.junit4.common.internal.impl.JUnit4TestSuiteChunkerImpl.run(JUnit4TestSuiteChunkerImpl.java:101)
    at org.powermock.modules.junit4.common.internal.impl.AbstractCommonPowerMockRunner.run(AbstractCommonPowerMockRunner.java:53)
    at org.powermock.modules.junit4.PowerMockRunner.run(PowerMockRunner.java:53)

将 mos 错误模拟为 mos 不是静态的。

任何建议。

Junit - ReduceDriver, withInput, withOutput,testRun  doesn't work.

谢谢。

我尝试按照建议模拟多个输出：

导入 org.apache.hadoop.mapreduce.lib.output.MultipleOutputs；

@Mock 私有 MyReducer.Context 模拟上下文；

    List<String> namedOut = new ArrayList<>();
    namedOut.add("NM1");
    namedOut.add("NM2");

MultipleOutputs spy = PowerMockito.spy(new MultipleOutputs(mockContext)); when(spy, "getNamedOutputsList(mockContext)").thenReturn(namedOut);

但这给了我错误：org.powermock.reflect.exceptions.MethodNotFoundException：在类 org.apache.hadoop.mapreduce 中找不到名称为“getNamedOutputsList(() anyObject())”且参数类型为 [] 的方法。 lib.output.MultipleOutputs。

Answer 1

看起来你没有定义 mockContext.getContext() 应该为你的测试返回什么，所以它返回 null 并且失败了。

基于 this sourcecode 的方法如下所示（因此您可能会使用不同的版本）：

private static List<String> getNamedOutputsList(JobContext job) {
   List<String> names = new ArrayList<String>();
   StringTokenizer st = new StringTokenizer(
     job.getConfiguration().get(MULTIPLE_OUTPUTS, ""), " ");
   while (st.hasMoreTokens()) {
     names.add(st.nextToken());
   }
   return names;
}

JobContext 似乎指的是您的模拟 Reducer.Context mockContext，因此您需要定义适当的行为，以便它返回应该返回的内容。

请注意，此调用源自MultipleOutputs 的构造函数。

还要注意从构造函数调用并与上下文交互的静态getCountersEnabled 方法。

---

将 mos 错误模拟为 mos 不是静态的。

您可能可以使用反射将 mos 的模拟版本放入您的 MyReducer 类中。

查看here 获取有关如何模拟私有静态字段的示例。

---

编辑：

如果您尝试模拟 conig，请这样做：

Configuration config = Mockito.mock(Configuration.class);
when(mockContext.getConfiguration()).thenReturn(config);

据我所知，在配置对象上调用的 get 始终提供默认值，因此键/值对是否在其中并不重要。