从简单的 java 程序调用 Giraph 作业

Question

我是 Giraph 和 Hadoop Yarn 的新手。跟随 Giraph 的quick start 引导我从命令行从源代码构建的 jar 运行示例作业。

我想从简单的 java 程序运行作业。这个问题的灵感来自之前类似的 MapReduce 工作question。寻找与 java 的依赖项类似的答案。

我在本地设置了纱线 - 需要有一种方法可以从 java 程序中提供作业。

很明显：https://giraph.apache.org/apidocs/org/apache/giraph/job/GiraphJob.html 肯定有办法解决这个问题 - 但我发现很难找到 Yarn 的示例。

Answer 1

从 GiraphRunner 的源代码中找到了一种方法：

@Test
public void testPageRank() throws IOException, ClassNotFoundException, InterruptedException {

    GiraphConfiguration giraphConf = new GiraphConfiguration(getConf());
    giraphConf.setWorkerConfiguration(1,1,100);
    GiraphConstants.SPLIT_MASTER_WORKER.set(giraphConf, false);

    giraphConf.setVertexInputFormatClass(JsonLongDoubleFloatDoubleVertexInputFormat.class);
    GiraphFileInputFormat.setVertexInputPath(giraphConf,
                                             new Path("/input/tiny-graph.txt"));
    giraphConf.setVertexOutputFormatClass(IdWithValueTextOutputFormat.class);

    giraphConf.setComputationClass(PageRankComputation.class);

    GiraphJob giraphJob = new GiraphJob(giraphConf, "page-rank");       

    FileOutputFormat.setOutputPath(giraphJob.getInternalJob(),
                                   new Path("/output/page-rank2"));
    giraphJob.run(true);
}

private Configuration getConf() {
    Configuration conf = new Configuration();
    conf.set("fs.defaultFS", "hdfs://localhost:9000");

    conf.set("yarn.resourcemanager.address", "localhost:8032");
    conf.set("yarn.resourcemanager.hostname", "localhost");

    // framework is now "yarn", should be defined like this in mapred-site.xm
    conf.set("mapreduce.framework.name", "yarn");
    return conf;
}