java.io.IOException：没有用于方案的文件系统：maprfs

Question

我正在尝试执行/运行一个 Web 应用程序（使用 Spring 工具套件），通过它可以将作业提交给 hadoop。我正在使用 apache-tomcat-6.0.36 服务器 和 ma​​pr 客户端。但是，当我运行我的应用程序时，出现以下错误：

java.io.IOException: No FileSystem for scheme: maprfs
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1534)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:69)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:1571)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1553)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:232)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:115)
at com.aexp.lookalike.PreInitialize.createVarFiles(PreInitialize.java:50)
at com.aexp.lookalike.PreInitialize.run(PreInitialize.java:98)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.springframework.data.hadoop.mapreduce.ToolExecutor.invokeTargetObject(ToolExecutor.java:31)
at org.springframework.data.hadoop.mapreduce.ToolExecutor.invokeTargetObject(ToolExecutor.java:27)
at org.springframework.data.hadoop.mapreduce.HadoopCodeExecutor.invokeTarget(HadoopCodeExecutor.java:185)
at org.springframework.data.hadoop.mapreduce.HadoopCodeExecutor.runCode(HadoopCodeExecutor.java:102)
at org.springframework.data.hadoop.mapreduce.ToolTasklet.execute(ToolTasklet.java:33)
at org.springframework.data.hadoop.mapreduce.ToolTasklet$$FastClassByCGLIB$$317c6986.invoke(<generated>)
at net.sf.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
at org.springframework.aop.framework.Cglib2AopProxy$CglibMethodInvocation.invokeJoinpoint(Cglib2AopProxy.java:688)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)
at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:119)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor.intercept(Cglib2AopProxy.java:621)
at org.springframework.data.hadoop.mapreduce.ToolTasklet$$EnhancerByCGLIB$$4b232693.execute(<generated>)
at org.springframework.batch.core.step.tasklet.TaskletStep$ChunkTransactionCallback.doInTransaction(TaskletStep.java:386)
at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:130)
at org.springframework.batch.core.step.tasklet.TaskletStep$2.doInChunkContext(TaskletStep.java:264)
at org.springframework.batch.core.scope.context.StepContextRepeatCallback.doInIteration(StepContextRepeatCallback.java:76)
at org.springframework.batch.repeat.support.RepeatTemplate.getNextResult(RepeatTemplate.java:367)
at org.springframework.batch.repeat.support.RepeatTemplate.executeInternal(RepeatTemplate.java:214)
at org.springframework.batch.repeat.support.RepeatTemplate.iterate(RepeatTemplate.java:143)
at org.springframework.batch.core.step.tasklet.TaskletStep.doExecute(TaskletStep.java:250)
at org.springframework.batch.core.step.AbstractStep.execute(AbstractStep.java:195)
at org.springframework.batch.core.job.SimpleStepHandler.handleStep(SimpleStepHandler.java:135)
at org.springframework.batch.core.job.flow.JobFlowExecutor.executeStep(JobFlowExecutor.java:61)
at org.springframework.batch.core.job.flow.support.state.StepState.handle(StepState.java:60)
at org.springframework.batch.core.job.flow.support.SimpleFlow.resume(SimpleFlow.java:144)
at org.springframework.batch.core.job.flow.support.SimpleFlow.start(SimpleFlow.java:124)
at org.springframework.batch.core.job.flow.FlowJob.doExecute(FlowJob.java:135)
at org.springframework.batch.core.job.AbstractJob.execute(AbstractJob.java:281)
at org.springframework.batch.core.launch.support.SimpleJobLauncher$1.run(SimpleJobLauncher.java:120)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)

我在配置时是否遗漏了什么？这个错误的原因是什么？我错过了将任何罐子添加到应用程序的类路径中吗？

Answer 1

问题是因为您的类路径没有映射器 jar。确保 CLASSPATH 和 LIBRARY_PATH 设置正确

CLASSPATH 应该包含 $MAPR_HOME/lib/maprfs-1.0.3-mapr-3.0.0.jar" LIB_PATH 应该包含 /opt/mapr/lib

Answer 2

我得到了类似的例外，但“hdfs”。我发现问题出在我使用具有依赖关系的 maven 创建的应用程序 jar 中，没有 hadoop hdfs jar 附带的 hdfs 的“服务”规范。我使用 maven shade 插件（maven-shade-plugin）使用其内置的转换器“org.apache.maven.plugins.shade.resource.ServicesResourceTransformer”解决了它。

我认为没有 shade 插件的 maven 不会将 hdfs 服务规范用于 hdfs 文件系统所需的“org.apache.hadoop.fs.FileSystem”。我的猜测是“maprfs”是另一个需要在 METAINF/services 文件夹中指定的文件系统。尝试手动提供。您可以从 hadoop hdfs jar 中获取它。

希望这会有所帮助。