nutch1.14 去重失败答案

【问题标题】：nutch1.14 deduplication failednutch1.14 去重失败
【发布时间】：2018-01-09 12:21:30
【问题描述】：

我在 CentOS Linux 版本 7.3.1611 上集成了 nutch 1.14 和 solr-6.6.0 我在 /usr/local/apache-nutch-1.13/urls/seed.txt 的种子列表中提供了大约 10 个 url关注tutorial

[root@localhost apache-nutch-1.14]# bin/nutch dedup http://ip:8983/solr/
DeduplicationJob: starting at 2018-01-09 15:07:52
DeduplicationJob: java.io.IOException: No FileSystem for scheme: http
    at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2660)
    at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2667)
    at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:94)
    at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2703)
    at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2685)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:373)
    at org.apache.hadoop.fs.Path.getFileSystem(Path.java:295)
    at org.apache.hadoop.mapred.FileInputFormat.singleThreadedListStatus(FileInputFormat.java:258)
    at org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:229)
    at org.apache.hadoop.mapred.SequenceFileInputFormat.listStatus(SequenceFileInputFormat.java:45)
    at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:315)
    at org.apache.hadoop.mapreduce.JobSubmitter.writeOldSplits(JobSubmitter.java:329)
    at org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:320)
    at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:196)
    at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1290)
    at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1287)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1746)
    at org.apache.hadoop.mapreduce.Job.submit(Job.java:1287)
    at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:575)
    at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:570)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1746)
    at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:570)
    at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:561)
    at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:870)
    at org.apache.nutch.crawl.DeduplicationJob.run(DeduplicationJob.java:326)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
    at org.apache.nutch.crawl.DeduplicationJob.main(DeduplicationJob.java:369)

所有与 solr 相关的命令都有效。请帮忙。他们在 nutch 教程中谈论的 hadoop 元素在哪里。我们是否必须安装除 java 之外的任何东西才能让 hadoop、nutch 和 solr 协同工作来构建搜索引擎？

【问题讨论】：

标签： hadoop solr web-crawler bigdata nutch

【解决方案1】：

试试这个

bin/nutch dedup -Dsolr.server.url=http://ip:8983/solr/

【讨论】：

【解决方案2】：

我正在阅读相同的指南并遇到了同样的问题。这可能会有所帮助：

(Step-by-Step: Deleting Duplicates)  
$ bin/nutch dedup crawl/crawldb/ -Dsolr.server.url=http://localhost:8983/solr/nutch

DeduplicationJob: starting at 2018-02-23 14:27:34  
Deduplication: 1 documents marked as duplicates  
Deduplication: Updating status of duplicate urls into crawl db.  
Deduplication finished at 2018-02-23 14:27:37, elapsed: 00:00:03

【讨论】：