在 nutch solrindex 命令后，Solr 索引为空答案

【问题标题】：Solr index empty after nutch solrindex command在 nutch solrindex 命令后，Solr 索引为空
【发布时间】：2011-08-05 00:29:25
【问题描述】：

我正在使用 Nutch 和 Solr 来索引文件共享。

我的第一个问题：bin/nutch 抓取网址

这给了我：

solrUrl is not set, indexing will be skipped...
crawl started in: crawl-20110804191414
rootUrlDir = urls
threads = 10
depth = 5
solrUrl=null
Injector: starting at 2011-08-04 19:14:14
Injector: crawlDb: crawl-20110804191414/crawldb
Injector: urlDir: urls
Injector: Converting injected urls to crawl db entries.
Injector: Merging injected urls into crawl db.
Injector: finished at 2011-08-04 19:14:16, elapsed: 00:00:02
Generator: starting at 2011-08-04 19:14:16
Generator: Selecting best-scoring urls due for fetch.
Generator: filtering: true
Generator: normalizing: true
Generator: jobtracker is 'local', generating exactly one partition.
Generator: Partitioning selected urls for politeness.
Generator: segment: crawl-20110804191414/segments/20110804191418
Generator: finished at 2011-08-04 19:14:20, elapsed: 00:00:03
Fetcher: Your 'http.agent.name' value should be listed first in 'http.robots.agents' property.
Fetcher: starting at 2011-08-04 19:14:20
Fetcher: segment: crawl-20110804191414/segments/20110804191418
Fetcher: threads: 10
QueueFeeder finished: total 1 records + hit by time limit :0
-finishing thread FetcherThread, activeThreads=9
-finishing thread FetcherThread, activeThreads=8
-finishing thread FetcherThread, activeThreads=7
-finishing thread FetcherThread, activeThreads=6
-finishing thread FetcherThread, activeThreads=5
-finishing thread FetcherThread, activeThreads=4
-finishing thread FetcherThread, activeThreads=3
-finishing thread FetcherThread, activeThreads=2
-finishing thread FetcherThread, activeThreads=1
fetching file:///mnt/public/Personal/Reminder Building Security.htm
-finishing thread FetcherThread, activeThreads=0
-activeThreads=0, spinWaiting=0, fetchQueues.totalSize=0
-activeThreads=0
Fetcher: finished at 2011-08-04 19:14:22, elapsed: 00:00:02
ParseSegment: starting at 2011-08-04 19:14:22
ParseSegment: segment: crawl-20110804191414/segments/20110804191418
ParseSegment: finished at 2011-08-04 19:14:23, elapsed: 00:00:01
CrawlDb update: starting at 2011-08-04 19:14:23
CrawlDb update: db: crawl-20110804191414/crawldb
CrawlDb update: segments: [crawl-20110804191414/segments/20110804191418]
CrawlDb update: additions allowed: true
CrawlDb update: URL normalizing: true
CrawlDb update: URL filtering: true
CrawlDb update: Merging segment data into db.
CrawlDb update: finished at 2011-08-04 19:14:24, elapsed: 00:00:01
Generator: starting at 2011-08-04 19:14:24
Generator: Selecting best-scoring urls due for fetch.
Generator: filtering: true
Generator: normalizing: true
Generator: jobtracker is 'local', generating exactly one partition.
Generator: 0 records selected for fetching, exiting ...
Stopping at depth=1 - no more URLs to fetch.
LinkDb: starting at 2011-08-04 19:14:25
LinkDb: linkdb: crawl-20110804191414/linkdb
LinkDb: URL normalize: true
LinkDb: URL filter: true
LinkDb: adding segment: file:/home/nutch/nutch-1.3/runtime/local/crawl-20110804191414/segments/20110804191418
LinkDb: finished at 2011-08-04 19:14:26, elapsed: 00:00:01
crawl finished: crawl-20110804191414

然后我：bin/nutch solrindex http://localhost:8983/solr/ crawl-20110804191414/crawldb crawl-20110804191414/linkdb crawl-20110804191414/segments/*

这给了我：

SolrIndexer: starting at 2011-08-04 19:17:07
SolrIndexer: finished at 2011-08-04 19:17:08, elapsed: 00:00:01

当我对 solr 进行 : 查询时，我得到：

<response>
     <lst name="responseHeader">
          <int name="status">0</int>
          <int name="QTime">2</int>
          <lst name="params">
               <str name="indent">on</str>
               <str name="start">0</str>
               <str name="q">*:*</str>
               <str name="version">2.2</str>
               <str name="rows">10</str>
          </lst>
     </lst>
     <result name="response" numFound="0" start="0"/>
</response>

请注意，当我尝试使用 protocol-http 来抓取网站时，这可以正常工作，但在我使用 protocol-file 抓取文件系统时不起作用。

---编辑--- 今天再次尝试后，我注意到名称中带有空格的文件导致 404 错误。我正在索引的共享中有很多文件。但是，thumbs.db 文件正常。这告诉我问题不是我想的那样。

【问题讨论】：

我还做了一个段转储，发现 pdf 文本内容正在被索引，这很好，因为这正是我需要的。我不明白为什么 solr 没有更新所有数据。
我还尝试索引一个重命名为一个单词的单个 pdf 文件。段数据在那里，文本被解析出来，但在执行 bin/nutch solrindex 后，solr 中没有显示搜索结果...
仍然无法解决这个问题。我已经就这个问题向 Apache 提出了一个问题。似乎至少分配了一名开发人员：issues.apache.org/jira/browse/NUTCH-1076

标签： solr nutch

【解决方案1】：

我今天花了很多时间回顾你的脚步。我最终求助于 /opt/nutch/src/java/org/apache/nutch/indexer/IndexerMapReduce.java 中的 printf 调试，这表明我尝试索引的每个 URL 出现了两次，一次以 file:// 开头/var/www/Engineering/，正如我最初指定的那样，并且从 file:/u/u60/Engineering/ 开始。在这个系统上，/var/www/Engineering 是 /u/u60/Engineering 的符号链接。此外，/var/www/Engineering URL 因未提供 parseText 字段而被拒绝，而 /u/u60/Engineering URL 因未提供 fetchDatum 字段而被拒绝。在 /u/u60/Engineering 表单中指定原始 URL 解决了我的问题。希望在这种情况下对下一个 sap 有所帮助。

【讨论】：

【解决方案2】：

这是因为 solr 没有获取要索引的数据。好像您没有正确执行前面的命令。重新启动整个过程，然后尝试最后一个命令。从这里复制命令：https://wiki.apache.org/nutch/NutchTutorial 或在 youtube 上参考我的视频--https://www.youtube.com/watch?v=aEap3B3M-PU&t=449s

【讨论】：