Nutch 未知过滤器和归一化答案

【问题标题】：Nutch Unknown Filter and NormalizationNutch 未知过滤器和归一化
【发布时间】：2013-12-18 23:13:35
【问题描述】：

我有一个 URL 列表，我想测试使用 Nutch 抓取...特别是那个 URL 列表并且没有抓取..

我指的是 post 禁用抓取..

我注意到我的 5 个测试 URL 经过规范化和过滤后结果为 0。

$:~/apache-nutch-1.7$ bin/nutch crawl urls -dir crawl -depth 3 -topN 1000
solrUrl is not set, indexing will be skipped...
crawl started in: crawl
rootUrlDir = urls
threads = 10
depth = 3
solrUrl=null
topN = 1000
Injector: starting at 2013-12-18 23:07:32
Injector: crawlDb: crawl/crawldb
Injector: urlDir: urls
Injector: Converting injected urls to crawl db entries.
Injector: total number of urls rejected by filters: 5
Injector: total number of urls injected after normalization and filtering: 0
Injector: Merging injected urls into crawl db.
Injector: finished at 2013-12-18 23:07:39, elapsed: 00:00:06
Generator: starting at 2013-12-18 23:07:39
Generator: Selecting best-scoring urls due for fetch.
Generator: filtering: true
Generator: normalizing: true
Generator: topN: 1000
Generator: jobtracker is 'local', generating exactly one partition.
Generator: 0 records selected for fetching, exiting ...
Stopping at depth=0 - no more URLs to fetch.
No URLs to fetch - check your seed list and URL filters.
crawl finished: crawl

实际上我将过滤器和规范化保留为默认值，我猜它不会过滤任何东西..

谁能帮我理解发生了什么？

Injector: total number of urls rejected by filters: 5

谁能告诉我应该更改哪个配置文件以删除上面一行中提到的“过滤器”

我的测试网址也是这样的：

http://example.com/store/em?action=products&cat=1&catalogId=500201&No=0
http://example.com/store/em?action=products&cat=1&catalogId=500201&No=25
http://example.com/store/em?action=products&cat=1&catalogId=500201&No=50
http://example.com/store/em?action=products&cat=1&catalogId=500201&No=75
http://example.com/store/em?action=products&cat=1&catalogId=500201&No=100

【问题讨论】：

标签： regex web-scraping web-crawler nutch

【解决方案1】：

http://svn.apache.org/repos/asf/nutch/tags/release-2.2.1/conf/

文件是regex-urlfilter.txt（复制模板）。看看那里的正则表达式。具体如下：

# skip URLs containing certain characters as probable queries, etc.
-[?*!@=]

考虑到它们是“查询等”，这肯定会过滤掉您的网址。

【讨论】：