Scrapy：Linkextractor 规则不起作用

Question

我已经尝试了 3 种不同的 LinkExtractor 变体，但它仍然忽略“拒绝”规则并在所有 3 种变体中抓取子域......我想从抓取中排除子域。

仅尝试使用“允许”规则。只允许主域，即example.edu.uk

rules = [Rule(LinkExtractor(allow=(r'^example\.edu.uk(\/.*)?$',)))] // Not Working

仅尝试使用“拒绝”规则。拒绝所有子域，即 sub.example.edu.uk

rules = [Rule(LinkExtractor(deny=(r'(?<=\.)[a-z0-9-]*\.edu\.uk',)))] // Not Working

尝试使用“允许和拒绝”规则

rules = [Rule(LinkExtractor(allow=(r'^http:\/\/example\.edu\.uk(\/.*)?$'),deny=(r'(?<=\.)[a-z0-9-]*\.edu\.uk',)))] // Not Working

示例：

点击这些链接

• example.edu.uk/fsdfs.htm
• example.edu.uk/nkln.htm
• example.edu.uk/vefr.htm
• example.edu.uk/opji.htm

丢弃子域链接

• 子域.example.edu.uk/fsdfs.htm
• 子域.example.edu.uk/nkln.htm
• 子域.example.edu.uk/vefr.htm
• 子域.example.edu.uk/opji.htm

这是完整的代码...

class NewsFields(Item):
    pagetype = Field()
    pagetitle = Field()
    pageurl = Field()
    pagedate = Field()
    pagedescription = Field()
    bodytext = Field()

class MySpider(CrawlSpider):
    name = 'profiles'
    start_urls = ['http://www.example.edu.uk/listing']
    allowed_domains = ['example.edu.uk']
    rules = (Rule(LinkExtractor(allow=(r'^https?://example.edu.uk/.*', ))), )
    def parse(self, response):
        hxs = Selector(response)
        soup = BeautifulSoup(response.body, 'lxml')
        nf = NewsFields()
        ptype = soup.find_all(attrs={"name":"nkdpagetype"})
        ptitle = soup.find_all(attrs={"name":"nkdpagetitle"})
        pturl = soup.find_all(attrs={"name":"nkdpageurl"})
        ptdate = soup.find_all(attrs={"name":"nkdpagedate"})
        ptdesc = soup.find_all(attrs={"name":"nkdpagedescription"})
        for node in soup.find_all("div", id="main-content__wrapper"):
             ptbody = ''.join(node.find_all(text=True))
             ptbody = ' '.join(ptbody.split())
             nf['pagetype'] = ptype[0]['content'].encode('ascii', 'ignore')
             nf['pagetitle'] = ptitle[0]['content'].encode('ascii', 'ignore')
             nf['pageurl'] = pturl[0]['content'].encode('ascii', 'ignore')
             nf['pagedate'] = ptdate[0]['content'].encode('ascii', 'ignore')
             nf['pagedescription'] = ptdesc[0]['content'].encode('ascii', 'ignore')
             nf['bodytext'] = ptbody.encode('ascii', 'ignore')
             yield nf
            for url in hxs.xpath('//p/a/@href').extract():
             yield Request(response.urljoin(url), callback=self.parse)

有人可以帮忙吗？ 谢谢

Answer 1

你的前两条规则是错误的

rules = [Rule(LinkExtractor(allow=(r'^example\.edu.uk(\/.*)?$',)))] // Not Working
rules = [Rule(LinkExtractor(deny=(r'(?<=\.)[a-z0-9-]*\.edu\.uk',)))] // Not Working

允许和拒绝是针对绝对 url 而不是域。以下应该适合你

rules = (Rule(LinkExtractor(allow=(r'^https?://example.edu.uk/.*', ))), )

Edit-1

首先你应该在下面改变

allowed_domains = ['example.edu.uk']

到

allowed_domains = ['www.example.edu.uk']

你的第二个URL提取规则应该是

rules = (Rule(LinkExtractor(allow=(r'^https?://www.example.edu.uk/.*', ))), )

第三，在你下面的代码中

for url in hxs.xpath('//p/a/@href').extract():
         yield Request(response.urljoin(url), callback=self.parse)

规则将不适用。您的收益受规则约束。规则将自动插入新请求，但它们不会阻止您生成规则配置不允许的其他链接。但是设置allowed_domains 将适用于规则和您的收益