如何使用带有文本列表的scrapy答案

【问题标题】：how to use scrapy with text list如何使用带有文本列表的scrapy
【发布时间】：2019-11-05 17:56:23
【问题描述】：

大家好，我在新项目中使用 scrappy 将 ip 转换为域名

我找不到如何在 scrappy 中的起始 url 上添加列表文本 (ip.txt)，用文本列表替换 (+ IP)

示例：

start_urls = [
    `"https://api.hackertarget.com/reverseiplookup/?q= + ip"`]

------------------------------------我的代码---------- -------------------

# -*- coding: utf-8 -*-
import scrapy

lists = open(raw_input('IP list file name: '), 'r').read().split('\n')

class jeffbullasSpider(scrapy.Spider):
    name = "iptohost"
    allowed_domains = ["api.hackertarget.com"]
    start_urls = [
    "https://api.hackertarget.com/reverseiplookup/?q=" + str(lists) ] 

    def parse(self, response):
       print response.xpath('//body//text()').get()

（我是python新手，非常感谢你。）

【问题讨论】：

# -- coding: utf-8 -- import scrapy lists = open(raw_input('IP list file name: '), 'r').read() .split('\n') 类 jeffbullasSpider(scrapy.Spider): name = "iptohost" allowed_domains = ["api.hackertarget.com"] start_urls = [ "api.hackertarget.com/reverseiplookup/?q=" + str(lists) ] def parse(self , response): print response.xpath('//body//text()').get()

标签： python web-scraping scrapy python-requests

【解决方案1】：

试试这个：

编辑：在发送请求之前也剥离 ip

import scrapy

lists = open(raw_input('IP list file name: '), 'r').read().split('\n')

class jeffbullasSpider(scrapy.Spider):
    name = "iptohost"
    allowed_domains = ["api.hackertarget.com"]
    url = "https://api.hackertarget.com/reverseiplookup/?q={}"

    def start_requests(self):
        for ip in lists:
            yield scrapy.Request(url=self.url.format(ip.strip()), callback=self.parse)

    def parse(self, response):
       print(response.xpath('//body//text()').get())

【讨论】：

不客气！如果它是你要找的，请批准答案:) 你可以查看 Scrapys 教程docs.scrapy.org/en/latest/intro/tutorial.html

【解决方案2】：

我还有其他问题，我在scrapy上添加了代理轮换，在我用这个命令保存之后：

scrapy crawl iptohost -o some.json -t json &> some.text

我的结果不只包含我的域包含代理结果和我的结果域

我的结果

    2019-11-10 10:39:50 [rotating_proxies.expire] DEBUG: Proxy <http://197.157.219.25:8080> is DEAD
2019-11-10 10:39:50 [rotating_proxies.middlewares] DEBUG: Retrying <GET https://api.hackertarget.com/reverseiplookup/?q=61.112.2.178> with another proxy (failed 4 times, max retries: 5)
2019-11-10 10:39:50 [rotating_proxies.expire] DEBUG: Proxy <http://139.59.99.119:8080> is DEAD
2019-11-10 10:39:50 [rotating_proxies.middlewares] DEBUG: Retrying <GET https://api.hackertarget.com/reverseiplookup/?q=195.11.184.130> with another proxy (failed 5 times, max retries: 5)
2019-11-10 10:39:50 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://api.hackertarget.com/reverseiplookup/?q=195.11.184.130> (referer: None)
2019-11-10 10:39:50 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://api.hackertarget.com/reverseiplookup/?q=185.179.235.40> (referer: None)
[u'capitalinstant.com']
[u'142.92.242.192']
[u'API count exceeded - Increase Quota with Membership']
[u'API count exceeded - Increase Quota with Membership']
[u'API count exceeded - Increase Quota with Membership']

如何删除代理结果以仅抓取我的域结果恳求非常感谢您 =)

【讨论】：