尝试使用 Scrapy 抓取 LinkedIn 时的 999 响应答案

【问题标题】：999 response when trying to crawl LinkedIn with Scrapy尝试使用 Scrapy 抓取 LinkedIn 时的 999 响应
【发布时间】：2017-03-20 17:33:40
【问题描述】：

我正在尝试使用 Scrapy 框架从 LinkedIn 中提取一些信息。我知道他们对试图爬取他们网站的人非常严格，所以我在我的 settings.py 中尝试了不同的用户代理。我还指定了很高的下载延迟，但它似乎仍然阻止了我。

USER_AGENT = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:39.0) Gecko/20100101 Firefox/39.0'
ROBOTSTXT_OBEY = False
DOWNLOAD_DELAY = 2
REDIRECT_ENABLED = False
RETRY_ENABLED = False
DEPTH_LIMIT = 5
DOWNLOAD_TIMEOUT = 10
REACTOR_THREADPOOL_MAXSIZE = 20
CONCURRENT_REQUESTS_PER_DOMAIN = 2
COOKIES_ENABLED = False
HTTPCACHE_ENABLED = True

这是我收到的错误：

2017-03-20 19:11:29 [scrapy.core.engine] INFO: Spider opened
2017-03-20 19:11:29 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min),
scraped 0 items (at 0 items/min)
2017-03-20 19:11:29 [scrapy.extensions.telnet] DEBUG: Telnet console listening on
127.0.0.1:6023
2017-03-20 19:11:29 [scrapy.core.engine] DEBUG: Crawled (999) <GET
https://www.linkedin.com/directory/people-1/> (referer: None) ['cached']
2017-03-20 19:11:29 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response
<999 https://www.linkedin.com/directory/people-1/>: HTTP status code is not handled or 
not allowed
2017-03-20 19:11:29 [scrapy.core.engine] INFO: Closing spider (finished)
2017-03-20 19:11:29 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 282,
'downloader/request_count': 1,
'downloader/request_method_count/GET': 1,
'downloader/response_bytes': 2372,
'downloader/response_count': 1,
'downloader/response_status_count/999': 1,
'finish_reason': 'finished',
'finish_time': datetime.datetime(2017, 3, 20, 17, 11, 29, 503000),
'httpcache/hit': 1,
'log_count/DEBUG': 2,
'log_count/INFO': 8,
'response_received_count': 1,
'scheduler/dequeued': 1,
'scheduler/dequeued/memory': 1,
'scheduler/enqueued': 1,
'scheduler/enqueued/memory': 1,
'start_time': datetime.datetime(2017, 3, 20, 17, 11, 29, 378000)}
2017-03-20 19:11:29 [scrapy.core.engine] INFO: Spider closed (finished)

蜘蛛本身只是打印访问过的 url。

class InfoSpider(CrawlSpider):
    name = "info"
    allowed_domains = ["www.linkedin.com"]
    start_urls = ['https://www.linkedin.com/directory/people-1/']
    rules = [
        Rule(LinkExtractor(
            allow=[r'.*']),
            callback='parse',
            follow=True)
    ]
    def parse(self, response):
        print(response.url)

【问题讨论】：

问题是什么？
你也在抓取个人数据吗？忽略服务条款是一回事 - 我个人认为它们不一定具有约束力 - 但在未经受影响个人许可的情况下获取和重复使用个人数据则是另一回事。
感谢您的回复。我不会使用任何数据。我只是在学习 Scrapy，并试图了解它在这样的场景中是如何工作的。
你有没有让这个和scrapy一起工作？如果是，您能否更新您的问题并添加解决方案？

标签： python web-scraping scrapy

【解决方案1】：

请注意请求中的标头。 LinkedIn 需要在每个请求中包含以下标头来提供响应。

headers = {
    "accept" : "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8",
    "accept-encoding" : "gzip, deflate, sdch, br",
    "accept-language" : "en-US,en;q=0.8,ms;q=0.6",
    "user-agent" : "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36"
}

您可以参考this documentation了解更多信息。

【讨论】：

感谢您的回答。不过，即使我使用您提供的标头，我仍然会收到 999 响应。在 Scrapy 外壳中：>>> headers = { ... } >>> req = Request('linkedin.com/directory/people-1/', headers=headers) >>> fetch(req) {...} 2017-03-20 21:00:20 [scrapy.core.engine] DEBUG: Crawled (999)
它非常适合我。虽然我使用 lxml 来测试它。只有user-agent 作为标题，我也得到了 999 状态码。

【解决方案2】：

在抓取任何其他页面之前，您必须先登录linkedin。使用scrapy登录，可以参考https://doc.scrapy.org/en/latest/topics/request-response.html#formrequest-objects

更新 1：这是我的代码示例。

from scrapy.contrib.spiders import CrawlSpider, Rule
from scrapy.contrib.spiders.init import InitSpider
from scrapy.contrib.linkextractors.sgml import SgmlLinkExtractor
from scrapy.selector import HtmlXPathSelector
from scrapy.http import Request, FormRequest
from linkedin.items import *

class LinkedinSpider(InitSpider):

"""
Define the crawler's start URIs, set its follow rules, parse HTML
and assign values to an item. Processing occurs in ../pipelines.py
"""

name = "linkedin"
allowed_domains = ["linkedin.com"]
user_name = 'my_user_name'
passwd = 'my_passwd'

# Uncomment the following lines for full spidering
# start_urls = ["http://www.linkedin.com/directory/people-%s-%d-%d-%d"
#               % (alphanum, num_one, num_two, num_three)
#                 for alphanum in "abcdefghijklmnopqrstuvwxyz"
#                 for num_one in xrange(1,11)
#                 for num_two in xrange(1,11)
#                 for num_three in xrange(1,11)
#               ]

# Temporary start_urls for testing; remove and use the above start_urls in production
# start_urls = ["http://www.linkedin.com/directory/people-a-23-23-2"]
start_urls = ["https://www.linkedin.com/in/rebecca-liu-93a12a28/"]
login_page = 'https://www.linkedin.com/uas/login'
# TODO: allow /in/name urls too?
# rules = (
#     Rule(SgmlLinkExtractor(allow=('\/pub\/.+')),
#          callback='parse_item'),
# )

def init_request(self):
    return Request(url=self.login_page,callback=self.login)

def login(self,response):
    return FormRequest.from_response(response,formdata={
        'session_key':user_name,'session_password':passwd
    },
                                     callback = self.check_login_response)

def check_login_response(self,response):
    return self.initialized()

如果您使用 InitSpider 或 parse()，请记住调用 self.initialized() > 方法不会被调用。

【讨论】：