【目标】要完成的任务如下：
※ 创建一个 Scrap项目。
※ 创建一个 Spider来抓取站点和处理数据。
※ 通过命令行将抓取的内容导出。
※ 将抓取的内容保存的到 MongoDB数据库。
==============================================
【准备工作】需要安装好 Scrapy框架、 MongoDB和 PyMongo库
1.创建项目：
【操作】在想创建项目的目录按：shift+右键——在此处打开命令窗口（或 在cmd里cd进入想要的目录）输入CMD命令（此处不用test，因为系统已经有一个此名字，将报错提示已存在）：
注意：在linux 系统内如提示权限问题，则在命令前加 sudo ~
scrapy startproject tutorial # 生成项目命令：最后跟的参数tutorial是项目名，也将是文件夹名
【说明】生成的目录结构如下：
tutorial # 项目总文件夹 tutorial # tutorial文件夹：保存的是要引用的模块； crapy.cfg # cfg文件：是scrapy部署时的配置文件；
【说明】../tutorial/tutorial 目录下文件及作用：
tutorial
__init__.py items.py # Items的定义,定义爬取的数据结构 middlewares.py # Middlewares的定义,定义爬取时的中间件 pipelines. py # Pipelines的定义,定义数据管道 settings.py # 配置文件 spiders # 放置 Spiders的文件夹 __init__.py
2.创建 Spider
【操作】Spider是自己定义的类, Scrapy用它来从网页里抓取内容,并解析抓取的结果。不过这个类必须继承 Scrapy提供的 Spider类 scrapy. Spider,还要定义Spider的名称和起始请求,以及怎样处理爬取后的结果的方法。也可以使用命令行创建一个 Spider。比如要生成 quotes这个 Spider,可以执行如下cmd命令
【命令解释】进入刚才创建的tutorial文件夹，然后执行 genspider命令：第一个参数是 Spider的名称，第二个参数是网站域名：
cd tutorial
scrapy genspider quotes quotes.toscrape.com
【说明】执行完毕之后, spiders文件夹中多了一个 quotes. py（目录：E:\a\scrapy_test\tutorial\tutorial\spiders）,它就是刚刚创建的 Spider,内容如下(其网址，名仅作表示，真代码见4步)：
# -*- coding: utf-8 -*- import scrapy class QuotesSpider(scrapy.Spider): name = \'baidu\' allowed_domains = [\'fanyi.baidu.com\\'] start_urls = [\'http://fanyi.baidu.com\\'] def parse(self, response): pass
【说明】这里有三个属性—name、 allowed domains和 start urls,还有一个方法 parse：
★name：它是每个项目唯一的名字,用来区分不同的 Spider
★allowed domains：它是允许爬取的域名,如果初始或后续的请求链接不是这个域名下的,则请求链接会被过滤掉。
★start urls：它包含了 Spider在启动时爬取的url列表,初始请求是由它来定义的。
★parse：它是 Spider的一个方法。默认情况下,被调用时 start urls里面的链接构成的请求完成下载执行后,返回的响应就会作为唯一的参数传递给这个函数。该方法负责解析返回的响应、提取数据或者进一步生成要处理的请求。
3.创建ltem
【说明】Item是保存爬取数据的容器,它的使用方法和字典类似。不过,相比字典,ltem多了额外的保护机制,可以避免拼写错误或者定义字段错误。
创建Iem需要继承 scrapy.Item类,并且定义类型为 scrapy. Field的字段。
【操作】定义Item,观察目标网站,我们可以获取到到内容有text、author、tags。此时将 Items.py修改如下:
import scrapy class QuoteItem(scrapy.Item) text= scrapy. Field() author= scrapy. Field() tags= scrapy. Field() #这里定义了三个字段,接下来爬取时我们会使用到这个Item。
4.解析response
【说明】quotes.py中我们看到, parse()方法的参数 respose是 start urls里面的链接爬取后的结果。所以在parse()方法中,我们可以直接对 response变量包含的内容进行解析,比如浏览请求结果的网页源代码,或者进一步分析源代码内容,或者找出结果中的链接而得到下一个请求。
【网页分析】我们可以看到网页中既有我们想要的结果,又有下一页的链接,这两部分内容我们都要进行处理。首先看看网页结构,如图13-2所示。每一页都有多个class为 quote的区块,每个区块内都包含text、 author、tags。那么我们先找出所有的 quote,然后提取每一个 quote中的内容。
quotes.py写成如下：
# -*- coding: utf-8 -*- import scrapy class QuotesSpider(scrapy.Spider): name = \'quotes\' allowed_domains = [\'quotes.toscrape.com\'] start_urls = [\'http://quotes.toscrape.com/\'] def parse(self, response): quotes=response.css(\'.quote\') for quote in quotes: text=quote.css(\'.text::text\').extract_first() author=quote.css(\'.author::text\').extract_first() tags=quote.css(\'.tags .tag::text\').extract()
#或<meta class="keywords" itemprop="keywords" content="change,deep-thoughts,thinking,world">
tags=quote.css(\'.keywords::attr(content)\').extract_first()
5.使用Item
【说明】上文定义了Item,接下来就要使用它了。Item可以理解为一个字典,不过在声明的时候需要实例化。然后依次用刚才解析的结果赋值Item的每一个字段,最后将Item返回即可。
Quotes Spider的改写quotes.py如下所示:
# -*- coding: utf-8 -*- import scrapy from tutorial.items import QuoteItem class QuotesSpider(scrapy.Spider): name = "quotes" allowed_domains = ["quotes.toscrape.com"] start_urls = [\'http://quotes.toscrape.com/\'] def parse(self, response): quotes = response.css(\'.quote\') for quote in quotes: item = QuoteItem() item[\'text\'] = quote.css(\'.text::text\').extract_first() item[\'author\'] = quote.css(\'.author::text\').extract_first() item[\'tags\'] = quote.css(\'.tags .tag::text\').extract() yield item
【代码解释】如此，所有首页内容被解析出来，赋值成一个个QuotesItem
6.后续Request（生成下一页请求）
【说明】抓取下一页内容：从当前页找到一个信息，生成下一页请求，从而往复迭代，抓取整站
【分析】找到下一页面按钮分析源码：
【方法说明】构造请求需用到：scrapy.Request(url=url, callback=self.parse)
参数1：url是请求的链接；
参数2：callback是回调函数。当指定了该回调函数的请求完成之后,获取到响应,引擎会将该响应作为参数传递给这个回调函数。回调函数进行解析或生成下一个请求,回调函数即刚写的 parse()函数。
由于 parse()就是解析text、 author、tags的方法,而下一页的结构和刚才已经解析的页面结构是一样的,所以我们可以再次使用 parse()方法来做页面解析。
【操作】写的下一页代码如下：
next = response.css(\'.pager .next a::attr(href)\').extract_first() url = response.urljoin(next) yield scrapy.Request(url=url, callback=self.parse)
【源码说明】：
第1行首先通过CSS选择器获取下一个页面的链接,即要获取a超链接中的href属性。这里用到了:attr(href)操作。然后再调用 extract first()方法获取内容。
第2行调用了 urljoin()方法, urljoin()方法可以将相对URL构造成一个绝对的URL。例如,获取到的下一页地址是page2, unjoint()方法处理后得到的结果就是:htp/ quotes.toscrape. com/page/2/
第3行通过ur1和 callback变量构造了一个新的请求,回调函数 callback依然使用 parse()方法。这个请求完成后,响应会重新经过 parse方法处理,得到第2页的解析结果,然后生成第2页的下一页,即第3页的请求
这样爬虫就进入了一个循环,直到最后一页。通过几行代码,我们就轻松实现了一个抓取循环,将每个页面的结果抓取下来了
【quotes.py 最终代码】如下：
# -*- coding: utf-8 -*- import scrapy from tutorial.items import QuoteItem class QuotesSpider(scrapy.Spider): name = "quotes" allowed_domains = ["quotes.toscrape.com"] start_urls = [\'http://quotes.toscrape.com/\'] def parse(self, response): quotes = response.css(\'.quote\') for quote in quotes: item = QuoteItem() item[\'text\'] = quote.css(\'.text::text\').extract_first() item[\'author\'] = quote.css(\'.author::text\').extract_first() item[\'tags\'] = quote.css(\'.tags .tag::text\').extract() yield item next = response.css(\'.pager .next a::attr(href)\').extract_first() url = response.urljoin(next) yield scrapy.Request(url=url, callback=self.parse)
7.运行爬虫
【运行前】记得把 settings.py中的，顺从爬虫规则去年注释，并改成False，否则，默认为Ture，可能会过滤掉许多网址
# Obey robots.txt rules，默认为Ture，即按网站规则来，把有些网址过滤掉 ROBOTSTXT_OBEY = False
【运行爬虫】进入项目目录（E:\a\scrapy_test\tutorial），运行命令即可运行爬虫，进行爬取：
scrapy crawl quotes
【结果】如下：
scrapy crawl quotes \'tags\': [\'friendship\', \'love\'], \'text\': \'“There is nothing I would not do for those who are...\'} 2019-04-26 13:53:50 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes .toscrape.com/page/6/> {\'author\': \'Eleanor Roosevelt\', \'tags\': [\'attributed\', \'fear\', \'inspiration\'], \'text\': \'“Do one thing every day that scares you.”\'} 2019-04-26 13:53:50 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes .toscrape.com/page/6/> {\'author\': \'Marilyn Monroe\', \'tags\': [\'attributed-no-source\'], \'text\': \'“I am good, but not an angel. I do sin, but I am n...\'} 2019-04-26 13:53:50 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes .toscrape.com/page/6/> {\'author\': \'Albert Einstein\', \'tags\': [\'music\'], \'text\': \'“If I were not a physicist, I would probably be a...\'} 2019-04-26 13:53:50 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes .toscrape.com/page/6/> {\'author\': \'Haruki Murakami\', \'tags\': [\'books\', \'thought\'], \'text\': \'“If you only read the books that everyone else is...\'} 2019-04-26 13:53:50 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes .toscrape.com/page/6/> {\'author\': \'Alexandre Dumas fils\', \'tags\': [\'misattributed-to-einstein\'], \'text\': \'“The difference between genius and stupidity is: g...\'} 2019-04-26 13:53:50 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes .toscrape.com/page/6/> {\'author\': \'Stephenie Meyer\', \'tags\': [\'drug\', \'romance\', \'simile\'], \'text\': "“He\'s like a drug for you, Bella.”"} 2019-04-26 13:53:50 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes .toscrape.com/page/6/> {\'author\': \'Ernest Hemingway\', \'tags\': [\'books\', \'friends\', \'novelist-quotes\'], \'text\': \'“There is no friend as loyal as a book.”\'} 2019-04-26 13:53:50 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes .toscrape.com/page/6/> {\'author\': \'Helen Keller\', \'tags\': [\'inspirational\'], \'text\': \'“When one door of happiness closes, another opens;...\'} 2019-04-26 13:53:50 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes .toscrape.com/page/6/> {\'author\': \'George Bernard Shaw\', \'tags\': [\'inspirational\', \'life\', \'yourself\'], \'text\': "“Life isn\'t about finding yourself. Life is about..."} 2019-04-26 13:53:50 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://quotes .toscrape.com/page/7/> (referer: http://quotes.toscrape.com/page/6/) 2019-04-26 13:53:50 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes .toscrape.com/page/7/> {\'author\': \'Charles Bukowski\', \'tags\': [\'alcohol\'], \'text\': "“That\'s the problem with drinking, I thought, as I..."} 2019-04-26 13:53:50 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes .toscrape.com/page/7/> {\'author\': \'Suzanne Collins\', \'tags\': [\'the-hunger-games\'], \'text\': \'“You don’t forget the face of the person who was y...\'} 2019-04-26 13:53:50 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes .toscrape.com/page/7/> {\'author\': \'Suzanne Collins\', \'tags\': [\'humor\'], \'text\': "“Remember, we\'re madly in love, so it\'s all right..."} 2019-04-26 13:53:50 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes .toscrape.com/page/7/> {\'author\': \'C.S. Lewis\', \'tags\': [\'love\'], \'text\': \'“To love at all is to be vulnerable. Love anything...\'} 2019-04-26 13:53:50 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes .toscrape.com/page/7/> {\'author\': \'J.R.R. Tolkien\', \'tags\': [\'bilbo\', \'journey\', \'lost\', \'quest\', \'travel\', \'wander\'], \'text\': \'“Not all those who wander are lost.”\'} 2019-04-26 13:53:50 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes .toscrape.com/page/7/> {\'author\': \'J.K. Rowling\', \'tags\': [\'live-death-love\'], \'text\': \'“Do not pity the dead, Harry. Pity the living, and...\'} 2019-04-26 13:53:50 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes .toscrape.com/page/7/> {\'author\': \'Ernest Hemingway\', \'tags\': [\'good\', \'writing\'], \'text\': \'“There is nothing to writing. All you do is sit do...\'} 2019-04-26 13:53:50 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes .toscrape.com/page/7/> {\'author\': \'Ralph Waldo Emerson\', \'tags\': [\'life\', \'regrets\'], \'text\': \'“Finish each day and be done with it. You have don...\'} 2019-04-26 13:53:50 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes .toscrape.com/page/7/> {\'author\': \'Mark Twain\', \'tags\': [\'education\'], \'text\': \'“I have never let my schooling interfere with my e...\'} 2019-04-26 13:53:50 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes .toscrape.com/page/7/> {\'author\': \'Dr. Seuss\', \'tags\': [\'troubles\'], \'text\': \'“I have heard there are troubles of more than one...\'} 2019-04-26 13:53:51 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://quotes .toscrape.com/page/8/> (referer: http://quotes.toscrape.com/page/7/) 2019-04-26 13:53:51 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes .toscrape.com/page/8/> {\'author\': \'Alfred Tennyson\', \'tags\': [\'friendship\', \'love\'], \'text\': \'“If I had a flower for every time I thought of you...\'} 2019-04-26 13:53:51 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes .toscrape.com/page/8/> {\'author\': \'Charles Bukowski\', \'tags\': [\'humor\'], \'text\': \'“Some people never go crazy. What truly horrible l...\'} 2019-04-26 13:53:51 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes .toscrape.com/page/8/> {\'author\': \'Terry Pratchett\', \'tags\': [\'humor\', \'open-mind\', \'thinking\'], \'text\': \'“The trouble with having an open mind, of course,...\'} 2019-04-26 13:53:51 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes .toscrape.com/page/8/> {\'author\': \'Dr. Seuss\', \'tags\': [\'humor\', \'philosophy\'], \'text\': \'“Think left and think right and think low and thin...\'} 2019-04-26 13:53:51 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes .toscrape.com/page/8/> {\'author\': \'J.D. Salinger\', \'tags\': [\'authors\', \'books\', \'literature\', \'reading\', \'writing\'], \'text\': \'“What really knocks me out is a book that, when yo...\'} 2019-04-26 13:53:51 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes .toscrape.com/page/8/> {\'author\': \'George Carlin\', \'tags\': [\'humor\', \'insanity\', \'lies\', \'lying\', \'self-indulgence\', \'truth\'], \'text\': \'“The reason I talk to myself is because I’m the on...\'} 2019-04-26 13:53:51 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes .toscrape.com/page/8/> {\'author\': \'John Lennon\', \'tags\': [\'beatles\', \'connection\', \'dreamers\', \'dreaming\', \'dreams\', \'hope\', \'inspirational\', \'peace\'], \'text\': "“You may say I\'m a dreamer, but I\'m not the only o..."} 2019-04-26 13:53:51 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes .toscrape.com/page/8/> {\'author\': \'W.C. Fields\', \'tags\': [\'humor\', \'sinister\'], \'text\': \'“I am free of all prejudice. I hate everyone equal...\'} 2019-04-26 13:53:51 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes .toscrape.com/page/8/> {\'author\': \'Ayn Rand\', \'tags\': [], \'text\': "“The question isn\'t who is going to let me; it\'s w..."} 2019-04-26 13:53:51 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes .toscrape.com/page/8/> {\'author\': \'Mark Twain\', \'tags\': [\'books\', \'classic\', \'reading\'], \'text\': "“′Classic′ - a book which people praise and don\'t..."} 2019-04-26 13:53:52 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://quotes .toscrape.com/page/9/> (referer: http://quotes.toscrape.com/page/8/) 2019-04-26 13:53:52 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes .toscrape.com/page/9/> {\'author\': \'Albert Einstein\', \'tags\': [\'mistakes\'], \'text\': \'“Anyone who has never made a mistake has never tri...\'} 2019-04-26 13:53:52 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes .toscrape.com/page/9/> {\'author\': \'Jane Austen\', \'tags\': [\'humor\', \'love\', \'romantic\', \'women\'], \'text\': "“A lady\'s imagination is very rapid; it jumps from..."} 2019-04-26 13:53:52 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes .toscrape.com/page/9/> {\'author\': \'J.K. Rowling\', \'tags\': [\'integrity\'], \'text\': \'“Remember, if the time should come when you have t...\'} 2019-04-26 13:53:52 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes .toscrape.com/page/9/> {\'author\': \'Jane Austen\', \'tags\': [\'books\', \'library\', \'reading\'], \'text\': \'“I declare after all there is no enjoyment like re...\'} 2019-04-26 13:53:52 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes .toscrape.com/page/9/> {\'author\': \'Jane Austen\', \'tags\': [\'elizabeth-bennet\', \'jane-austen\'], \'text\': \'“There are few people whom I really love, and stil...\'} 2019-04-26 13:53:52 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes .toscrape.com/page/9/> {\'author\': \'C.S. Lewis\', \'tags\': [\'age\', \'fairytales\', \'growing-up\'], \'text\': \'“Some day you will be old enough to start reading...\'} 2019-04-26 13:53:52 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes .toscrape.com/page/9/> {\'author\': \'C.S. Lewis\', \'tags\': [\'god\'], \'text\': \'“We are not necessarily doubting that God will do...\'} 2019-04-26 13:53:52 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes .toscrape.com/page/9/> {\'author\': \'Mark Twain\', \'tags\': [\'death\', \'life\'], \'text\': \'“The fear of death follows from the fear of life....\'} 2019-04-26 13:53:52 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes .toscrape.com/page/9/> {\'author\': \'Mark Twain\', \'tags\': [\'misattributed-mark-twain\', \'truth\'], \'text\': \'“A lie can travel half way around the world while...\'} 2019-04-26 13:53:52 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes .toscrape.com/page/9/> {\'author\': \'C.S. Lewis\', \'tags\': [\'christianity\', \'faith\', \'religion\', \'sun\'], \'text\': \'“I believe in Christianity as I believe that the s...\'} 2019-04-26 13:53:52 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://quotes .toscrape.com/page/10/> (referer: http://quotes.toscrape.com/page/9/) 2019-04-26 13:53:52 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes .toscrape.com/page/10/> {\'author\': \'J.K. Rowling\', \'tags\': [\'truth\'], \'text\': \'“The truth." Dumbledore sighed. "It is a beautiful...\'} 2019-04-26 13:53:52 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes .toscrape.com/page/10/> {\'author\': \'Jimi Hendrix\', \'tags\': [\'death\', \'life\'], \'text\': "“I\'m the one that\'s got to die when it\'s time for..."} 2019-04-26 13:53:52 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes .toscrape.com/page/10/> {\'author\': \'J.M. Barrie\', \'tags\': [\'adventure\', \'love\'], \'text\': \'“To die will be an awfully big adventure.”\'} 2019-04-26 13:53:52 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes .toscrape.com/page/10/> {\'author\': \'E.E. Cummings\', \'tags\': [\'courage\'], \'text\': \'“It takes courage to grow up and become who you re...\'} 2019-04-26 13:53:52 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes .toscrape.com/page/10/> {\'author\': \'Khaled Hosseini\', \'tags\': [\'life\'], \'text\': \'“But better to get hurt by the truth than comforte...\'} 2019-04-26 13:53:52 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes .toscrape.com/page/10/> {\'author\': \'Harper Lee\', \'tags\': [\'better-life-empathy\'], \'text\': \'“You never really understand a person until you co...\'} 2019-04-26 13:53:52 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes .toscrape.com/page/10/> {\'author\': "Madeleine L\'Engle", \'tags\': [\'books\', \'children\', \'difficult\', \'grown-ups\', \'write\', \'writers\', \'writing\'], \'text\': \'“You have to write the book that wants to be writt...\'} 2019-04-26 13:53:52 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes .toscrape.com/page/10/> {\'author\': \'Mark Twain\', \'tags\': [\'truth\'], \'text\': \'“Never tell the truth to people who are not worthy...\'} 2019-04-26 13:53:52 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes .toscrape.com/page/10/> {\'author\': \'Dr. Seuss\', \'tags\': [\'inspirational\'], \'text\': "“A person\'s a person, no matter how small.”"} 2019-04-26 13:53:52 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes .toscrape.com/page/10/> {\'author\': \'George R.R. Martin\', \'tags\': [\'books\', \'mind\'], \'text\': \'“... a mind needs books as a sword needs a whetsto...\'} 2019-04-26 13:53:52 [scrapy.dupefilters] DEBUG: Filtered duplicate request: <GET http://quotes.toscrape.com/page/10/> - no more duplicates will be shown (see DU PEFILTER_DEBUG to show all duplicates) 2019-04-26 13:53:52 [scrapy.core.engine] INFO: Closing spider (finished) 2019-04-26 13:53:52 [scrapy.statscollectors] INFO: Dumping Scrapy stats: {\'downloader/request_bytes\': 2870, \'downloader/request_count\': 11, \'downloader/request_method_count/GET\': 11, \'downloader/response_bytes\': 24812, \'downloader/response_count\': 11, \'downloader/response_status_count/200\': 10, \'downloader/response_status_count/404\': 1, \'dupefilter/filtered\': 1, \'finish_reason\': \'finished\', \'finish_time\': datetime.datetime(2019, 4, 26, 5, 53, 52, 939800), \'item_scraped_count\': 100, \'log_count/DEBUG\': 112, \'log_count/INFO\': 9, \'request_depth_max\': 10, \'response_received_count\': 11, \'robotstxt/request_count\': 1, \'robotstxt/response_count\': 1, \'robotstxt/response_status_count/404\': 1, \'scheduler/dequeued\': 10, \'scheduler/dequeued/memory\': 10, \'scheduler/enqueued\': 10, \'scheduler/enqueued/memory\': 10, \'start_time\': datetime.datetime(2019, 4, 26, 5, 53, 38, 767800)} 2019-04-26 13:53:52 [scrapy.core.engine] INFO: Spider closed (finished) C:\Users\Administrator\Desktop\py\scrapytutorial>
【解释】结果：
a) 首先Scrap输出了当前的版本号以及正在启动的项目名称。
b) 接着输出了当前 settings. py中一些重写后的配置。
c) 然后输出了当前所应用的 Middlewares和 Pipelines Middlewares默认是启用的,可以在 settings.py中修改。
d) Pipelines默认是空,同样也可以在 settings. py中配置。后面会对它们进行讲解。
e) 接下来就是输出各个页面的抓取结果了,可以看到爬虫一边解析,一边翻页,直至将所有内容抓取完毕,然后终止。
f) 最后, Scrap输出了整个抓取过程的统计信息,如请求的字节数、请求次数、响应次数、完成原因等。
整个 Scrap程序成功运行。我们通过非常简单的代码就完成了一个网站内容的爬取,这样相比之前一点点写程序简洁很多。
8.1保存到文件json csv xml pickle marshal
【说明】运行完 Scrap后,我们只在控制台看到了输出结果。如果想保存结果可直接用, Scrap提供的【 Feed Exports】轻松将抓取结果输出。
【操作a】 将上面的结果【保存成JSON文件】,可以执行如下命令：
#CMD命令，完成后，目录下多出一个quotes.json文件即结果 scrapy crawl quotes -o quotes.json
【操作b】还可以【每一个Item输出一行】JSON,输出后缀为jl,为 jsonline的缩写,命令如下所示:
#cmd命令：每个item导出为一行的joson文件。写法1 scrapy crawl quotes -o quotes.jl #-----或-------： #cmd命令：每个item导出为一行的joson文件.写法2 scrapy crawl quotes -o quotes.jsonlines
【操作c】其它格式输出：
1.输出格式还支持很多种,例如csv、xml、 pickle、 marshal等,还支持仰、s3等远程输出；
2.另外还可以通过自定义 ItemExporter来实现其他的输出。
【操作】下面命令对应的输出分别为【csv、xml、 pickle、 marshal格式、ftp远程输出】：
#cmd命令：其它格式输出 scrapy crawl quotes -o quotes.csv scrapy crawl quotes -o quotes.xml scrapy crawl quotes -o quotes.pickle scrapy crawl quotes -o quotes.marshal #ftp输出需要正确配置用户名、密码、地址、输岀路径,否则会报错。其它ftp文件格式参考上一段 scrapy crawl quotes -o ftp://user:pass@ftp.example.com/path/to/quotes.csv
8.2其它复杂格式输出（用Pipline.py 输出到数据库）
【引言】通过 Scrap提供的 Feed Exports,我们可以轻松地输出抓取结果到文件。对于一些小型项目来说,这应该足够了。如想要更复杂的输出,如输出到数据库等,则可用 Item Pipeline来完成
【目标】如果想进行更复杂的操作,如【将结果保存到 MongoDB数据库】,或者筛选某些有用的Iem,则我们可以定义 Item Pipeline来实现
【操作说明】Item Pipeline为项目管道。当Item生成后,它会自动被送到 Item Pipeline进行处理,我们【常用ltem Pipeline来做如下操作】：
§ 清理HTML数据。
§ 验证爬取数据,检查爬取字段。
§ 查重并丢弃重复内容。
§ 将爬取结果保存到数据库。
【用到的方法】很简单,只需要定义一个类并实现 process_item()方法即可。启用 Item Pipeline后, Item Pipeline会自动调用这个方法：
process_item()方法必须返回包含数据的字典或Item对象 或 者抛出 DropItem异常。
process_item()方法有两个参数。一个参数是item,每次 Spider生成的Item都会作为参数传递过来。
另一个参数是 spider,就是 Spider的实例。
【实操】接下来,我们实现一个 Item Pipeline,筛掉text长度大于50的Item,并将结果保存到 MongoDB：
【步骤1】修改项目里的 [ pipelines.py ] 文件,之前用命令行自动生成的文件内容可以删掉,增加一个TextPipeline类,内容如下所示：
import pymongo from scrapy.exceptions import DropItem #新写部分 class TextPipeline(object): def __init__(self): self.limit = 50 def process_item(self, item, spider): if item[\'text\']: if len(item[\'text\']) > self.limit: item[\'text\'] = item[\'text\'][0:self.limit].rstrip() + \'...\' return item else: return DropItem(\'Missing Text\')
【代码说明】：这段代码在构造方法里定义了限制长度为50,实现了 process item()方法,其参数是item和spider。首先该方法判断item的text属性是否存在,如果不存在,则抛出 DropItem异常;如果存在,再判断长度是否大于50,如果大于,那就截断然后拼接省略号,再将item返回即可。
【步骤2】接下来,我们将处理后的【item存入 MongoDB】,定义另外一个 Pipeline。同样在 pipelines. py中,我们实现另一个类 MongoPipeline,内容如下所示:
class MongoPipeline(object): def __init__(self, mongo_uri, mongo_db): self.mongo_uri = mongo_uri self.mongo_db = mongo_db @classmethod def from_crawler(cls, crawler): return cls( mongo_uri=crawler.settings.get(\'MONGO_URI\'), mongo_db=crawler.settings.get(\'MONGO_DB\') ) def open_spider(self, spider): self.client = pymongo.MongoClient(self.mongo_uri) self.db = self.client[self.mongo_db] def process_item(self, item, spider): name = item.__class__.__name__ self.db[name].insert(dict(item)) return item def close_spider(self, spider): self.client.close()
【步骤3】MongoPipeline类实现了API定义的另外几个方法。
§ from crawler。它是一个类方法,用@ classmethod标识,是一种依赖注入的方式。它的参数就是 crawler,通过 crawler我们可以拿到全局配的每个配置信息。在全局配置 settings.py中,我们可以定义 MONGO URI和MoN0DB来指定 MongoDB连接需要的地址和数据库名称,拿到配置信息之后返回类对象即可。所以这个方法的定义主要是用来获取 settings. py中的配置的。
§ open spider。当 Spider开启时,这个方法被调用。上文程序中主要进行了一些初始化操作
§ close spider。当 Spider关闭时,这个方法会调用。上文程序中将数据库连接关闭。
最主要的 process item()方法则执行了数据插入操作。定义好 TextPipeline和 MongoPipeline这两个类后,我们需要在 [ settings.py ] 中使用它们 MongoDB的连接信息还需要定义，在其中加入如下信息：
ITEM_PIPELINES = { \'tutorial.pipelines.TextPipeline\': 300, \'tutorial.pipelines.MongoPipeline\': 400, } MONGO_URI=\'localhost\' MONGO_DB=\'tutorial\'
【代码说明】赋值 ITEM PIPELINES字典,键名是 Pipeline的类名称,键值是调用优先级,是一个数字,数字越小则对应的 Pipeline越先被调用。
【步骤4】再重新执行爬取,命令如下所示:
#重新运行，爬取 scrapy crawl quotes
【代码说明】爬取结束后, MongoDB中创建了一个 tutorial的数据库、 Quoteltem的表,如图所示：
【说明】其中text长的部分已经被追加省略号…短的不变，其它author,tags也有加入其中
【结语】至此已经完成最简单用scrapy爬取quotes网站。但更复杂的功能还需进一步学习。
实例代码
quote.py
# -*- coding: utf-8 -*- import scrapy from tutorial.items import TutorialItem class QuotesSpider(scrapy.Spider): name = \'quotes\' allowed_domains = [\'quotes.toscrape.com\'] start_urls = [\'http://quotes.toscrape.com/\'] def parse(self, response): quotes=response.css(".quote") for quote in quotes: item=TutorialItem() item[\'text\']=quote.css(".text::text").extract_first() item[\'author\']=quote.css(\'.author::text\').extract_first() #tags=quote.css(\'.tags .tag::text\').extract() #或<meta class="keywords" itemprop="keywords" content="change,deep-thoughts,thinking,world"> item[\'tags\']=quote.css(\'.keywords::attr(content)\').extract_first() yield item #print(text,author,tags) #下一页 next=response.css(\'.next >a::attr(href)\').extract_first() #函数.urljoin()把当前网址：http://quotes.toscrape.com/+next获取的 拼接成完整网址 url=response.urljoin(next) yield scrapy.Request(url=url, callback=self.parse)
settings.py
# -*- coding: utf-8 -*- # Scrapy settings for tutorial project # # For simplicity, this file contains only settings considered important or # commonly used. You can find more settings consulting the documentation: # # https://doc.scrapy.org/en/latest/topics/settings.html # https://doc.scrapy.org/en/latest/topics/downloader-middleware.html # https://doc.scrapy.org/en/latest/topics/spider-middleware.html BOT_NAME = \'tutorial\' SPIDER_MODULES = [\'tutorial.spiders\'] NEWSPIDER_MODULE = \'tutorial.spiders\' # Crawl responsibly by identifying yourself (and your website) on the user-agent #USER_AGENT = \'tutorial (+http://www.yourdomain.com)\' # Obey robots.txt rules ROBOTSTXT_OBEY = True # Configure maximum concurrent requests performed by Scrapy (default: 16) #CONCURRENT_REQUESTS = 32 # Configure a delay for requests for the same website (default: 0) # See https://doc.scrapy.org/en/latest/topics/settings.html#download-delay # See also autothrottle settings and docs #DOWNLOAD_DELAY = 3 # The download delay setting will honor only one of: #CONCURRENT_REQUESTS_PER_DOMAIN = 16 #CONCURRENT_REQUESTS_PER_IP = 16 # Disable cookies (enabled by default) #COOKIES_ENABLED = False # Disable Telnet Console (enabled by default) #TELNETCONSOLE_ENABLED = False # Override the default request headers: #DEFAULT_REQUEST_HEADERS = { # \'Accept\': \'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\', # \'Accept-Language\': \'en\', #} # Enable or disable spider middlewares # See https://doc.scrapy.org/en/latest/topics/spider-middleware.html #SPIDER_MIDDLEWARES = { # \'tutorial.middlewares.TutorialSpiderMiddleware\': 543, #} # Enable or disable downloader middlewares # See https://doc.scrapy.org/en/latest/topics/downloader-middleware.html #DOWNLOADER_MIDDLEWARES = { # \'tutorial.middlewares.TutorialDownloaderMiddleware\': 543, #} # Enable or disable extensions # See https://doc.scrapy.org/en/latest/topics/extensions.html #EXTENSIONS = { # \'scrapy.extensions.telnet.TelnetConsole\': None, #} # Configure item pipelines # See https://doc.scrapy.org/en/latest/topics/item-pipeline.html #ITEM_PIPELINES = { # \'tutorial.pipelines.TutorialPipeline\': 300, #} ITEM_PIPELINES = { \'tutorial.pipelines.TutorialPipeline\': 300, \'tutorial.pipelines.MongoPipeline\': 400, } MONGO_URI=\'localhost\' MONGO_DB=\'tutorial2\' # Enable and configure the AutoThrottle extension (disabled by default) # See https://doc.scrapy.org/en/latest/topics/autothrottle.html #AUTOTHROTTLE_ENABLED = True # The initial download delay #AUTOTHROTTLE_START_DELAY = 5 # The maximum download delay to be set in case of high latencies #AUTOTHROTTLE_MAX_DELAY = 60 # The average number of requests Scrapy should be sending in parallel to # each remote server #AUTOTHROTTLE_TARGET_CONCURRENCY = 1.0 # Enable showing throttling stats for every response received: #AUTOTHROTTLE_DEBUG = False # Enable and configure HTTP caching (disabled by default) # See https://doc.scrapy.org/en/latest/topics/downloader-middleware.html#httpcache-middleware-settings #HTTPCACHE_ENABLED = True #HTTPCACHE_EXPIRATION_SECS = 0 #HTTPCACHE_DIR = \'httpcache\' #HTTPCACHE_IGNORE_HTTP_CODES = [] #HTTPCACHE_STORAGE = \'scrapy.extensions.httpcache.FilesystemCacheStorage\'
piplines.py
# -*- coding: utf-8 -*- # Define your item pipelines here # # Don\'t forget to add your pipeline to the ITEM_PIPELINES setting # See: https://doc.scrapy.org/en/latest/topics/item-pipeline.html import pymongo from scrapy.exceptions import DropItem class TutorialPipeline(object): def __init__(self): self.limit=50 def process_item(self, item, spider): if item[\'text\']: if len(item[\'text\']) > self.limit: item[\'text\']=item[\'text\'][0:self.limit].rstrip()+"..." return item else: return DropItem("text were missing!") class MongoPipeline(object): def __init__(self, mongo_uri, mongo_db): self.mongo_uri = mongo_uri self.mongo_db = mongo_db @classmethod def from_crawler(cls, crawler): return cls( mongo_uri=crawler.settings.get(\'MONGO_URI\'), mongo_db=crawler.settings.get(\'MONGO_DB\') ) def open_spider(self, spider): self.client = pymongo.MongoClient(self.mongo_uri) self.db = self.client[self.mongo_db] def process_item(self, item, spider): name = item.__class__.__name__ self.db[name].insert(dict(item)) return item def close_spider(self, spider): self.client.close()
items.py
# -*- coding: utf-8 -*- # Define here the models for your scraped items # # See documentation in: # https://doc.scrapy.org/en/latest/topics/items.html import scrapy class TutorialItem(scrapy.Item): # define the fields for your item here like: # name = scrapy.Field() text=scrapy.Field() author=scrapy.Field() tags=scrapy.Field() #pass
main.py
# -*- coding: utf-8 -*- __author__=\'pasaulis\' #在程序中运行命令行，方法调试，如：在jobbole.py中打个断点，运行就会停在那 from scrapy.cmdline import execute import sys,os #获取到当前目录：E:\a\scrapy_test\Aticle，方便后面cmd命令运行不必去找目录 sys.path.append(os.path.dirname(os.path.abspath(__file__))) #测试目录获取是否正确 # print(os.path.dirname(os.path.abspath(__file__))) #调用命令运行爬虫scrapy crawl quotes execute(["scrapy","crawl","quotes"])