使用scrapy从网站抓取数据时如何获得干净的结果

Question

我是 python 新手，我正在尝试从黄页中抓取数据。我能够刮掉它，但我得到了一个混乱的结果。

这是我得到的结果：

2013-03-24 20:26:47+0800 [scrapy] INFO: Scrapy 0.14.4 started (bot: eyp)
2013-03-24 20:26:47+0800 [scrapy] DEBUG: Enabled extensions: LogStats, TelnetConsole, CloseSpider, WebService, CoreStats, MemoryUsage, SpiderState
2013-03-24 20:26:47+0800 [scrapy] DEBUG: Enabled downloader middlewares: HttpAuthMiddleware, DownloadTimeoutMiddleware, UserAgentMiddleware, RetryMiddleware,DefaultHeadersMiddleware, RedirectMiddleware, CookiesMiddleware, HttpCompressionMiddleware, ChunkedTransferMiddleware, DownloaderStats
2013-03-24 20:26:47+0800 [scrapy] DEBUG: Enabled spider middlewares: HttpErrorMiddleware, OffsiteMiddleware, RefererMiddleware, UrlLengthMiddleware, DepthMiddleware
2013-03-24 20:26:47+0800 [scrapy] DEBUG: Enabled item pipelines: 
2013-03-24 20:26:47+0800 [eyp] INFO: Spider opened
2013-03-24 20:26:47+0800 [eyp] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2013-03-24 20:26:47+0800 [scrapy] DEBUG: Telnet console listening on 0.0.0.0:6023
2013-03-24 20:26:47+0800 [scrapy] DEBUG: Web service listening on 0.0.0.0:6080

我怎样才能得到一个干净的结果？我只想获取姓名、地址、电话号码和链接。

顺便说一句，我用来执行此操作的代码是；

from scrapy.spider import BaseSpider
from scrapy.selector import HtmlXPathSelector
from eyp.items import EypItem
class EypSpider(BaseSpider):
    def parse(self, response):
        hxs = HtmlXPathSelector(response)
        titles = hxs.select('//ol[@class="result"]/li')
        items = []
        for title in titles:
            item = EypItem()
            item['title'] = title.select(".//p/text()").extract()
            item['link'] = title.select(".//a/@href").extract()
            items.append(item)
        return items

Answer 1

您的代码有点乱，但我会尽力提供帮助：

from scrapy.spider import BaseSpider
from scrapy.selector import HtmlXPathSelector
from scrapy.item import Item, Field
import string

class EypItem(Item):
    name = Field()
    address = Field()
    phone = Field()

class eypSpider(BaseSpider):
    name = "eyp.ph"
    allowed_domains = ["eyp.ph"]
    start_urls = ["http://www.eyp.ph/home-real-estate/search/real-estate/davao/cat/real-estate-brokers"]
    def parse(self, response):
        hxs = HtmlXPathSelector(response)
        sites = hxs.select("//li/div[@class='details']")
        items = []
        for site in sites:
            itemE = EypItem()
            itemE["name"] = site.select("normalize-space(p[1]/text())").extract()
            itemE["address"] = site.select("normalize-space(p[2]/text())").extract()
            itemE["phone"] = site.select("normalize-space(p[3]/text())").extract()
            items.append(itemE)
        return items

您缺少class EypItem 的定义。我已经推荐了一个。将上述内容保存为test.py 运行命令行：

$ scrapy runspider test.py -o items.json -t json

将为您提供一个名为 items.json 的带有 JSON 输出的文件。输出样本是

[{"phone": ["Phone: +63(907)6390603"], "name": ["(CARLOS A. VARGAS)"], "address": ["Mezzanine Wee Eng Apartment, Guerrero Street, Davao City, Davao Del Sur"]},
 {"phone": ["Phone: +63(921)9566577"], "name": ["(ROGELIO G. CARBIERO)"], "address": ["Sto. Nino Heights, Pantinople Village, Davao City, Davao Del Sur"]},
 {"phone": ["Phone: +63(917)3137855"], "name": ["(FLORIZEL C. CHAVEZ)"], "address": ["12 Tulip Street, El Rio Vista Village P4a, Davao City, Davao Del Sur"]},
..........