无法检索属性之间的值，抓取

Question

我正在尝试从网站检索值，但在属性之间没有得到任何值。（id=Avg Played 除外）。我试过同时使用 Scrapy 和 Beautiful Soup 都无济于事！ 这是我的 BeautifulSoup/Urllib2 代码：

import urllib2
from bs4 import BeautifulSoup


site = "http://www.lolking.net/champions/singed?#/overview"
request= urllib2.Request(site, headers={'User-Agent':'Chrome/44.0.2403.107'})
response = urllib2.urlopen(request)
html = response.read()

soup = BeautifulSoup(html, 'lxml')

champ_stats = soup.findAll('div', attrs={"class" : "champ-stats"})

champ_stats2 = soup.findAll('strong', attrs={"class" : "champ-stats"})


for x in champ_stats:
    print x.text, x

print '\n now showing more specifically: \n'    
for x in champ_stats2:
    print x.text, x

我还使用 Scrapy 制作了一个刮板（得到了相同的结果）：

import scrapy


class StatsSpider(scrapy.Spider):
    name = "stat_spider"
    start_urls = ["http://www.lolking.net/champions/singed?#/overview"]
    def parse(self, response):
        selector = '.champ-stats'
        for stats in response.css(selector):    
            stat_selector = 'strong ::text'
            name_selector = 'span ::text'
            yield {
                'stat': stats.css(stat_selector).extract_first(),
                'name' : stats.css(name_selector).extract_first()
            }   

这是浏览器中 html 的样子（我要检索的内容）：

html = """            <div class="champ-stats">
                <strong id="winrate">48.3</strong><small>%</small>
                <span>Win Rate</span>
            </div>
            <div class="divider"></div>
            <div class="champ-stats">
                <strong id="popularity">0.8</strong><small>%</small>
                <span>Popularity</span>
            </div>
            <div class="divider"></div>
            <div class="champ-stats">
                <strong id="banrate">0.5</strong><small>%</small>
                <span>Ban Rate</span>
            </div>
            <div class="divider"></div>
            <div class="champ-stats">
                <strong>10.2</strong>
                <span>Avg Played</span>
            </div>
        </div> """

我猜该网站有一种方法可以防止人们抓取这些数据？如果是这样，有没有办法解决这个问题？

Answer 1

你最好使用 requests 模块，而不是 urllib2；它使用起来更简单。我应该提一下，BeautifulSoup 可能不足以完全抓取此页面，具体取决于您想要的内容。您可能需要使用 selenium 或 scrapy。

>>> import requests
>>> page = requests.get('http://www.lolking.net/champions/singed?#/overview').content
>>> import bs4
>>> soup = bs4.BeautifulSoup(page, 'lxml')
>>> champ_stats = soup.findAll('div', attrs={"class" : "champ-stats"})
>>> for x in champ_stats:
...     x.text, x
...     
('\n%\nWin Rate\n', <div class="champ-stats">
<strong id="winrate"></strong><small>%</small>
<span>Win Rate</span>
</div>)
('\n%\nPopularity\n', <div class="champ-stats">
<strong id="popularity"></strong><small>%</small>
<span>Popularity</span>
</div>)
('\n%\nBan Rate\n', <div class="champ-stats">
<strong id="banrate"></strong><small>%</small>
<span>Ban Rate</span>
</div>)
('\n10.2\nAvg Played\n', <div class="champ-stats">
<strong>10.2</strong>
<span>Avg Played</span>
</div>)

编辑：

我不确定这是否完全合适。如果我理解正确，可以使用 selenium 刮取这些值。

>>> from selenium import webdriver
>>> driver = webdriver.Chrome()
>>> driver.get('http://www.lolking.net/champions/singed?#/overview')
>>> for item in driver.find_elements_by_xpath('.//div[@class="champ-stats"]/strong'):
...     item.text
...     
'48.4'
'0.8'
'0.4'
'10.2'