Python beatuifulsoup：从 div 类中提取值

Question

我想构建一个自动获取德国指数 (DAX) 实时价格的程序。因此，我在价格提供商 FXCM 中使用website。

在我的代码中，我使用 beautifulsoup 和 requests 作为包。存储当前值的 div Box 如下所示：

<div class="left" data-item="quoteContainer" data-bg_quotepush="133962:74:bid">
      <div class="wrapper cf">
        <div class="left">
          <span class="quote quote_standard" data-bg_quotepush="quote" data-bg_quotepush_i="133962:74:bid" data-bg_quotepush_f="quote" data-bg_quotepush_c="40">13.599,24</span>
          <span class="label" data-bg_quotepush="time" data-bg_quotepush_i="133962:74:bid" data-bg_quotepush_f="time" data-bg_quotepush_c="41">25.12.2020</span>
          <span class="label"> • </span>
          <span class="label" data-item="currency"></span>
        </div>
        <div class="right">
          <span class="percent up" data-bg_quotepush="percent" data-bg_quotepush_i="133962:74:bid" data-bg_quotepush_f="percent" data-bg_quotepush_c="42">+0,00<span>%</span></span>
          <span class="label up" data-bg_quotepush="change" data-bg_quotepush_i="133962:74:bid" data-bg_quotepush_f="change" data-bg_quotepush_c="43">0,00</span>
        </div>
      </div>
    </div>

我想要的值是data-bg_quotepush_c="40" 之后的值，并且值是13.599,24。

我的 Python 代码如下所示：

import requests as rq
from bs4 import BeautifulSoup as bs
    
url = "https://news.guidants.com/#Ticker/Profil/?i=133962&e=74"
    
response = rq.get(url)
soup = bs(response.text, "lxml")

price = soup.find_all("div", {"class":"left"})[0].find("span")

print(price["data-bg_quotepush_c"])

它返回以下错误：

File "C:\Users\Felix\anaconda3\lib\site-packages\bs4\element.py", line 1406, in __getitem__ 
return self.attrs[key]

KeyError: 'data-bg_quotepush_c'

Answer 1

如果使用动态生成的内容，请使用 Selenium 而不是请求

发生了什么事？

用requests请求网站只是提供初始内容，不包含所有动态生成的信息，所以你找不到你要找的东西。

要等到网站完全加载，请使用Selenium 和sleep() 作为简单方法或使用selenium waits 高级方法。

避免错误

使用price.text 获取元素的文本，如下所示：

<span class="quote quote_standard" data-bg_quotepush="quote" data-bg_quotepush_c="40" data-bg_quotepush_f="quote" data-bg_quotepush_i="133962:74:bid">13.599,24</span>

示例

from selenium import webdriver
from bs4 import BeautifulSoup

url = "https://news.guidants.com/#Ticker/Profil/?i=133962&e=74"

driver = webdriver.Chrome(executable_path=r'C:\Program Files\ChromeDriver\chromedriver.exe')
driver.get(url)
driver.implicitly_wait(3) 

soup = BeautifulSoup(driver.page_source,"html5lib")
price = soup.find_all("div", {"class":"left"})[0].find("span")
print(price.text)
driver.close()

输出

13.599,24

Answer 2

如果你抓取 div 类的值试试这个，例子

driver = webdriver.Chrome(YourPATH to driver)

from bs4 import BeautifulSoup

# create variable to store a url strings
url = 'https://news.guidants.com/#Ticker/Profil/?i=133962&e=74'

driver.get(url)

# scraping proccess

soup = BeautifulSoup(driver.page_source,"html5lib")

# parse
prices = soup.find_all("div", attrs={"class":"left"})

for price in prices:
    total_price = price.find('span')

# close the driver
driver.close()

如果您使用 requests 模块，请尝试使用不同的解析器 您可以使用 pip 示例安装html5lib

pip install html5lib

谢谢