使用请求从网页中收集一些值时遇到问题

Question

我正在尝试从网页的表格中提取一些动态值。 This image 代表我希望从该页面获取的值。应该有任何方法可以使用请求来获取它们。为了让您知道，我在开发工具中查找了任何隐藏的 api，并通过页面源中的脚本标签找出了值，但我找不到。

这是site url

这是我想要的expected output。

这是我到目前为止写的：

import requests
from bs4 import BeautifulSoup

url = "https://www.dailyfx.com/sentiment"

headers = {'User-Agent':'Mozilla/5.0 (Windows NT 6.1; ) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.116 Safari/537.36'}

r = requests.get(url,headers=headers)
soup = BeautifulSoup(r.text,"lxml")
for items in soup.select(".dfx-technicalSentimentCard__barContainer"):
    data = [item.get("data-value") for item in items.select("[data-type='long-value-info'],[data-type='short-value-info']")]
    print(data)

上面的脚本产生如下的空输出：

['--', '--']
['--', '--']
['--', '--']
['--', '--']
['--', '--']
['--', '--']
['--', '--']

如何使用请求从该表中获取值？

Answer 1

您的代码没有问题。问题是这些数字是动态的。如果您检查页面来源，则找不到这些数字，而只能找到“--”。

item_list = soup.find_all(attrs={"class":"dfx-technicalSentimentCard__barContainer"}) print(item_list[-1])

<div class="dfx-technicalSentimentCard__barContainer">
<div class="dfx-sentimentPercentageBar dfx-sentimentPercentageBar--textHidden dfx-technicalSentimentCard__bar">
<div class="dfx-sentimentPercentageBar__long font-weight-bold" data-market-id="LTCUSD" data-stream-type="sentiment" data-type="long-bar" data-value="--">
</div>
<div class="dfx-sentimentPercentageBar__short font-weight-bold" data-market-id="LTCUSD" data-stream-type="sentiment" data-type="short-bar" data-value="--">
</div>
</div>
<div class="dfx-technicalSentimentCard__netLongContainer">
<span class="dfx-technicalSentimentCard__netLongText">Net Long</span>
<span class="dfx-rateDetail__percentageInfoText font-weight-bold" data-market-id="LTCUSD" data-stream-type="sentiment" data-type="long-value-info" data-value="--"></span>
</div>
<div class="dfx-technicalSentimentCard__netShortContainer">
<span class="dfx-technicalSentimentCard__netShortText">Net Short</span>
<span class="dfx-rateDetail__percentageInfoText font-weight-bold" data-market-id="LTCUSD" data-stream-type="sentiment" data-type="short-value-info" data-value="--"></span>
</div>
</div>

所以你应该切换到 Selenium 来获取这些数字

Answer 2

由于内容动态加载，您必须使用 selenium 来收集所需信息

import time

from bs4 import BeautifulSoup
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.keys import Keys

chrome_options = Options()
chrome_options.add_argument("--window-size=1920x1080")
chrome_options.add_argument("--headless")

path_to_chromedriver = 'chromedriver'
driver = webdriver.Chrome(chrome_options=chrome_options, executable_path=path_to_chromedriver)

driver.get('https://www.dailyfx.com/sentiment')

driver.find_element_by_tag_name('body').send_keys(Keys.PAGE_DOWN)
time.sleep(5)
driver.find_element_by_tag_name('body').send_keys(Keys.PAGE_DOWN)

soup = BeautifulSoup(driver.page_source, "lxml")
for items in soup.select(".dfx-technicalSentimentCard__barContainer"):
    data = [item.get("data-value") for item in items.select("[data-type='long-value-info'],[data-type='short-value-info']")]
    print(data)

driver.quit()

对于这段代码，我们可以看到以下输出：

['43', '57']
['53', '47']
['38', '62']
['56', '44']
['57', '43']
['39', '61']
['48', '52']
['77', '23']
['41', '59']
['55', '45']
['56', '44']
['74', '26']
['65', '35']
['87', '13']
['55', '45']
['32', '68']
['43', '57']
['45', '55']
['64', '36']
['56', '44']
['84', '16']
['86', '14']
['97', '3']
['90', '10']