如何用 Python 抓取网站数据？答案

【问题标题】：How to scrape website data with Python?如何用 Python 抓取网站数据？
【发布时间】：2018-08-02 13:00:49
【问题描述】：

我正在尝试从这个网页http://www.mcxliverates.in 抓取实时商品价值它有一个 iframe 地址：http://213.136.84.136:8000/

这是我用 BeautifulSoup 尝试的：

from bs4 import BeautifulSoup
#import time
import urllib

data = []
url=urllib.urlopen("http://213.136.84.136:8000/")
html=url.read()
url.close()
soup = BeautifulSoup(html,"html.parser")
span=soup.find('table', attrs={'class':'table2'})
table_body = span.find('tbody')

rows = table_body.find_all('tr')
for row in rows:
    cols = row.find_all('td')
    cols = [ele.text.strip() for ele in cols]
    data.append([ele for ele in cols if ele]) # Get rid of empty values

print([data])

下面是我得到的输出：

[[[], [u'INTERNATIONAL MARKET'], [u'SPOT Gold'], [u'SPOT Silver'], [u'CrudeOil'], [u'Copper'], [u'NaturalGas'], [u'Dow Jones'], [u'Bank Nifty'], [u'INDIAN MARKET'], [u'MCXGold'], [u'MCXSilver'], [u'MCXCrudeOil'], [u'MCXCopper'], [u'MCXLead'], [u'MCXNickel'], [u'MCXZinc'], [u'MCXNaturalGas'], [u'MCXAluminium'], [u'MCXMenthaOil'], [u'USDINR'], [], [u"Disclaimer: We can't assure any guarantee about the accuaracy of the data."]]]

它不返回任何引号。

这是 HTML 代码：

<table class="table2" border="1">
    <tbody>
        <tr style="background-color: #DFD9D9; font-size: 16px; ">
            <th style="border:4px solid white; text-align:left; ">SYMBOL</th>
            <th style="border:4px solid white; text-align:right;">LTP</th>
            <th style="border:4px solid white; text-align:right;">HIGH</th>
            <th style="border:4px solid white; text-align:right;">LOW</th>
        </tr>
        <tr style="font-size: 13px; color: black;">
            <td colspan="4" style="color:black; text-align:center; font-size:initial; background-color:lightyellow;">INTERNATIONAL MARKET</td>
        </tr>
        <tr style="font-size: 15px; color: black;">
            <td style="color:#3399ff; background-color:#E0F5D6;">SPOT Gold</td>
            <td id="goldsell" style="text-align:right;"></td>
            <td id="goldhigh" style="text-align:right;"></td>
            <td id="goldlow" style="text-align:right;"></td>

        </tr>
        <tr style="font-size: 15px; color: black;">
            <td style="color:#3399ff; background-color:#E0F5D6;">SPOT Silver</td>
            <td id="Silversell" style="text-align:right;"></td>
            <td id="Silverhigh" style="text-align:right;"></td>
            <td id="Silverlow" style="text-align:right;"></td>

        </tr>
        <tr style="font-size: 15px; color: black;">
            <td style="color:#3399ff; background-color:#E0F5D6;">CrudeOil</td>
            <td id="CrudeOilsell" style="text-align:right;"></td>
            <td id="CrudeOilhigh" style="text-align:right;"></td>
            <td id="CrudeOillow" style="text-align:right;"></td>

        </tr>
        <tr style="font-size: 15px; color: black;">
            <td style="color:#3399ff; background-color:#E0F5D6;">Copper</td>
            <td id="Coppersell" style="text-align:right;"></td>
            <td id="Copperhigh" style="text-align:right;"></td>
            <td id="Copperlow" style="text-align:right;"></td>

        </tr>
        <tr style="font-size: 15px; color: black;">
            <td style="color:#3399ff; background-color:#E0F5D6;">NaturalGas</td>
            <td id="NaturalGassell" style="text-align:right;"></td>
            <td id="NaturalGashigh" style="text-align:right;"></td>
            <td id="NaturalGaslow" style="text-align:right;"></td>
        </tr>
        <tr style="font-size: 15px; color: black;">
            <td style="color:#3399ff; background-color:#E0F5D6;">Dow Jones</td>
            <td id="DJsell" style="text-align:right;"></td>
            <td id="DJhigh" style="text-align:right;"></td>
            <td id="DJlow" style="text-align:right;"></td>

        </tr>
        <tr style="font-size: 15px; color: black;">
            <td style="color:#3399ff; background-color:#E0F5D6;">Bank Nifty</td>
            <td id="BNsell" style="text-align:right;"></td>
            <td id="BNhigh" style="text-align:right;"></td>
            <td id="BNlow" style="text-align:right;"></td>
        </tr>
        <tr style="font-size: 13px; color: black;">
            <td colspan="4" style="color:black; text-align:center; font-size:initial; background-color:lightyellow">INDIAN MARKET</td>
        </tr>
        <tr style="font-size: 15px; color: black;">
            <td style="color:#3399ff; background-color:#E0F5D6;">MCXGold</td>
            <td id="MCXGoldsell" style="text-align:right;"></td>
            <td id="MCXGoldhigh" style="text-align:right;"></td>
            <td id="MCXGoldlow" style="text-align:right;"></td>

        </tr>
        <tr style="font-size: 15px; color: black;">
            <td style="color:#3399ff; background-color:#E0F5D6;">MCXSilver</td>
            <td id="MCXSilversell" style="text-align:right;"></td>
            <td id="MCXSilverhigh" style="text-align:right;"></td>
            <td id="MCXSilverlow" style="text-align:right;"></td>

        </tr>
        <tr style="font-size: 15px;color: black;">
            <td style="color:#3399ff; background-color:#E0F5D6;">MCXCrudeOil</td>
            <td id="MCXCrudeOilsell" style="text-align:right;"></td>
            <td id="MCXCrudeOilhigh" style="text-align:right;"></td>
            <td id="MCXCrudeOillow" style="text-align:right;"></td>

        </tr>
        <tr style="font-size: 15px;color: black;">
            <td style="color:#3399ff; background-color:#E0F5D6;">MCXCopper</td>
            <td id="MCXCoppersell" style="text-align:right;"></td>
            <td id="MCXCopperhigh" style="text-align:right;"></td>
            <td id="MCXCopperlow" style="text-align:right;"></td>

        </tr>
        <tr style="font-size: 15px;color: black;">
            <td style="color:#3399ff; background-color:#E0F5D6;">MCXLead</td>
            <td id="MCXLeadsell" style="text-align:right;"></td>
            <td id="MCXLeadhigh" style="text-align:right;"></td>
            <td id="MCXLeadlow" style="text-align:right;"></td>

        </tr>
        <tr style="font-size: 15px;color: black;">
            <td style="color:#3399ff; background-color:#E0F5D6;">MCXNickel</td>
            <td id="MCXNickelsell" style="text-align:right;"></td>
            <td id="MCXNickelhigh" style="text-align:right;"></td>
            <td id="MCXNickellow" style="text-align:right;"></td>

        </tr>
        <tr style="font-size: 15px;color: black;">
            <td style="color:#3399ff; background-color:#E0F5D6;">MCXZinc</td>
            <td id="MCXZincsell" style="text-align:right;"></td>
            <td id="MCXZinchigh" style="text-align:right;"></td>
            <td id="MCXZinclow" style="text-align:right;"></td>

        </tr>
        <tr style="font-size: 15px;color: black;">
            <td style="color:#3399ff; background-color:#E0F5D6;">MCXNaturalGas</td>
            <td id="MCXNaturalGassell" style="text-align:right;"></td>
            <td id="MCXNaturalGashigh" style="text-align:right;"></td>
            <td id="MCXNaturalGaslow" style="text-align:right;"></td>
        </tr>
        <tr style="font-size: 15px;color: black;">
            <td style="color:#3399ff; background-color:#E0F5D6;">MCXAluminium</td>
            <td id="MCXAluminiumsell" style="text-align:right;"></td>
            <td id="MCXAluminiumhigh" style="text-align:right;"></td>
            <td id="MCXAluminiumlow" style="text-align:right;"></td>
        </tr>
        <tr style="font-size: 15px;color: black;">
            <td style="color:#3399ff; background-color:#E0F5D6;">MCXMenthaOil</td>
            <td id="MCXMenthaOilsell" style="text-align:right;"></td>
            <td id="MCXMenthaOilhigh" style="text-align:right;"></td>
            <td id="MCXMenthaOillow" style="text-align:right;"></td>
        </tr>
        <tr style="font-size: 15px;color: black;">
            <td style="color:#3399ff; background-color:#E0F5D6;">USDINR</td>
            <td id="USDINRsell" style="text-align:right;"></td>
            <td id="USDINRhigh" style="text-align:right;"></td>
            <td id="USDINRlow" style="text-align:right;"></td>

        </tr>

        <tr>
            <td id="lastupdate" colspan="4" style="color:grey; text-align:center; background-color:lavender;">
            </td>
        </tr>

        <tr>
            <td id="disclaimer" colspan="4" style="color:red; text-align:center; background-color:lavender;">
                Disclaimer: We can't assure any guarantee about the accuaracy of the data.
            </td>
        </tr>

    </tbody>
</table>

有谁知道获取此实时报价的其他方法吗？

【问题讨论】：

您提供的 URL 访问受限，因此如果您正在寻求帮助，最好在您的问题中包含 HTML 的 sn-p。
感谢您提供 HTML sn-p。看起来这只是一个模板，实际的报价数据是用 JavaScript 添加到 DOM 中的。这就是数据提供者提供更新的方式。看？ <td id="goldhigh" style="text-align:right;"></td> td 标签中没有内容。
没错！有没有办法从 JavaScript 中获取数据？
Selenium.

标签： python selenium web-scraping beautifulsoup

【解决方案1】：

from bs4 import BeautifulSoup
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

# De-comment next two lines and add chrome_options=chrome_options to ChromeOptions() run Chrome as headless
# chrome_options = webdriver.ChromeOptions()
# chrome_options.add_argument('--headless')
driver = webdriver.Chrome()

url = "http://www.mcxliverates.in/"
driver.get(url)
xpath = '/html/body/div[1]/div[4]/div[1]/iframe'  # this is the where the table that holds our data is located
elem = driver.find_element_by_xpath(xpath)
driver.switch_to.frame(elem)

wait = WebDriverWait(driver, 10)
xpath = '/html/body/div/div/div/table/tbody'
wait.until(EC.presence_of_element_located((By.XPATH, xpath)))  # waits until the data is loaded
html = driver.page_source
driver.close()

soup = BeautifulSoup(html, "html.parser")
table = soup.find('table', class_='table2')  # BeautifulSoup's built-in way of specifying class
table_body = table.find('tbody')

data = []
rows = table_body.find_all('tr')
for row in rows:
    cols = row.find_all('td')
    cols = [ele.text.strip() for ele in cols]
    data.append([ele for ele in cols if ele])  # Get rid of empty values

print([data])

输出：

[[[], ['INTERNATIONAL MARKET'], ['SPOT Gold', '1330.09', '1335.92', '1320.63'], ['SPOT Silver', '16.585', '16.734', '16.338'], ['CrudeOil', '62.82', '63.02', '60.71'], ['Copper', '3.2300', '3.2304', '3.1430'], ['NaturalGas', '2.660', '2.684', '2.638'], ['Dow Jones', '25034', '25250', '24572'], ['Bank Nifty', '24955.0', '24957.0', '24763.0'], ['INDIAN MARKET'], ['MCXGold', '30564', '30575', '30400'], ['MCXSilver', '38713', '38755', '38250'], ['MCXCrudeOil', '4092', '4104', '3974'], ['MCXCopper', '464.85', '464.85', '453.30'], ['MCXLead', '166.00', '166.30', '161.65'], ['MCXNickel', '895.00', '896.30', '871.10'], ['MCXZinc', '231.60', '231.60', '226.00'], ['MCXNaturalGas', '172.00', '173.10', '170.90'], ['MCXAluminium', '142.10', '142.35', '140.65'], ['MCXMenthaOil', '1252.00', '1299.80', '1246.00'], ['USDINR', '64.9360', '65.1040', '64.8780'], ['Last Updated on  : 11:45:57 22-Feb-2018'], ["Disclaimer: We can't assure any guarantee about the accuaracy of the data."]]]

欢迎来到Selenium的精彩世界

【讨论】：

出于这个原因，我将汤属性放在所有页面对象上：) 使用 webdriver 抓取一次，解析树。快速、优雅、避免多次 selenium 调用。