如何从滚动网络表格中抓取特定的单词？答案

【问题标题】：How to scrape specific words from the scrolling web table?如何从滚动网络表格中抓取特定的单词？
【发布时间】：2020-10-07 07:07:04
【问题描述】：

我想从表格中抓取股票代码并检查“最终结算价格”行中是否存在某些关键字{“GD”，“每日价格”，“TSI”}。在表中，有多个动态列。下面是一个例子：我想抓取“Ambuja Cements Futures”的股票代码，即“ZACE”并将其存储在列表中，并检查“最终结算价格”行中是否存在某些关键字。

这是股票代码行的 HTML 代码：

<tr>
   <th>Ticker Symbol</th>
   <td>ZADS</td>
   <td>ZACE</td>
   <td>ZAPN</td>
   <td>ZARB</td>
   <td>ZAXS</td>
   <td>ZBJA</td>
   <td>ZBAF</td>
   <td>ZBPC</td>
   <td>ZBHA</td>
   <td>ZBHI</td>
   <td>ZBOS</td>
   <td>ZCIP</td>
   <td>ZCOA</td>
   <td>ZDRR</td>
   <td>ZEIM</td>
   <td>ZGAI</td>
   <td>ZHCL</td>
   <td>ZHDB</td>
   <td>ZHMC</td>
   <td>ZHND</td>
   <td>ZHPC</td>
   <td>ZHUV</td>
   <td>ZHDF</td>
   <td>ZICI</td>
   <td>ZIHF</td>
   <td>ZIOC</td>
   <td>ZIIB</td>
   <td>ZINF</td>
   <td>ZITC</td>
   <td>ZKMB</td>
   <td>ZLT</td>
   <td>ZLPC</td>
   <td>ZMM</td>
   <td>ZMSI</td>
   <td>ZNTP</td>
   <td>ZONG</td>
   <td>ZPWG</td>
   <td>ZRIL</td>
   <td>ZSBI</td>
   <td>ZSUN</td>
   <td>ZTCS</td>
   <td>ZTTM</td>
   <td>ZTAT</td>
   <td>ZTEC</td>
   <td>ZUTC</td>
   <td>ZUPL</td>
   <td>ZVED</td>
   <td>ZWPR</td>
   <td>ZYES</td>
   <td>ZZEE</td>
</tr>

这是最终结算价格行的 HTML 代码：

<tr>
   <th>Final Settlement Price</th>
   <td>The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.</td>
   <td>The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.</td>
   <td>The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.</td>
   <td>The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.</td>
   <td>The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.</td>
   <td>The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.
      <br>
   </td>
   <td></td>
   <td>The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.</td>
   <td>The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.</td>
   <td>The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.</td>
   <td>The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.</td>
   <td>The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.</td>
   <td>The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.</td>
   <td>The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.
      <br>
   </td>
   <td>The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.
      <br>
   </td>
   <td>The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.</td>
   <td>The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.</td>
   <td>The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.</td>
   <td>The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.</td>
   <td>The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.</td>
   <td>The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.</td>
   <td>The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.</td>
   <td>The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.</td>
   <td>The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.</td>
   <td>The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.</td>
   <td>The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.</td>
   <td>The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.</td>
   <td>The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.</td>
   <td>The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.</td>
   <td>The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.</td>
   <td>The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.</td>
   <td>The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.</td>
   <td>The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.</td>
   <td>The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.</td>
   <td>The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.</td>
   <td>The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.</td>
   <td>The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.</td>
   <td>The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.</td>
   <td>The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.</td>
   <td>The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.</td>
   <td>The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.</td>
   <td>The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.</td>
   <td>The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.</td>
   <td>The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.</td>
   <td>The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.</td>
   <td>The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.</td>
   <td>The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.</td>
   <td>The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.</td>
   <td>The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.</td>
   <td>The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.</td>
</tr>

您可以访问相同的网站：https://www.sgx.com/derivatives/products/indiassf?cc=ZADS

【问题讨论】：

您能否正确缩进您的问题中的 HTML？您可以找到在线 HTML 格式化程序，让这一切变得简单快捷。
请同时添加您正在使用的python代码来抓取文本。
@GregBurghardt 在这里。我更新了 HTML 代码。但是，由于字数限制，我无法上传完整代码。您可以通过单击该网站的链接来查看。我尝试用硒来废弃这个，但没有成功。因此，如果您能提供解决方案，我将不胜感激。

标签： python selenium web-scraping beautifulsoup html-table

【解决方案1】：

表中的数据是动态加载的，所以 BeautifulSoup 看不到它。但是你可以使用re/json/requests模块来获取数据：

import json
import re
import requests


url = 'https://www.sgx.com/derivatives/products/indiassf?cc=ZADS'

app_config_url = 'https://www.sgx.com/config/appconfig.js'
n = re.search(r"apiVersion: '(.*?)'", requests.get(app_config_url).text).group(1)
variables = '''{{"path":"/derivatives/products/{}","lang":"EN"}}'''.format( url.split('?')[0].split('/')[-1] )
url2 = 'https://api2.sgx.com/content-api?queryId={}:page&variables={}'.format(n, variables)

data = requests.get(url2).json()

# uncomment this to prin all data:
#print(json.dumps(data, indent=4))

for c in data['data']['route']['data']['data']['contracts']:
    if c['data']['title'] == 'Ambuja Cements Futures':
        print(c['data']['tickerSymbol'])
        print(c['data']['finalSettlementPrice'])

打印：

ZACE
The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.

编辑：网站更新后的新版本：

import json
import requests


url = 'https://www.sgx.com/derivatives/products/indiassf?cc=ZADS'

config = requests.get('https://www.sgx.com/config/appconfig.json').json()

variables = '''{{"path":"/derivatives/products/{}","lang":"EN"}}'''.format( url.split('?')[0].split('/')[-1] )
url2 = 'https://api2.sgx.com/content-api?queryId={}:page&variables={}'.format(config['cms']['apiVersion'], variables)

data = requests.get(url2).json()

# # uncomment this to prin all data:
# print(json.dumps(data, indent=4))

for c in data['data']['route']['data']['data']['contracts']:
    if c['data']['title'] == 'Ambuja Cements Futures':
        print(c['data']['tickerSymbol'])
        print(c['data']['finalSettlementPrice'])

【讨论】：

我认为在这种情况下不应该发布答案。因为，它强化了“SO 是一个免费的代码问题解决网站”的概念。这个 Q 没有显示出最小的reprex 或任何研究工作的迹象。
@Andrej Kesely 嘿，这段代码运行良好，直到我今天检查。它给出了一个错误。 " AttributeError: 'NoneType' 对象没有属性 'group'" 。看起来网站有维护更新。由于刚开始学习json，所以不知道如何获取app_config_url。因此，如果您能帮助我，我将不胜感激。另外如果你可以分享任何学习json的来源。
@chintanpatel 查看我的更新。关于 JSON：那只是交换格式，类似于 Yaml 或 XML。我将从 HTTP/HTML 协议开始——网络如何工作（请求、http 标头）、正则表达式、CSS 选择器等。