使用 requests-html 从 Python 网页中提取特定元素

Question

说我在看这个网页

https://openpaymentsdata.cms.gov/search/physicians/by-name-and-location?firstname=robert&lastname=b&city=Palo_Alto

我想提取指向该医生个人资料的链接，但是当我尝试网络抓取时，我找不到该元素，即使使用 CSS 选择器也是如此。

from requests_html import HTMLSession

firstname = 'robert'
lastname = 'b'
city = 'Palo_Alto'

url = 'https://openpaymentsdata.cms.gov/search/physicians/by-name-and-location?firstname='\
        + firstname + '&lastname=' + lastname + '&city=' + city

session = HTMLSession()

r = session.get(url)

sel = 'body > div.siteOuterWrapper > div.siteInnerWrapper > div.siteContentWrapper'
print(r.html.find(sel, first=True).text)

这一切都有效，直到我到达内容包装器，在那里我再也看不到任何元素。为什么是这样？我看不到这个元素有什么原因吗？一开始我以为是因为 Javascript，但是这个库声称有完整的 javascript 支持https://requests-html.kennethreitz.org/

Answer 1

下面的 HTTP 请求应该返回您正在查找的数据。 （在浏览器中按 F12 > Network > XHR）

HTTP GET https://openpaymentsdata.cms.gov/resource/khdp-6xuy.json?%24select=%3Aid%2Cphysician_profile_id%2Cphysician_profile_last_name%2Cphysician_profile_middle_name%2Cphysician_profile_first_name%2Cphysician_profile_suffix%2Cphysician_profile_primary_specialty%2Cphysician_profile_address_line_1%2Cphysician_profile_address_line_2%2Cphysician_profile_city%2Cphysician_profile_state%2Cphysician_profile_province_name%2Cphysician_profile_country_name%2Cphysician_profile_zipcode%2Cphysician_profile_alternate_first_name1%2Cphysician_profile_alternate_last_name1%2Cphysician_profile_alternate_first_name2%2Cphysician_profile_alternate_last_name2%2Cphysician_profile_alternate_first_name3%2Cphysician_profile_alternate_last_name3%2Cphysician_profile_alternate_first_name4%2Cphysician_profile_alternate_last_name4%2Cphysician_profile_alternate_first_name5%2Cphysician_profile_alternate_last_name5%2Clocation&%24where=STARTS_WITH(UPPER(physician_profile_first_name)%2C%20%27ROBERT%27)%20AND%20STARTS_WITH(UPPER(physician_profile_last_name)%2C%20%27B%27)%20AND%20STARTS_WITH(UPPER(physician_profile_city)%2C%20%27PALO_ALTO%27)&%24order=physician_profile_last_name%20ASC%2Cphysician_profile_first_name%20ASC&%24limit=300

使用请求

print(requests.get('https://openpaymentsdata.cms.gov/resource/khdp-6xuy.json?%24select=%3Aid%2Cphysician_profile_id%2Cphysician_profile_last_name%2Cphysician_profile_middle_name%2Cphysician_profile_first_name%2Cphysician_profile_suffix%2Cphysician_profile_primary_specialty%2Cphysician_profile_address_line_1%2Cphysician_profile_address_line_2%2Cphysician_profile_city%2Cphysician_profile_state%2Cphysician_profile_province_name%2Cphysician_profile_country_name%2Cphysician_profile_zipcode%2Cphysician_profile_alternate_first_name1%2Cphysician_profile_alternate_last_name1%2Cphysician_profile_alternate_first_name2%2Cphysician_profile_alternate_last_name2%2Cphysician_profile_alternate_first_name3%2Cphysician_profile_alternate_last_name3%2Cphysician_profile_alternate_first_name4%2Cphysician_profile_alternate_last_name4%2Cphysician_profile_alternate_first_name5%2Cphysician_profile_alternate_last_name5%2Clocation&%24where=STARTS_WITH(UPPER(physician_profile_first_name)%2C%20%27ROBERT%27)%20AND%20STARTS_WITH(UPPER(physician_profile_last_name)%2C%20%27B%27)%20AND%20STARTS_WITH(UPPER(physician_profile_city)%2C%20%27PALO_ALTO%27)&%24order=physician_profile_last_name%20ASC%2Cphysician_profile_first_name%20ASC&%24limit=300').json())

输出

[{':id': 'row-9mfk-w6hd-ejup', 'physician_profile_id': '966387', 'physician_profile_last_name': 'BOCIAN', 'physician_profile_middle_name': 'C', 'physician_profile_first_name': 'ROBERT', 'physician_profile_primary_specialty': 'Allopathic & Osteopathic Physicians|Allergy & Immunology|Allergy', 'physician_profile_address_line_1': '795 EL CAMINO REAL', 'physician_profile_city': 'PALO ALTO', 'physician_profile_state': 'CA', 'physician_profile_country_name': 'UNITED STATES', 'physician_profile_zipcode': '94301-2302', 'physician_profile_alternate_first_name1': 'ROBERT', 'physician_profile_alternate_last_name1': 'BOCIAN'}]

Answer 2

您提到的网站从 API - this 获取数据。

您可以使用requests 直接向该API 发出GET 请求并获取您的数据。

您可以使用 Chrome Devtools 找到 API 端点。