无法提取正确/所有信息需要答案

【问题标题】：Cant extract correct/all information need无法提取正确/所有信息需要
【发布时间】：2019-07-01 19:40:21
【问题描述】：

我正在尝试从该网站获取手机/办公室电话号码信息：https://www.zillow.com/lender-profile/DougShoemaker/

我试过玩 bs4，但我只能得到第一个电话号码。我正在尝试获取办公室和手机号码。

from selenium import webdriver
from bs4 import BeautifulSoup
import time


#Chrome webdriver filepath...Chromedriver version 74
driver = webdriver.Chrome(r'C:\Users\mfoytlin\Desktop\chromedriver.exe')
driver.get('https://www.zillow.com/lender-profile/DougShoemaker/')
soup = BeautifulSoup(driver.page_source, 'html.parser')
time.sleep(2)
phoneNum = driver.find_element_by_class_name('zsg-list_definition')
trial = phoneNum.find_element_by_class_name('zsg-sm-hide')
print(trial.text)

【问题讨论】：

你有什么问题？当您尝试获取办公室和手机号码时会发生什么？您会得到不正确的结果、空结果还是错误消息？
我得到了第一个手机号码的正确结果，我只是不知道如何获得正确的路径或搜索才能获得提供的两个电话号码。上面的代码成功找到并打印了第一个电话号码，但我无法通过...标签的设置方式使得获取所需信息变得很棘手@John Gordon

标签： python selenium web-scraping beautifulsoup selenium-chromedriver

【解决方案1】：

您不必使用 Selenium，甚至是 BeautifulSoup。如果您检查来自 Developer Tools (F12) > Network 的网络请求，您可以看到数据是使用 XHR 请求获取的

您可以自己提出此请求，然后随意使用 JSON 响应。

POST https://mortgageapi.zillow.com/getRegisteredLender?partnerId=RD-CZMBMCZ
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:68.0) Gecko/20100101 Firefox/68.0
Referer: https://www.zillow.com/lender-profile/DougShoemaker/
Content-Type: application/json

{
  "fields": [
    "aboutMe",
    "address",
    "cellPhone",
    # ... other fields
    "website"
  ],
  "lenderRef": {
    "screenName": "DougShoemaker"
  }
}

现在，您可以尝试使用requests 库：

import requests

if __name__ == '__main__':
    payload = {
        "fields": [
            "screenName",
            "cellPhone",
            "officePhone",
            "title",
        ],
        "lenderRef": {
            "screenName": "DougShoemaker"
        }
    }

    res = requests.post('https://mortgageapi.zillow.com/getRegisteredLender?partnerId=RD-CZMBMCZ',
                        json=payload)
    res.raise_for_status()
    data = res.json()

    cellphone, office_phone = data['lender']['cellPhone'], data['lender']['officePhone']
    cellphone_num = '({areaCode}) {prefix}-{number}'.format(**cellphone)
    office_phone_num = '({areaCode}) {prefix}-{number}'.format(**office_phone)
    print(office_phone_num, cellphone_num)

哪个打印：

(618) 619-4120 (618) 795-0790

【讨论】：

【解决方案2】：

为每个电话号码尝试遵循 xpath

Office Phone:
//dt[contains(text(),'Office')]/following-sibling::dd/div/span
Cell Phone:
//dt[contains(text(),'Cell')]/following-sibling::dd/div/span
Fax Number:
//dt[contains(text(),'Fax')]/following-sibling::dd/div/span

【讨论】：

【解决方案3】：

要提取 Office、Cell 和 Fax 号码，您必须为 @987654322 诱导 WebDriverWait @ 并且您可以使用以下任一Locator Strategies：

代码块：

from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

options = webdriver.ChromeOptions()
options.add_argument('start-maximized')
# options.add_argument('disable-infobars')
options.add_argument('--disable-extensions')
driver = webdriver.Chrome(chrome_options=options, executable_path=r'C:\WebDrivers\chromedriver.exe')
driver.get('https://www.zillow.com/lender-profile/DougShoemaker/')
print(WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, "//dt[text()='Office']//following::dd[1]//span"))).get_attribute("innerHTML"))
print(WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, "//dt[text()='Cell']//following::dd[1]//span"))).get_attribute("innerHTML"))
print(WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, "//dt[text()='Fax']//following::dd[1]//span"))).get_attribute("innerHTML"))

控制台输出：

(618) 619-4120
(618) 795-0790
(618) 619-4120

【讨论】：