试图抓取“hellopeter.com”无法深入了解实际评论

Question

我正在尝试制作一个自动化系统，该系统将建立一个来自 hellopeter 的公司的评论数据库。

我想在这个数据集上应用情感分析，我尝试过使用美丽的汤，但我无法挖掘到重要信息

page = requests.get('https://www.hellopeter.com/telkom/reviews/appalling-service-cdc4559e8084e0db01dd6d7e807875460607ac77-2851593')
soup = BeautifulSoup(page.content, 'html.parser')

print(soup.prettify())

我期望的输出是能够深入到实际审查数据所在的类。我实际上得到的是

下面大概有10层

<div id="app">
  </div>

它输出，我不知道如何进行实际审查

更具体地说，我想要

<p data-v-403b1c0a="" itemprop="reviewRating" itemscope="itemscope" itemtype="http://schema.org/Rating">

1

<p data-v-403b1c0a="" itemprop="reviewBody" class="full-content">

<p data-v-403b1c0a="" itemprop="name" class="is-detail-card">APPALLING SERVICE</p>

元素 任何建议

Answer 1

对于visibility_of_element_located()，您可以只使用Selenium 诱导WebDriverWait 提取评论文本，并且您可以使用以下Locator Strategy：

• 使用XPATH：

  print(WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, "//p[@itemprop='reviewRating']//following::p[@class='full-content' and @itemprop='reviewBody']"))).get_attribute("innerHTML"))
  

• 注意：您必须添加以下导入：

  from selenium.webdriver.support.ui import WebDriverWait
  from selenium.webdriver.common.by import By
  from selenium.webdriver.support import expected_conditions as EC
  

• 控制台输出：

  I have had the most horrific experience with Telkom, I have never dealt with such incompetence in my whole 31 years of being alive on earth. 
  
  We opened a contract with Telkom for unlimited WIFI in our home. Instead of Telkom opening one account they opened two and delivered two routers to our home. Why? (Only God knows) I called and notified them, they advised they would send someone out to collect the router. 
  
  Over two weeks later, nothing was resolved! Instead of them debiting the agreed amount of R900 they debited over R3000. I have called several times to ask why this has not been canceled as yet as this was their error and all I was told is they do not know... (How sway!) The following month again I was debited over R2000. Still nothing...
  
  Everyday when I call I am told it will be canceled within 24hours or that someone will give me a call... but NOTHING! ABSOLUTE NOTHING! I have to constantly follow up and nag and I am always getting empty promises. I have called over 20 times for the same issue to be resolved.
  
  The staff are incompetent and the agents have so much attitude and do not care about their customers. I am up to my head in frustration. This Morning I almost broke down in tears. How do they manage to not care at all? This month I received another message to say that they will debit another amount of over R2000.
  
   It amazes me that when it comes to clients not paying in time their services are immediately terminated yet when they have to pay you your money it becomes such a drag. Talk about double standards.
  
  I have never in my life experienced such emotional abuse from a service provider. I started questioning to myself, is it maybe because I am black? female? I am not too sure at this point. I just do not understand how hard could it be to fix an error that they made? It surely must be a literal click of a button or something. Why is it taking so long? We are in the THIRD MONTH now. Does Telkom care about it's customers? From this experience I believe they do not.
  
  I am sick and tired of the excuses and I would just like everything to be sorted. I have been very patient and understanding with Telkom but they are just taking advantage now.  
  

Answer 2

如果你去网络选项卡你会得到以下API。

https://api.hellopeter.com/consumer/business/telkom/reviews/appalling-service-cdc4559e8084e0db01dd6d7e807875460607ac77-2851593

先加载json，获取review_content的key值

import requests
import json
page = requests.get('https://api.hellopeter.com/consumer/business/telkom/reviews/appalling-service-cdc4559e8084e0db01dd6d7e807875460607ac77-2851593')
jsondata=json.loads(page.text)
print(jsondata['review_content'])

输出：

I have had the most horrific experience with Telkom, I have never dealt with such incompetence in my whole 31 years of being alive on earth. 

We opened a contract with Telkom for unlimited WIFI in our home. Instead of Telkom opening one account they opened two and delivered two routers to our home. Why? (Only God knows) I called and notified them, they advised they would send someone out to collect the router. 

两个多星期后，什么都没有解决！他们没有扣除约定的 R900 金额，而是扣除了 R3000。我打了好几次电话问为什么这还没有被取消，因为这是他们的错误，我被告知他们不知道......（多么摇摆不定！）接下来的一个月我又被扣了 2000 多兰特。还是什么都没有……

Everyday when I call I am told it will be canceled within 24hours or that someone will give me a call... but NOTHING! ABSOLUTE NOTHING! I have to constantly follow up and nag and I am always getting empty promises. I have called over 20 times for the same issue to be resolved.

The staff are incompetent and the agents have so much attitude and do not care about their customers. I am up to my head in frustration. This Morning I almost broke down in tears. How do they manage to not care at all? This month I received another message to say that they will debit another amount of over R2000.

It amazes me that when it comes to clients not paying in time their services are immediately terminated yet when they have to pay you your money it becomes such a drag. Talk about double standards.

I have never in my life experienced such emotional abuse from a service provider. I started questioning to myself, is it maybe because I am black? female? I am not too sure at this point. I just do not understand how hard could it be to fix an error that they made? It surely must be a literal click of a button or something. Why is it taking so long? We are in the THIRD MONTH now. Does Telkom care about it's customers? From this experience I believe they do not.

I am sick and tired of the excuses and I would just like everything to be sorted. I have been very patient and understanding with Telkom but they are just taking advantage now.