【发布时间】：2022-11-13 14:55:05
【问题描述】：
我正在尝试从以下内容中提取客户评论： https://www.booking.com/hotel/sg/great-world-service-apartments.html#tab-reviews
该网站目前有 6 页包含评论。使用美丽的汤和 Selenium，我只设法刮掉了评论的第一页，因为每个页面的 URL 似乎没有改变。但是，根据请求，每个新页面之间的差异在于偏移值（每个新页面+10）。例如，
到目前为止，我也找到了页码列表，如下所示（在检查中）。是否建议使用页码列表或偏移值的差异来执行其余页面的抓取？
我目前的代码如下：
import scrapy
import pandas as pd
import numpy as np
import re
import requests
import bs4
from bs4 import BeautifulSoup
import csv
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
import time
#Install driver
driver = webdriver.Chrome('./chromedriver.exe')
#specify URL
url1 = 'https://www.booking.com/hotel/sg/great-world-service-apartments.html#tab-reviews'
driver.get(url1)
time.sleep(5)
html = driver.page_source
headers = {'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.82 Safari/537.36'}
r= requests.get(url1,{'headers':headers})
soup = bs4.BeautifulSoup(r.text,'html.parser')
title = soup.head.title
print(title)
print(list(soup.children))
print(list(soup.parents))
titleparent = soup.title.parent.name
print(titleparent)
#Get all reviews
from selenium.webdriver.common.by import By
reviews = driver.find_element(by=By.CLASS_NAME, value='review_list_new_item_block')
print(reviews)
#reviews = soup.find(class_='review_list')
#title = reviews['title']
#reviews = soup.find_all('ul',class_='review_list')
#print(title)
#Find attributes
#1) Customer Name
names = soup.find_all('span',class_='bui-avatar-block__title')
#print(names)
cust_names = []
for i in range(0,len(names)):
cust_names.append(names[i].get_text())
cust_names = cust_names[:10]
print(cust_names)
#2) Country
country = soup.find_all('span',class_='bui-avatar-block__subtitle')
#print(country)
country_list = []
for i in range(0,len(country)):
country_list.append(country[i].get_text())
country_list[:] = [country.strip('\n') for country in country_list]
country_list = country_list[:10]
print(country_list)
#3) Header
header = soup.find_all('div',string=lambda text:"bui-grid__column-9 c-review-block__right")
#print(header)
headers_list = []
for i in range(0,len(header)):
headers_list.append(header[i].get_text())
headers_list[:] = [header.strip('\n') for header in headers_list]
headers_list = headers_list[:10]
#print(headers_list)
#4) Positive Reviews
positive_reviews = soup.find_all('span',{'class':'c-review__body'},{'svg':'bk-icon -iconset-review_great c-review__icon'})
#print(positive_reviews)
pos_reviews_list = []
for i in range(0,len(positive_reviews)):
pos_reviews_list.append(positive_reviews[i].get_text())
#pos_reviews_list[:] = [positive_reviews.strip('\n') for positive_reviews in pos_reviews_list]
print(pos_reviews_list)
#Store into a dataframe
data = {'Customer Name': cust_names,
'Country': country_list,'Positive Reviews':pos_reviews_list}
GWSAReviews1 = pd.DataFrame(data=data)
print(GWSAReviews1)
#GWSAReviews1.to_csv('GWSAReviews1.csv')如果有人可以帮助我，将不胜感激，在此先感谢您！
【问题讨论】：
标签： selenium web-scraping beautifulsoup request webdriver