Scape Selenium to Pandas - 列表问题

Question

我正在尝试使用此代码抓取烂番茄网站，但出现错误： “ValueError：值的长度与索引的长度不匹配”。

如果有人可以帮助我纠正索引问题，我将不胜感激。

非常感谢您的帮助。

（原代码来自：How to scrape more than one page of critic reviews from Rotten Tomatoes?）

from selenium import webdriver
from selenium.webdriver.common.keys import Keys

# for explicit wait:
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

import time
import pandas as pd

PATH = "C:\Program Files (x86)\chromedriver.exe"
driver = webdriver.Chrome(PATH)
driver.get("https://www.rottentomatoes.com/m/avengers_endgame/reviews")

WebDriverWait(driver, 15).until(EC.element_to_be_clickable((By.CLASS_NAME, 'review_table')))


review_1df = pd.DataFrame(columns=['Date', 'Reviewer', 'Website', 'Review', 'Score'])
dates = []
reviews = []
scores = []
newscores = []
names = []
sites = []


results = driver.find_elements_by_class_name("review_area") # right box
reviewnum = 1

reviewers = driver.find_elements_by_xpath("//div[@class='row review_table_row']")
print(len(reviewers))

for r in results:
    dates.append(r.find_element_by_class_name('subtle').text)
    reviews.append(r.find_element_by_class_name('the_review').text)
    scores.append(r.find_element_by_xpath(".//div[contains(@class,'small subtle review-link')]").text)
    
    for r in reviewers:
        names.append(r.find_element_by_xpath("(.//div[contains(@class,'critic_name')]/a)[1]").text)
        sites.append(r.find_element_by_xpath("(.//div[contains(@class,'critic_name')]/a)[2]").text)
        reviewnum+=1
        
for score in scores:
    if score == ('Full Review'):
        newscores.append('no score')
    else:
        score2 = score[30:]
        newscores.append(score2)

driver.quit()

review_1df['Date'] = dates
review_1df['Review'] = reviews
review_1df['Score'] = newscores
review_1df['Reviewer'] = names
review_1df['Website'] = sites

Answer 1

您可以继续点击“下一步”按钮（带箭头的蓝色框），直到达到页数限制。在每个点击实例中，您可以使用BeautifulSoup 来提取结果：

from bs4 import BeautifulSoup as soup
from selenium import webdriver
import pandas as pd, re, time
d = webdriver.Chrome('/path/to/chromedriver') 
d.get('https://www.rottentomatoes.com/m/avengers_endgame/reviews')
results = []
while True:
   p = soup(d.page_source, 'html.parser')
   results.extend([{'date': i.select_one('div.review-date').get_text(strip=True),
                    'reviewer':i.select_one('div.critic_name a').get_text(strip=True),
                    'website':getattr(i.select_one('em.critic-publication'), 'get_text', lambda **_:None)(strip=True),
                    'review':i.select_one('div.the_review').get_text(strip=True),
                    'score':x[0] if (x:=re.findall('(?<=Original Score:\s)[\d\./]+', i.select_one('div.review_area').text)) else None}
       for i in p.select('div.row.review_table_row')])
   k1, k2 = map(int, re.findall('\d+', p.select_one('span.pageInfo').text))
   if k1 == k2:
       break
   d.execute_script('''document.querySelector('div a[data-qa="next-btn"]').click()''')
   time.sleep(1)

df = pd.DataFrame(results)

输出

                 date          reviewer  ...                                             review score
0       July 28, 2021    Nathan Mattise  ...  The true superteam event releases marked somet...  None
1       July 23, 2021    Robin Holabird  ...                  This goes beyond the usual scale.  None
2       July 13, 2021     Rick Marshall  ...  A film that somehow manages to be as epic as f...  None
3        May 11, 2021       Nick Levine  ...  Films don't come any huger than this: the clos...  None
4      April 29, 2021     Olly Richards  ...  There really is very little that could be impr...   5/5
..                ...               ...  ...                                                ...   ...
539    April 23, 2019  Matthew Pejkovic  ...  A grand, clever and heartfelt blockbuster sure...   4/5
540    April 23, 2019   Jason Escamilla  ...  If you're a Marvel fan, this film is made for ...  None
541    April 23, 2019     Clayton Davis  ...  Jeremy Renner's new haircut-having Hawkeye and...   3/4
542    April 23, 2019     Donald Clarke  ...  Endgame spends too much time wallowing in its ...   3/5
543  November 5, 2018        Brian Gill  ...  Endgame works very, very well as a movie, as a...  None

[544 rows x 5 columns]