网页抓取； javascript运行后如何渲染html？

Question

我正在尝试从“GlassDoor”中抓取大诺伊达的科技公司名单：

但是 问题：页面在加载几秒钟后改变了它的内容（可能使用了 javascript）。

链接：https://www.glassdoor.co.in/Explore/browse-companies.htm?overall_rating_low=3.5&page=1&isHiringSurge=0&locId=4475367&locType=C&locName=Greater%20Noida

您可以更改链接中页面的值.. 尝试一下。 更改页码后我正在渲染相同的第一页内容..更改页码后。！

from bs4 import BeautifulSoup
from selenium import webdriver

lst=[]

options = webdriver.ChromeOptions()
options.add_argument('--headless')

browser = webdriver.Chrome(options=options, executable_path='chromedriver.exe')

browser.get("https://www.glassdoor.co.in/Explore/browse-companies.htm?overall_rating_low=3.5&page=8&isHiringSurge=0&locId=4475367&locType=C&locName=Greater%20Noida")
html = browser.page_source
soup = BeautifulSoup(html, features="html.parser")
company_lst = soup.find_all('section', {'class': 'employerCard__EmployerCardStyles__employerCard'})

for item in company_lst:
    d={}
    d['Name'] = item.find('h2', {'data-test': 'employer-short-name'}).text
    d['Rating'] = item.find('span', {'data-test': 'rating'}).text
    d['Reviews-Count'] = item.find('div', {'data-test': 'cell-Reviews-count'}).text
    d['Salaries-Count'] = item.find('div', {'data-test': 'cell-Salaries-count'}).text
    d['Jobs-Count'] = item.find('div', {'data-test': 'cell-Jobs-count'}).text
    d['Industry'] = item.find('span', {'data-test': 'employer-industry'}).text
    print(d['Name'])

Answer 1

不知道为什么页面会重新加载到首页。但是您可以尝试单击下面的页码按钮（效果很好）。

browser.get("https://www.glassdoor.co.in/Explore/browse-companies.htm?overall_rating_low=3.5&page=1&isHiringSurge=0&locId=4475367&locType=C&locName=Greater%20Noida")

for i in range(2,8):
    browser.find_element_by_xpath("//ul[contains(@class,'pagination')]//li/button[text()='{}']".format(i)).click()
    print(browser.current_url) # Prints the URL from page 2 to 7.
    time.sleep(5)