Python Beautifulsoup findAll 找到一些但不是全部

Question

借助一点 Python 知识，我尝试抓取一些 LinkedIn 公司的帖子。

使用我从this website 获取的以下代码，在提取其内容之前首先找到公司LinkedIn 页面上的所有帖子。问题是我知道，我计算过，无论我使用的是 lxml、html5lib 或 html.parser 中的哪一个解析器，都比 findAll 函数返回的帖子多。在一种情况下，它会返回 67 个帖子中的 43 个，在另一种情况下，它会返回 14 个帖子中的 10 个。通常，它会找到大约 3 或 4 个，然后跳过 4 或 5 个帖子，然后再次找到一些，等等。

我怎样才能知道为什么会发生这种情况？

#!/usr/bin/env python
# coding: utf-8

# Import
from selenium import webdriver
from bs4 import BeautifulSoup as bs
import time
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from webdriver_manager.chrome import ChromeDriverManager

# Get credentials to log in to LinkedIn
username = input('Enter your linkedin username: ')
password = input('Enter your linkedin password: ')
company_name = input('Name of the company: ')

# Access Webdriver
s=Service(ChromeDriverManager().install())
browser = webdriver.Chrome(service=s)
browser.maximize_window()

# Define page to open
page = "https://www.linkedin.com/company/{}/posts/?feedView=all".format(company_name)

# Open login page
browser.get('https://www.linkedin.com/login?fromSignIn=true&trk=guest_homepage-basic_nav-header-signin')

# Enter login info:
elementID = browser.find_element_by_id('username')
elementID.send_keys(username)
elementID = browser.find_element_by_id('password')
elementID.send_keys(password)
elementID.submit()

# Go to webpage
browser.get(page + 'posts/')

# Define scrolling time
SCROLL_PAUSE_TIME = 1.5

# Get scroll height
last_height = browser.execute_script("return document.body.scrollHeight")

# Scroll all the way to the bottom of the page
while True:

    # Scroll down to bottom
    browser.execute_script("window.scrollTo(0, document.body.scrollHeight);")
    
    # Wait to load page
    time.sleep(SCROLL_PAUSE_TIME)

    # Calculate new scroll height and compare with last scroll height
    new_height = browser.execute_script("return document.body.scrollHeight")
    if new_height == last_height:
        break
    last_height = new_height

# Get content of page
content = browser.page_source.encode('utf-8').strip()

# Create soup
linkedin_soup = bs(content, "html5lib")
linkedin_soup.prettify()

# Find entities that contain posts
containers = linkedin_soup.findAll("div",{"class":"occludable-update ember-view"})

Answer 1

问题是当您向下滚动到底部时，它会跳过一些要呈现的帖子。可能有更好的方法来做到这一点，但基本上我已经滚动了 1/4，然后是 1/2，然后是完整的（希望能抓住所有的帖子）。试试这个调整：

# Scroll all the way to the bottom of the page
while True:

    # Scroll down to bottom
    browser.execute_script("window.scrollTo(0, document.body.scrollHeight/4);")
    browser.execute_script("window.scrollTo(0, document.body.scrollHeight/2);")
    browser.execute_script("window.scrollTo(0, document.body.scrollHeight);")
    
    # Wait to load page
    time.sleep(SCROLL_PAUSE_TIME)

    # Calculate new scroll height and compare with last scroll height
    new_height = browser.execute_script("return document.body.scrollHeight")
    if new_height == last_height:
        break
    last_height = new_height

Answer 2

所以@chitown88 让我走上了正轨，这是我现在拥有的最终代码，它可以让我得到我需要的结果：

# Define scrolling height and time
SCROLL_PAUSE_TIME = 1.5 # [sec]
SCROLL_HEIGHT = 1000

# Pause to be sure page is loaded
time.sleep(SCROLL_PAUSE_TIME)

# Scroll all the way to the bottom of the page
new_height = SCROLL_HEIGHT
while True:

    # Get maximal scroll height
    max_height = browser.execute_script("return document.body.scrollHeight")

    # Check whether maximal scroll height has been exceeded
    if new_height > max_height:
        break

    # Scroll to position
    browser.execute_script("window.scrollTo(0, {});".format(new_height))
    time.sleep(SCROLL_PAUSE_TIME)

    # Get current scroll position
    #current_height = browser.execute_script("return window.pageYOffset")

    # Increase scroll position
    new_height = new_height + SCROLL_HEIGHT

# Make sure to reach last position
browser.execute_script("window.scrollTo(0, {});".format(max_height))

我留在了current_height 变量中，不确定是否需要再次使用它，此代码需要更多验证。可能对保存有用。