为什么在抓取linkedin时网页没有加载？答案

【问题标题】：Why isn't the webpage loding while scraping linkedin?为什么在抓取linkedin时网页没有加载？
【发布时间】：2021-06-05 16:37:34
【问题描述】：

在这段代码中，我尝试使用 Selenium 抓取 Linkedin 配置文件但是驱动程序无法加载页面我猜IP已经被阻止了，我对代理轮换的概念或任何概念都不熟悉在这种情况下使用。如果你能提供帮助，那将是一个很大的帮助我明白这是怎么做到的。

from selenium import webdriver
import time
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
driver = webdriver.Chrome(executable_path=r'C:\Users\chromedriver.exe')


def linkedin_login():
    global driver
    options = webdriver.ChromeOptions()
    options.add_argument("start-maximized")
    options.add_experimental_option('excludeswitches', ['enable-automation'])
    options.add_experimental_option("detach", True)

    try:
        driver.get('https://www.linkedin.com/login')

        username = 'username'
        password = 'password'
    
        WebDriverWait(driver, 30).until(EC.element_to_be_clickable((By.ID, 'username'))).send_keys(username)
        driver.find_element_by_id('password').send_keys(password)
        driver.find_element_by_class_name('btn__primary--large from__button--floating').click()
        time.sleep(8)
    except ImportError:
        print('Closing')

def search_profiles():
   search_profile = input('What profile do you want to search?')
   search_profile = search_profile.split()
   search = search_profile[0] + "%20" + search_profile[1]

【问题讨论】：

标签： python selenium web-scraping proxy

【解决方案1】：

“登录”按钮的类名不正确，特别是缺少点 (.)

代替：

driver.find_element_by_class_name('btn__primary--large from__button--floating').click()

用途：

driver.find_element_by_class_name('btn__primary--large.from__button--floating').click()

这将点击按钮。

另外，如果你运行你共享的代码，你调用的是 webdriver，而不是调用你的函数。

我测试了以下代码并且运行良好（记得更新您的路径和 LinkedIn 凭据）：

from selenium import webdriver
import time
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
driver = webdriver.Chrome(executable_path='/home/armentaahumada/Downloads/chromedriver')


def linkedin_login():
    global driver
    options = webdriver.ChromeOptions()
    options.add_argument("start-maximized")
    options.add_experimental_option('excludeswitches', ['enable-automation'])
    options.add_experimental_option("detach", True)

    try:
        driver.get('https://www.linkedin.com/login')

        username = 'username'
        password = 'password'
    
        WebDriverWait(driver, 30).until(EC.element_to_be_clickable((By.ID, 'username'))).send_keys(username)
        driver.find_element_by_id('password').send_keys(password)
        driver.find_element_by_class_name('btn__primary--large.from__button--floating').click()
        time.sleep(8)
    except ImportError:
        print('Closing')

def search_profiles():
   search_profile = input('What profile do you want to search?')
   search_profile = search_profile.split()
   search = search_profile[0] + "%20" + search_profile[1]

linkedin_login()

【讨论】：