如何在使用 Python 进行网络抓取时绕过 cookie 协议页面？答案

【问题标题】：How can I bypass a cookie agreement page while web scraping using Python?如何在使用 Python 进行网络抓取时绕过 cookie 协议页面？
【发布时间】：2019-12-19 01:59:51
【问题描述】：

我对 cookie 协议页面感到头疼...

我在做什么：

import requests
url = "https://stockhouse.com/community/bullboards/"
r = requests.get(url)
soup = BeautifulSoup(r.content, "html.parser")
print(soup)

从 cookie 协议页面返回 HTML。然后我正在寻找的是绕过此页面并在我们接受 cookie 后抓取实际页面的内容...

我尝试了question中的代码：

cookies = dict(BCPermissionLevel='PERSONAL')
html = requests.get(website, headers={"User-Agent": "Mozilla/5.0"}, cookies=cookies)

但我仍然从 cookie 页面获取 html。

注意：我成功使用了 Selenium，但是 selenium 是一个非常低效的最后手段......

【问题讨论】：

标签： python web-scraping python-requests

【解决方案1】：

对于这个站点，指定“虚拟”cookie privacy-policy 就足够了：

import requests
from bs4 import BeautifulSoup

url = "https://stockhouse.com/community/bullboards/"

cookies = {
    'privacy-policy': '1,XXXXXXXXXXXXXXXXXXXXXX'
}

r = requests.get(url, cookies=cookies)
soup = BeautifulSoup(r.content, "html.parser")

for h3 in soup.select('h3'):
    print(h3.get_text(strip=True))

打印标题：

Perfect timing: Mach offer no good as per AMF
'Explosive' Move Up Next Week"
Repsol/ Tullow
Assessment
$5.96
Possible Deal?
Massive Investor(s) Buys Over 1 Million JE Shares Last Close
This CEO is really on the ball , right flubber
slow bb
Situation
Loadddddd
Numerology of the number 36
TIMBERRRR!!.. it will go down fast to $1.50
Employees in the know do the right thing Whistelblow
News finally
Will be bought out...halt coming
Green today
Somebody is buying
re re :350 mil is not enough
And Trump fk up another day

【讨论】：