【问题标题】：How to get around anti-adblock html page while scraping using python requests library?使用 python 请求库抓取时如何绕过反广告块 html 页面？
【发布时间】：2023-02-08 01:46:17
【问题描述】：
我正在尝试自动执行分配给我的文书报告，其中一部分涉及从街道地址获取邮政编码。我已经尝试了很多东西，比如 geopy 库，我什至尝试使用 chatgpt API 来获取邮政编码。这些方法给了我大部分准确的结果，但并不完美，所以我不能真正使用它们。现在，我正在尝试从可靠的网站 https://www.unitedstateszipcodes.org 中获取信息，以使用请求库获取邮政编码。
作为测试，我编写了这段代码以查看我会得到什么：
import requests
test_URL = "https://www.unitedstateszipcodes.org"
def get_data(link):
hdr = {'user-agent': 'Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.107 Mobile Safari/537.36'}
req = requests.get(link, headers=hdr)
content = req.text
return content
data = get_data(test_URL)
但是当我查看我收到的 HTML 页面时，它只说要删除广告拦截器。有没有办法解决？
【问题讨论】：
标签：
python
html
web-scraping
https
【解决方案1】：
一种解决方案可能是在您的 URL 之前添加 Google Web 缓存 URL，例如：
import requests
test_URL = "https://www.unitedstateszipcodes.org"
cache_URL = "https://webcache.googleusercontent.com/search?q=cache:"
def get_data(link):
hdr = {
"User-Agent": "Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.107 Mobile Safari/537.36"
}
req = requests.get(cache_URL + link, headers=hdr)
content = req.text
return content
data = get_data(test_URL)
print(data)
印刷：
<!DOCTYPE html><meta http-equiv="Content-Type" content="text/html; charset=UTF-8"><base href="https://www.unitedstateszipcodes.org/"><style>body{margin-left:0;margin-right:0;margin-top:0}#bN015htcoyT__google-cache-hdr{background:#f8f9fa;font:13px arial,sans-serif;text-align:left;color:#202124;border:0;margin:0;border-bottom:1px solid #dadce0;line-height:16px;padding:16px 28px 24px 28px}#bN015htcoyT__google-cache-hdr *{display:inline;font:inherit;text-align:inherit;color:inherit;line-height:inherit;background:none;border:0;margin:0;padding:0;letter-spacing:0}#bN015htcoyT__google-cache-hdr a{text-decoration:none;color:#1558d6;}#bN015htcoyT__google-cache-hdr a:hover{text-decoration:underline}#bN015htcoyT__google-cache-hdr a:visited{color:#681da8}#bN015htcoyT__google-cache-hdr div{display:block;margin-top:4px}#bN015htcoyT__google-cache-hdr b{font-weight:bold;display:inline-block;direction:ltr}</style><div id="bN015htcoyT__google-cache-hdr"><div><span>Toto je vyrovnávacia pamäť Google pre <a href="https://www.unitedstateszipcodes.org/">https://www.unitedstateszipcodes.org/</a>.</span> <span>Je to snímka stránky, ako sa zobrazila dňa 7. feb. 2023 13:26:55 GMT.</span> <span><a href="https://www.unitedstateszipcodes.org/">Aktuálna stránka</a> sa odvtedy mohla zmeniť.</span> <a href="http://support.google.com/websearch/bin/answer.py?hl=sk&p=cached&answer=1687222"><span>Ďalšie informácie</span>.</a></div><div><span style="display:inline-block;margin-top:8px;margin-right:104px;white-space:nowrap"><span style="margin-right:28px"><span style="font-weight:bold">Úplná verzia</span></span><span style="margin-right:28px"><a href="http://webcache.googleusercontent.com/search?q=cache:https://www.unitedstateszipcodes.org&prmd=ivn&strip=1&vwsrc=0"><span>Len textová verzia</span></a></span><span style="margin-right:28px"><a href="http://webcache.googleusercontent.com/search?q=cache:https://www.unitedstateszipcodes.org&prmd=ivn&strip=0&vwsrc=1"><span>Zobraziť zdroj</span></a></span></span></div><span style="display:inline-block;margin-top:8px;color:#70757a"><span>Tip: Ak chcete rýchlo nájsť hľadaný výraz na tejto stránke, stlačte klávesy <b>Ctrl+F</b> alebo <b>⌘-F</b> (Mac) a použite panel vyhľadávania.</span></span></div><div style="position:relative;"><!DOCTYPE html>
<html lang="en">
<head>
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>Free ZIP code map, zip code lookup, and zip code list</title>
<meta name="description" content="United States Zip Codes provides a free zip code map and list of zip codes by state. Find zip codes by address, zip codes by city, or zip codes by state.">
...