【发布时间】：2020-06-11 20:19:31
【问题描述】：
我需要抓取 Google 搜索结果链接。
但是，即使我在代码中输入了time.sleep()，我仍然会收到 HTTP 错误 429。
它适用于 50 - 100 行，然后给出错误 429。但是我需要抓取数百个条形码链接。
我该如何解决这个问题？
import time
from itertools import chain
import pandas as pd
import requests
from bs4 import BeautifulSoup
barcode_df = pd.read_csv("C:/Users/emina/Coding_Projects/PycharmProjects/drug_interaction(pycharm)/barcodes.csv")
barcode_list2d = barcode_df.values.tolist()
barcode_list = list(chain.from_iterable(barcode_list2d)) # This is the list we'll iterate over
barcode_list = [x for x in barcode_list if type(x) == str]
barcode_list_deneme = barcode_list[0:20]
barcode_list1 = barcode_list[0:1000]
USER_AGENT = "some user agent"
headers = {"user-agent": USER_AGENT}
def append_links_to_csv(barcode):
if resp.status_code == 200:
soup = BeautifulSoup(resp.content, "html.parser")
for g in soup.find_all('div', class_='r'):
anchors = g.find_all('a')
if anchors:
link = anchors[0]['href'] # Parses search link
l.write(barcode + "," + link + "\n")
time.sleep(0.06)
else:
print(resp.status_code)
count = 0
l = open("links.csv", "a")
for barcode in barcode_list1:
query = barcode + "+" + "site:ilacabak.com"
url = f"https://google.com/search?q={query}"
resp = requests.get(url, headers=headers)
append_links_to_csv(barcode)
count += 1
print(count)
time.sleep(1.5)
if count % 100 == 0:
l.close()
l = open("links.csv", "a")
l.close()
【问题讨论】：
-
你想爬取谷歌？祝你好运。简单的睡眠不会阻止 Google 将您识别为机器人。
标签： python python-requests http-error