【发布时间】:2018-07-21 07:27:48
【问题描述】:
我正在用 Python 创建一个小型网络抓取程序,它从 newegg.com 获取 GPU 信息并记下所有价格。
到目前为止,我还没有实现电子表格,因为每次运行它时,我都会遇到 2 个错误之一。
代码如下:
from urllib.request import urlopen as uReq
from bs4 import BeautifulSoup as soup
import numpy as np
myURL = "https://www.newegg.com/global/uk/Product/ProductList.aspx?Submit=ENE&N=-1&IsNodeId=1&Description=graphics%20card&bop=And&PageSize=96&order=BESTMATCH" # defining my url as a variable
uClient = uReq(myURL) #opening the connection
page_html = uClient.read() # getting html
uClient.close() # closing the client
page_soup = soup(page_html, "html.parser") # html parsing
containers = page_soup.findAll("div", {"class":"item-container"}) #get all
item containers/product
container = containers[0]
count = 0
for container in containers:
print(count)
brand = container.div.div.a.img["title"]# get the brand of the card
if brand == None:
print("N/A")
else:
print(brand)
title_container = container.findAll("a", {"class", "item-title"})
product_name = title_container[0].text # getting the product name
if product_name == None:
print("N/A")
else:
print(product_name)
price1 = container.find("div",{"class":"item-action"})
price1 = price1.ul
price2 = price1.find("li", {"class": "price-current"}).contents #defining the product price
if not price2:
print("N/A")
else:
print(price2[2])
print(price2[3].text)
print(price2[4].text)
print()
count+=1
错误说明如下:
Traceback(最近一次调用最后一次): 文件“C:/Users/Ethan Price/Desktop/test.py”,第 23 行,在 brand = container.div.div.a.img["title"]# 获取卡片的品牌 TypeError: 'NoneType' 对象不可下标
Traceback(最近一次调用最后一次): 文件“C:/Users/Ethan Price/Desktop/test.py”,第 43 行,在 打印(价格2 [2]) IndexError: 列表索引超出范围
在尝试修复它时,我尝试将列表转换为数组并尝试更改 if 语句。
【问题讨论】:
-
您需要在访问之前验证标签和元素是否存在,而不是在之后检查 None 。错误处理比完美的功能更重要
标签: python web-scraping beautifulsoup urllib