【发布时间】:2019-04-19 17:32:40
【问题描述】:
我想从超市产品中抓取信息,但考虑到某些信息(产品的来源)并不总是可用。
我正在尝试遍历超市链接的数据框。我想从他们每个人那里得到一些信息。但是,产品的来源并不总是可用的。我不知道如何让 Python 仅在可用时才查找“原点”。我试过以下代码:
import urllib.request
from bs4 import BeautifulSoup
import csv
import os
dir = ''
file = 'data.xlsx'
sheetname="Hoja1"
# create and write headers to a list
rows = []
rows.append(['Brand', 'Product', 'Product_Number', 'Gross_Weight', 'Origin'])
# Change working directory:
os.chdir(dir)
# Retrieve current working directory ('cwd'):
cwd = os.getcwd()
cwd
# Load spreadsheet:
xl = pd.ExcelFile(file)
# Load a sheet into a DataFrame by name: df1
df = xl.parse(sheetname)
for index, row in df.iterrows():
# specify the url
urlpage = row['link']
#print(urlpage)
# query the website and return the html to the variable 'page'
page = urllib.request.urlopen(urlpage)
# parse the html using beautiful soup and store in variable 'soup'
soup = BeautifulSoup(page, 'html.parser')
# find results within table
results = soup.find_all('dl', attrs={'class': 'des_info clearfix'})
#print('Number of results', len(results))
for result in results:
# find all columns per result
data = result.find_all('dd')
# check that columns have data
if len(data) == 0:
continue
# write columns to variables
brand = data[0].getText()
product = data[1].getText()
number = data[2].getText()
weight = data[3].getText()
if data[4].getText() == None:
origin = 0
else:
origin = data[4].getText()
# write each result to rows
rows.append([brand, product, number, weight, origin])
我收到以下错误:
if data[4].getText() == None:
IndexError: list index out of range
我想将所有数据排序在一个列表中,如果某个项目的来源不可用,则为零。
【问题讨论】:
标签: python-3.x web-scraping beautifulsoup