【发布时间】：2022-01-24 21:08:23
【问题描述】：
我的爬虫机器人正常工作，直到我在数据库中添加商家列。 Scraper.py 文件抓取成功商家并正确记录在数据库中，但添加商家列后比较机器人不比较
我的原始代码：（比较成功）
# The max_date a product can have in order to be considered "old"
limit = datetime.datetime.strptime(products[-1][1], "%Y-%m-%d %H:%M:%S.%f") + \
datetime.timedelta(minutes=OLDER_PRODUCTS_RANGE)
# Separate "old" from "new" products
for i, product in enumerate(products[::-1]):
date = datetime.datetime.strptime(
product[1], "%Y-%m-%d %H:%M:%S.%f")
index = len(products) - i - 1
if date > limit:
old_products = products[index+1:]
new_products = products[:index+1]
break
# If we have only one or even none of the lists, return
if len(new_products) == 0 or len(old_products) == 0:
c.close()
self.save_db(db)
return True
older_product = min(old_products, key=lambda x: x[5])
current_product = min(new_products, key=lambda x: x[5])
first_price = older_product[5]
current_price = current_product[5]
current_date = current_product[1]
url = current_product[6]
price_difference = first_price - current_price
percentage = price_difference / first_price
percentage_str = str("%.2f" % (percentage * 100))
# If the drop in the price was greater then the expected percentage, warn the user
if percentage >= self.percentage:
rowid = current_product[0]
product_name = current_product[4]
product_id = current_product[3]
print(
f"[Bot] [{current_date}] Price of \"{product_name}\" is {percentage_str}% off")
message = product_name + "\n\n" + \
str(first_price) + " TL >>>> " + \
str(current_price) + f" TL - {percentage_str}%" + "\n\n" + \
MAIN_URL + url + "\n\n" + \
MAIN_URL + "ara?q=" + product_id
context.bot.send_message(
chat_id=CHANNEL_ID,
text=message
)
c.execute(
"INSERT INTO deleted SELECT rowid FROM products WHERE product_id = %s AND rowid != %s;",
(product_id, rowid)
)
c.close()
self.save_db(db)
return True
我编辑的代码：（没有价格比较）
# The max_date a product can have in order to be considered "old"
limit = datetime.datetime.strptime(products[-1][1], "%Y-%m-%d %H:%M:%S.%f") + \
datetime.timedelta(minutes=OLDER_PRODUCTS_RANGE)
# Separate "old" from "new" products
for i, product in enumerate(products[::-1]):
date = datetime.datetime.strptime(
product[1], "%Y-%m-%d %H:%M:%S.%f")
index = len(products) - i - 1
if date > limit:
old_products = products[index+1:]
new_products = products[:index+1]
break
# If we have only one or even none of the lists, return
if len(new_products) == 0 or len(old_products) == 0:
c.close()
self.save_db(db)
return True
older_product = min(old_products, key=lambda x: x[5])
current_product = min(new_products, key=lambda x: x[5])
first_price = older_product[5]
current_price = current_product[5]
current_date = current_product[1]
url = current_product[6]
merchant = current_product[7]
price_difference = first_price - current_price
percentage = price_difference / first_price
percentage_str = str("%.2f" % (percentage * 100))
# If the drop in the price was greater then the expected percentage, warn the user
if percentage >= self.percentage:
rowid = current_product[0]
product_name = current_product[4]
product_id = current_product[3]
print(
f"[Bot] [{current_date}] Price of \"{product_name}\" is {percentage_str}% off")
message = product_name + "\n\n" + \
+ "Satici Adi:" + merchant + "\n\n" + \
str(first_price) + " TL >>>> " + \
str(current_price) + f" TL - {percentage_str}%" + "\n\n" + \
MAIN_URL + url + "?magaza=" + merchant + "\n\n" + \
MAIN_URL + "ara?q=" + product_id
context.bot.send_message(
chat_id=CHANNEL_ID,
text=message
)
c.execute(
"INSERT INTO deleted SELECT rowid FROM products WHERE product_id = %s AND rowid != %s;",
(product_id, rowid)
)
c.close()
self.save_db(db)
return True
我使用 python 和 beautifulsoup4
怎么了？
【问题讨论】：
标签： python web-scraping beautifulsoup scrape