【问题标题】:web scraping, back in stock notification网页抓取,库存通知
【发布时间】:2018-06-14 05:01:18
【问题描述】:

我想设置一个 Python 脚本来告诉我产品是否有货。目前它会抓取下面的 url 并解析网站的相关部分,但我无法弄清楚如何获取这个我称之为 stock 的输出变量并将其存储为另一个名为 stock_history 的变量,然后运行另一行询问 stock 是否相等或不去stock_history

我在尝试将 html 数据存储在 stock_history 中时扫描字符串文字错误时也遇到了 EOL。有没有更好的方法来做到这一点?

import requests
from datetime import datetime 
from bs4 import BeautifulSoup
import csv
now = datetime.now()
#enter website address
url = requests.get('https://shop.bitmain.com/antminer_s9_asic_bitcoin_miner.htm')

soup = BeautifulSoup(url.content,'html')

stock = (soup.find("div", "buy-now-bar-con"))

stock_history = '<div class="buy-now-bar-con">
<a class="current" href="antminer_s9_asic_bitcoin_miner.htm?flag=overview">Overview</a>
<a href="antminer_s9_asic_bitcoin_miner.htm?flag=specifications">Specification</a>
<a href="antminer_s9_asic_bitcoin_miner.htm?flag=gallery">Gallery</a>
<a class="btn-buy-now" href="javascript:;" style="background:#a7a4a4; cursor:not-allowed;" target="_self" title="sold out!">Coming soon</a>
</div>'


print(stock)

if stock == stock_history 
    print("still not in stock")

【问题讨论】:

    标签: python html web beautifulsoup screen-scraping


    【解决方案1】:

    首先,EOL 代表“行尾”,如果 python 不喜欢您定义字符串的方式或使用了一些不可靠的字符,您通常会收到此错误。为避免这种情况,您可以在原始代码中对字符串进行三重引用,如下所示:

    stock_history = '''<div class="buy-now-bar-con">
    <a class="current" href="antminer_s9_asic_bitcoin_miner.htm?
    flag=overview">Overview</a>
    <a href="antminer_s9_asic_bitcoin_miner.htm?
    flag=specifications">Specification</a>
    <a href="antminer_s9_asic_bitcoin_miner.htm?flag=gallery">Gallery</a>
    <a class="btn-buy-now" href="javascript:;" style="background:#a7a4a4; 
    cursor:not-allowed;" target="_self" title="sold out!">Coming soon</a>
    </div>'''
    

    这很难看,所以我去掉了那个大字符串,因为它没有必要。您应该想从stock 变量中获取的唯一信息是产品是否售罄。为此,您可以将bs4.element.Tag 转换为str 类型并使用正则表达式检查是否存在“已售罄!”子串。无论您在进行抓取、处理文本数据或执行任何形式的 XML 或 HTML 解析,正则表达式确实派上用场,因此我鼓励您阅读它们。

    更多信息:https://www.regular-expressions.info/

    您可以在这里轻松测试 python 正则表达式捕获:https://pythex.org/

    这是修改后的代码,它可以完成您试图让它做的事情。

    import re
    import csv
    import requests
    from datetime import datetime 
    from bs4 import BeautifulSoup
    
    def stock_check(url):
        """Function checks url for 'sold out!' substring in url.content"""
        soup = BeautifulSoup(url.content, "lxml") #Need to use lxml parser
        stock = soup.find("div", "buy-now-bar-con") #Check the html tags for sold out/coming soon info.
        stock_status = re.findall(r"(sold out!)", str(stock)) #Returns list of captured substring if exists.
        return stock_status[0] # returns "sold out!" from soup string.
    
    now = datetime.now()
    
    url = requests.get('https://shop.bitmain.com/antminer_s9_asic_bitcoin_miner.htm')
    
    if stock_check(url) == "sold out!":
        print(str(now) + ": Still not in stock...")
    else:
        print(str(now) + ": Now in stock!")
    

    试试看,如果您有任何问题,请告诉我!

    编辑:OP 询问如何定期检查网页并包含电子邮件通知。需要对原始解决方案进行一些更改,例如在requests headers 字段中设置userAgent 信息。还为BeautifulSoup 对象切换到html.parser 而不是lxml,以正确处理url.content 中的javascript。

    import re
    import time
    import smtplib
    import requests
    from datetime import datetime 
    from bs4 import BeautifulSoup
    
    def stock_check(url):
        """Checks url for 'sold out!' substring in buy-now-bar-con"""
        soup = BeautifulSoup(url.content, "html.parser") #Need to use lxml parser
        stock = soup.find("div", "buy-now-bar-con") #Check the html tags for sold out/coming soon info.
        stock_status = re.findall(r"sold out!", str(stock)) #Returns list of captured substring if exists.
        return stock_status # returns "sold out!" from soup string.
    
    def send_email(address, password, message):
        """Send an e-mail to yourself!"""
        server = smtplib.SMTP("smtp.gmail.com", 587) #e-mail server
        server.ehlo()
        server.starttls()
        server.login(address,password) #login
        message = str(message) #message to email yourself
        server.sendmail(address,address,message) #send the email through dedicated server
        return
    
    def stock_check_listener(url, address, password, run_hours):
        """Periodically checks stock information."""
        listen = True # listen boolean
        start = datetime.now() # start time
        while(listen): #while listen = True, run loop
            if "sold out!" in stock_check(url): #check page
                now = datetime.now()
                print(str(now) + ": Not in stock.")
            else:
                message = str(now) + ": NOW IN STOCK!"
                print(message)
                send_email(address, password, message)
                listen = False
    
            duration = (now - start)
            seconds = duration.total_seconds()
            hours = int(seconds/3600)
            if hours >= run_hours: #check run time
                print("Finished.")
                listen = False
    
            time.sleep(30*60) #Wait N minutes to check again.    
        return
    
    if __name__=="__main__":
    
        #Set url and userAgent header for javascript issues.
        page = "https://shop.bitmain.com/antminer_s9_asic_bitcoin_miner.htm"
        headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.80 Safari/537.36',
        'Content-Type': 'text/html'}
    
        #URL request.
        url = requests.get(url=page,
                           headers=headers)
    
        #Run listener to stream stock checks.
        address = "user@gmail.com" #your email
        password = "user.password" #your email password
        stock_check_listener(url=url,
                             address=address,
                             password=password,
                             run_hours=1) 
    

    现在,程序将启动一个while 循环,该循环会定期从网页请求信息。您可以通过更改run_hours 变量来设置超时(以小时为单位)。您还可以通过在stock_check_listener 中更改N 来设置睡眠/等待时间(以分钟为单位)。在这种情况下,我使用了gmail,如果您在给自己发送电子邮件时收到错误消息,那么您需要点击此链接:https://myaccount.google.com/lesssecureapps,并允许不太安全的应用程序(您的 python 程序)访问您的 gmail 帐户。

    【讨论】:

    • 我如何在一个合理的休息时间(x 秒)的循环中运行它,然后在现在有库存时给自己发电子邮件! = 真
    • @Twiggy 我不确定你需要它做什么,但你可以尝试构建一个 while 循环,每隔一小时左右检查一次库存。您需要在每个 stock_check(url) 请求之后合并一个 time.sleep() 步骤,然后可能会使其在控制台或终端窗口中运行。您的 while 循环中断的条件取决于您,您可以根据您启动它的时间给它一个持续时间。另外,如果您需要帮助,我可以调查一下。
    猜你喜欢
    • 1970-01-01
    • 2020-06-18
    • 1970-01-01
    • 1970-01-01
    • 1970-01-01
    • 1970-01-01
    • 2021-12-20
    • 2019-04-02
    相关资源
    最近更新 更多