使用 Json 和 BS4 在 HTML 中抓取脚本标签答案

【问题标题】：Scraping Script tag in HTML with Json and BS4使用 Json 和 BS4 在 HTML 中抓取脚本标签
【发布时间】：2019-10-29 21:35:14
【问题描述】：

代码错误，重启项目www

【问题讨论】：

也许你的意思不是.find_all("script")[1] ？
请准确解释您从该脚本标签中所追求的内容

标签： javascript python json web-scraping beautifulsoup

【解决方案1】：

您不能使用 BeautifulSoup 来解析 Javascript 数据，但您可以使用 re 模块（data 是您的 HTML 代码）：

import re    
from bs4 import BeautifulSoup

soup = BeautifulSoup(data, 'lxml')    
txt = soup.select('.Actions script')[1].text

print(re.search(r'sharedPaymentUrl:\s*"(.*?)"', txt)[1])

打印：

https://secure.ewaypayments.com/sharedpage/sharedpayment?AccessCode=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx==

【讨论】：

您需要在网站上发布一些数据（例如使用requests.post()）。刚刚获取页面https://www.supplystore.com.au/shop/checkout/submit.aspx 不包含您的数据。

【解决方案2】：

使用 bs4 4.7.1 的另一种方式。 :包含和拆分

from bs4 import BeautifulSoup as bs
 #html would be response text  e.g. r = requests.get(url): soup = bs(r.content, 'lxml')

html = '''

<div class="Actions">
                <input class="action" type="submit" id="submit-button" value="Place Order" title="Place Order" onclick="return showModal()" disabled="disabled" />
              <input type="hidden" id="EWAY_TransactionID" name="EWAY_TransactionID" value="" />
              <script src="https://secure.ewaypayments.com/scripts/eCrypt.js"> </script>
              <script type="text/javascript">
                var eWAYConfig = {
                  sharedPaymentUrl: "https://secure.ewaypayments.com/sharedpage/sharedpayment?AccessCode=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx=="
                };
                function showModal()
                {
                  // verify captcha

                  // show modal
                  return eCrypt.showModalPayment(eWAYConfig, resultCallback);
                }
                function resultCallback(result, transactionID, errors) {
                  if (result == "Complete") {
                    document.getElementById("EWAY_TransactionID").value = transactionID;
                    document.getElementById("Form_PaymentForm").submit();
                    //Please wait until we process your order, James at 9/10/2017
                    document.getElementById("overlay").style.display = "block";
                  }
                  else if (errors != "")
                  {
                    alert("There was a problem completing the payment: " + errors);
                  }
                }
              </script>

'''
soup = bs(html, 'lxml') 
target = 'sharedPaymentUrl: '
script = soup.select_one('.Actions script:contains("' + target + '")')
if script is None:
    url = 'N/A'
else:
    url = script.text.split(target)[1].split('\n')[0]
print(url)

【讨论】：