如何抓取 <script text/javascript>答案

【问题标题】：How to scrape <script text/javascript>如何抓取 <script text/javascript>
【发布时间】：2019-11-26 00:44:18
【问题描述】：

所以我想弄清楚如何使用正则表达式来抓取 javascript 标签，我认为这可能是最简单的方法。

标签看起来像：

<script type="text/javascript">

var spConfig=newApex.Config({
  "attributes": {
    "199": {
      "id": "199",
      "code": "legend",
      "label": "Weapons",
      "options": [
        {
          "label": "10",
          "priceInGame": "0",          
          "id": [

          ]
        },
        {
          "label": "10.5",
          "priceInGame": "0",          
          "id": [

          ]
        },
        {
          "label": "11",
          "priceInGame": "0",          
          "id": [
            "66659"
          ]
        },
        {
          "label": "11.5",
          "priceInGame": "0",          
          "id": [            
          ]
        },
        {
          "label": "12",
          "priceInGame": "0",          
          "id": [

          ]
        },
        {
          "label": "12.5",
          "priceInGame": "0",          
          "id": [           
          ]
        },
        {
          "label": "13",
          "priceInGame": "0",         
          "id": [

          ]
        },
        {
          "label": "4",
          "priceInGame": "0",          
          "id": [

          ]
        },
        {
          "label": "4.5",
          "priceInGame": "0",          
          "id": [

          ]
        },
        {
          "label": "5",
          "priceInGame": "0",         
          "id": [

          ]
        },
        {
          "label": "5.5",
          "priceInGame": "0",        
          "id": [

          ]
        },
        {
          "label": "6",
          "priceInGame": "0",         
          "id": [

          ]
        },
        {
          "label": "6.5",
          "priceInGame": "0",         
          "id": [

          ]
        },
        {
          "label": "7",
          "priceInGame": "0",         
          "id": [

          ]
        },
        {
          "label": "7.5",
          "priceInGame": "0",         
          "id": [

          ]
        },
        {
          "label": "8",
          "priceInGame": "0",          
          "id": [
            "66672"
          ]
        },
        {
          "label": "8.5",
          "priceInGame": "0",          
          "id": [
            "66673"
          ]
        },
        {
          "label": "9",
          "priceInGame": "0",          
          "id": [

          ]
        },
        {
          "label": "9.5",
          "priceInGame": "0",        
          "id": [
            "66675"
          ]
        }
      ]
    }
  },
  "weaponID": "66733",
  "chooseText": "Apex Legends",
  "Config": {
    "includeCoins": false,
  }
});

</script>

我想刮掉所有标签

我试图做的是：

        for nosto_sku_tag in bs4.find_all('script', {'type': 'text/javascript'}):
            try:
                test = re.findall('var spConfig = (\{.*}?);', nosto_sku_tag.text.strip())
                print(test)
            except:  # noqa
                continue

但它只返回一个空值[]

所以我在这里问我该怎么做才能刮掉标签？

【问题讨论】：

请注意，type="text/javascript" 不再需要（我认为是用 html5 引入的），所以如果您要抓取网络，它不会出现在每个页面上

标签： javascript python regex beautifulsoup

【解决方案1】：

您需要使用attr=value 或attrs={'attr': 'value'} 语法指定属性。

https://www.crummy.com/software/BeautifulSoup/bs4/doc/#the-keyword-arguments

import json
import re
from ast import literal_eval

from bs4 import BeautifulSoup

if __name__ == '__main__':
    html = '''
<script type="text/javascript">

var spConfig=newApex.Config({
  "attributes": {
    "199": {
      "id": "199",
      "code": "legend",
      "label": "Weapons",
      "options": [
        { "label": "10", "priceInGame": "0", "id": [] },
        { "label": "10.5", "priceInGame": "0", "id": [] },
        { "label": "11", "priceInGame": "0", "id": [ "66659" ] },
        { "label": "7.5", "priceInGame": "0", "id": [] },
        { "label": "8", "priceInGame": "0", "id": ["66672"] }
      ]
    }
  },
  "weaponID": "66733",
  "chooseText": "Apex Legends",
  "taxConfig": {
    "includeCoins": False,
  }
});

</script>    
    '''

    soup = BeautifulSoup(html, 'html.parser')
    # this one works too
    # script = soup.find('script', attrs={'type':'text/javascript'})
    script = soup.find('script', type='text/javascript')
    js: str = script.text.replace('\n', '')
    raw_json = re.search('var spConfig=newApex.Config\(({.*})\);', js, flags=re.MULTILINE).group(1)
    # if `"includeCoins": False,` weren't in the JSON,
    # you could have used json.loads() but it fails here.
    # Yet, ast.literal_eval works fine.
    data = literal_eval(raw_json)
    labels = [opt['label'] for opt in data['attributes']['199']['options']]
    print(labels)

输出：

['10', '10.5', '11', '7.5', '8'] ... some removed for brevity

【讨论】：

json.loads() 比ast.literal_eval() 更合适。
我同意。但它对语法错误（额外的逗号等）和非 json 字符串更宽容。
"includeCoins": False, 是一个语法错误（可能是错字），所以json.loads 在这里不起作用
您好！所以我只是好奇，因为我不像你那样知识渊博，但是做正则表达式来匹配newApex.Config 并抓住标签内的 json 然后使用 json.loads 不是更好吗？通过正则表达式进行操作与您的操作方式有什么不同？ @abdusco
@abdusco 对，你实际上是对的。我想我现在知道它应该如何工作了，多亏了你，我想我已经完成了！

【解决方案2】：

如果您只是在 JSON 对象中查找整个行字段，请使用以下内容；

("label":) "([^"]+)",

那么如果要返回实际值，就用

\2

拉回第二组

【讨论】：