使用 python 和请求抓取放置在 </html> 标记下方的 <script> 标记答案

【问题标题】：Scraping <script> tag placed below </html> tag with python and requests使用 python 和请求抓取放置在 </html> 标记下方的 <script> 标记
【发布时间】：2020-07-03 09:33:25
【问题描述】：

我正在尝试抓取像 this one 这样的页面

他们所做的是从他们的服务器加载所有信息并将其存储在一个 javascript 函数中，以便根据您单击的按钮加载一个或另一个部分。我试图只请求页面，并从脚本函数中获取所有数据，页面结构类似于这样

<!DOCTYPE html>
<html lang="en" xmlns:wb="http://open.weibo.com/wb">
<head>
    <meta charset="utf-8">
        <title>Historical Statistics of Kristiansund BK vs Molde on 2020/07/03 - ScoreBing</title>

#Several script tags over here....
</head>

<body class="vEn">

#Some stuff here...

#This is where the buttons that deploy the data are
<div class="panel-body">
     <div id="live-filter-bar">
         <div class="row MBTitle">
               <div class="small-6 columns PL0">
                    <a href="javascript:set_type(1);" id="tabtypeid1" class="button tiny radius MB0 MRMini VM font-bold">All</a>
                    <a href="javascript:set_type(2);" id="tabtypeid2" class="button tiny radius action MB0 MRMini VM">This League</a>
                    <a href="javascript:" onClick="select(1)" id="tabid1" class="button tiny radius MB0 MRMini VM">All</a>
                    <a href="javascript:" onClick="select(2)" id="tabid2" class="button tiny radius action MB0 MRMini VM">HA</a>
                    <a href="javascript:" onClick="select(3)" id="tabid3" class="button tiny radius action MB0 MRMini VM">AH</a>
                    <a href="javascript:" onClick="select(4)" id="tabid4" class="button tiny radius action MB0 MRMini VM">HH</a>
                    <a href="javascript:" onClick="select(5)" id="tabid5" class="button tiny radius action MB0 MRMini VM">AA</a>
                </div>
                <div class="small-6 columns text-right PR0">
                    <a href="javascript:set_num(10);" id=td10 class="button tiny radius action MB0 MRMini VM">Last 10</a>
                    <a href="javascript:set_num(8);" id=td8 class="button tiny radius action MB0 MRMini VM">Last 8</a>
                    <a href="javascript:set_num(6);" id=td6 class="button tiny radius MB0 MRMini VM">Last 6</a>
                    <a href="javascript:set_num(4);" id=td4 class="button tiny radius action MB0 MRMini VM">Last 4</a>
                </div>
         </div>
     </div>
     <div id="history_table">

     </div>
     <div id="history1">

     </div>
     <div id="history2">

     </div>
</div>

</body>
</html>
<script type="text/javascript">

var kind=1,num=6,typenum=1;
var race=[],league_bgcolor=[],league_i= 1;
var race_have_corner_handicap=1;
var home_id = [];
var guest_id = [];
home_id.push(1405);    guest_id.push(4503);
var sclass='',leaue_id=198;
var tongji_info=[];
var half_goal_av='-',goal_av='-',half_corner_av='-',corner_av='-';
var tmp_host_name,tmp_guest_name,tmp_league_name;
        tmp_host_name = "Mjondalen";
tmp_guest_name = "Kristiansund BK";
tmp_league_name = "Norway Tippeligaen";
race[0]=[746711,198,'20/06/29 12:01','903',1410,tmp_host_name,'957',1405,tmp_guest_name,'0.0','2.5','11 ',tmp_league_name,'2' ,'1','0','0','3',' 3','1','2','0.0','5.5','1.0',1];
    tmp_host_name = "Molde";
tmp_guest_name = "Stabaek";
tmp_league_name = "Norway Tippeligaen";
race[1]=[746712,198,'20/06/29 12:00','661',4503,tmp_host_name,'1162',1396,tmp_guest_name,'-1.0','3.0','10 ',tmp_league_name,'2' ,'1','1','0','5',' 3','4','1','-0.5','4.5','1.0,1.5',1];
    tmp_host_name = "Haugesund";
tmp_guest_name = "Kristiansund BK";
tmp_league_name = "Norway Tippeligaen";
race[2]=[746167,198,'20/06/25 12:00','673',1390,tmp_host_name,'957',1405,tmp_guest_name,'0.0,-0.5','2.5','10.5 ',tmp_league_name,'4' ,'1','0','1','8',' 3','2','2','0.0','5','1.0',1];
    tmp_host_name = "IK Start";
tmp_guest_name = "Molde";
tmp_league_name = "Norway Tippeligaen";
race[3]=[746169,198,'20/06/25 12:00','667',1392,tmp_host_name,'661',4503,tmp_guest_name,'+1.0','3.0','10 ',tmp_league_name,'4' ,'3','1','2','6',' 8','2','3','+0.5','4.5','1.0,1.5',1];
    tmp_host_name = "Kristiansund BK";
tmp_guest_name = "Aalesund";
tmp_league_name = "Norway Tippeligaen";
race[4]=[744697,198,'20/06/22 12:01','957',1405,tmp_host_name,'677',1321,tmp_guest_name,'0.0,-0.5','2.5','10.5 ',tmp_league_name,'4' ,'2','3','2','6',' 2','7','2','0.0','5','1.0',1];
    tmp_host_name = "Molde";
tmp_guest_name = "Rosenborg";
tmp_league_name = "Norway Tippeligaen";
race[5]=[744698,198,'20/06/21 02:30','661',4503,tmp_host_name,'1161',2482,tmp_guest_name,'0.0,-0.5','2.5','10.5 ',tmp_league_name,'4' ,'0','0','0','9',' 4','1','0','0.0','5','1.0',1];
    tmp_host_name = "Aalesund";
tmp_guest_name = "Molde";
tmp_league_name = "Norway Tippeligaen";
race[6]=[743531,198,'20/06/17 12:00','677',1321,tmp_host_name,'661',4503,tmp_guest_name,'0.0,+0.5','2.5,3.0','9.5 ',tmp_league_name,'8' ,'1','1','2','8',' 4','1','4','0.0,+0.5','4.5','1.0,1.5',1];
    tmp_host_name = "Rosenborg";
tmp_guest_name = "Kristiansund BK";
tmp_league_name = "Norway Tippeligaen";
race[7]=[743533,198,'20/06/17 12:00','1161',2482,tmp_host_name,'957',1405,tmp_guest_name,'-1.0','2.5,3.0','11.5 ',tmp_league_name,'7' ,'4','0','0','8',' 9','0','0','0.0,-0.5','5.5','1.0',1];

而且脚本标签比 sn-p 更长。所以，我有两个问题。一，当我做response=requests.get(url=url)和我做response.content时，我可以看到它只到达html标签的末尾，所以我的脚本标签不包括所有数据。如何将其包含在请求中？

第二个问题，我得到它后如何刮掉它？

【问题讨论】：

你确定是图书馆的问题吗？只是知道在 html 正文之后制作了一个带有脚本标记的示例页面，并且能够成功检索它而没有任何问题。也许这是网页本身返回不同响应的问题，具体取决于您是否使用网络浏览器（用户代理等用于拒绝抓取工具的访问）

标签： javascript python-3.x web-scraping python-requests

【解决方案1】：

好吧，看来这只是一个parser 设置，应该使用 BeautifulSoup 进行调整：

import requests
from bs4 import BeautifulSoup

headers = {
    'authority': 'www.scorebing.com',
    'pragma': 'no-cache',
    'cache-control': 'no-cache',
    'upgrade-insecure-requests': '1',
    'user-agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.138 Safari/537.36',
    'accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9',
    'sec-fetch-site': 'none',
    'sec-fetch-mode': 'navigate',
    'sec-fetch-user': '?1',
    'sec-fetch-dest': 'document',
    'accept-language': 'en-US,en;q=0.9',
    
}

response = requests.get('https://www.scorebing.com/match_history/747514', headers=headers)


soup = BeautifulSoup(response.content, 'html.parser', encoding='UTF-8')
soup.find('script', text = re.compile('race_have_corner_handicap'))

输出

<script type="text/javascript">
    var is_en = 1;
    var kind=1,num=6,typenum=1;
    var race=[],league_bgcolor=[],league_i= 1;
    var race_have_corner_handicap=1;
    var home_id = [];
    var guest_id = [];
    home_id.push(1405);    guest_id.push(4503);
...
</script>

【讨论】：

多么好奇，如果你像 BeautifulSoup(response.content,'lxml') 那样解析它，你在 html 标记之后不会得到任何东西，但如果你这样做，就像你建议的那样，BeautifulSoup(response.content,'html.parser') 然后出现之后的脚本标记。谢谢！

【解决方案2】：

~~加载后页面看起来会被脚本更新。~~

你可以~~通过使用~~绕过这个，使用 Selenium 代替请求：

from selenium import webdriver
from bs4 import BeautifulSoup
import re

firefox_profile = webdriver.FirefoxProfile()
firefox_profile.set_preference("browser.privatebrowsing.autostart", True)

driver = webdriver.Firefox(firefox_profile=firefox_profile)

driver.get("https://www.scorebing.com/match_history/747514")
soup = BeautifulSoup(driver.page_source)
#Find the script tag that contains specific text:
data = soup.find('script', text = re.compile('race_have_corner_handicap'))
print(data)

输出

<script type="text/javascript">
    var is_en = 1;
    var kind=1,num=6,typenum=1;
    var race=[],league_bgcolor=[],league_i= 1;
    var race_have_corner_handicap=1;
    var home_id = [];
    var guest_id = [];
    home_id.push(1405);    guest_id.push(4503);
...
</script>

【讨论】：

问题是我需要从 100 多个链接中获取数据，而 selenium 太耗时。如果在加载页面后他们调用数据库以在脚本中上传数据，是否可以使用请求复制该调用？