python爬取喜马拉雅音频，突破xm-sign校验反爬（爬虫）

 

目录

一、python执行js代码生成xm-sign

1.1获取喜马拉雅服务器时间戳

1.2生成xm-sign

二、根据需要爬取的专辑Id获取音频链接

三、爬取音频并保存

四、测试

好久没写博客了，写个玩玩

一、python执行js代码生成xm-sign

1.1获取喜马拉雅服务器时间戳

js生成xm-sign的函数需要用到这个时间戳，直接上代码啦

\'\'\'爬取喜马拉雅服务器系统时间戳，用于生成xm-sign\'\'\'
def getxmtime():
    headers = {
        \'User-Agent\': \'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.131 Safari/537.36\',
        \'Accept\': \'text/html,application/xhtml+ xml,application/xml;q = 0.9,image/webp,image/apng,*/*;q=0.8, application/signe-exchange;v = b3\',
        \'Host\': \'www.ximalaya.com\'
    }
    url="https://www.ximalaya.com/revision/time"
    response = requests.get(url, headers=headers)
    # print(response)
    html = response.text
    return html

 

1.2生成xm-sign

这一步需要用到的js代码请到这里(CSDN)或者这里(蓝奏云)下载，之后将其与python代码放在同一路径下

\'\'\'生成xm-sign\'\'\'
def exec_js():
    #获取喜马拉雅系统时间戳
    time = getxmtime()
    
    #读取同一路径下的js文件
    with open(\'xmSign.js\', encoding=\'utf-8\') as f:
        js = f.read()

    # 通过compile命令转成一个js对象
    docjs = execjs.compile(js)
    # 调用js的function
    res = docjs.call(\'python\',time)
    #res = docjs.call(\'getnow\')
    return res

 

二、根据需要爬取的专辑Id获取音频链接

def spider_list(albumId):
    all_list = []
    headers = {
        \'User-Agent\': \'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.131 Safari/537.36\',
        \'Accept\':\'text/html,application/xhtml+ xml,application/xml;q = 0.9,image/webp,image/apng,*/*;q=0.8, application/signe-exchange;v = b3\',
        \'Host\': \'www.ximalaya.com\'
    }
    sign = exec_js()
    #加入xm-sign到header中
    headers[\'xm-sign\'] = sign
    # 音频地址
    data_url = \'https://www.ximalaya.com/revision/play/album?albumId={}&pageNum=1&sort=-1&pageSize=30\'.format(albumId)
    # print(data_url)
    response = requests.get(data_url, headers=headers)
    py_dict = json.loads(response.content.decode())
    # print(py_dict)
    book_list = py_dict[\'data\'][\'tracksAudioPlay\']
    for book in book_list:
        # 获取每段音频的名称和地址
        list = {}
        list[\'name\'] = book[\'trackName\']
        list[\'bookName\'] = book[\'albumName\']
        list[\'src\'] = book[\'src\']
        # 打印list
        # print(list)
        all_list.append(list)
    # 返回字典
    return all_list

 

三、爬取音频并保存

调用第二步中的函数获取到链接，进行爬取保存

def spider_songs(list):
    \'\'\'保存音频（字典）\'\'\'
    for i in list:
        dir = "ximalaya/{}/".format(i[\'bookName\'])
        if not os.path.exists(dir):
            print("创建目录:.%s" % dir)
            os.makedirs(dir)
        i[\'name\'] = i[\'name\'].replace("?", "").replace(\'"\', "")
        # 在目录下创建一个喜马拉雅的文件夹
        with open(r\'{}/{}.m4a\'.format(dir, i[\'name\']), \'ab\')as f:
            r = requests.get(i[\'src\'])
            print("正在下载:{}...".format(i[\'name\']), end="")
            f.write(r.content)
            print("\t下载完成！")
def spider_ximalaya(albumId):
    all_list = spider_list(albumId)
    spider_songs(all_list)

 

四、测试

if __name__ ==\'__main__\':
    spider_ximalaya(12642314)