bs4 下载文件甚至 jQuery 点击

Question

我正在尝试自动从公共网站下载字幕。单击下载链接（西班牙语为Descargar）后，即可访问字幕。 检查网站的代码，我可以看到链接是jQuery事件：

我猜这个事件中有一个函数处理下载（我对 JS 一点也不熟悉）：

function(a) {
  if (ajaxflagon()) return !1;
  var r = $(this).attr("rel");
  if (r = r.split(","), 3 == r.length) var e = "/updated/" + r[0] + "/" + r[1] + "/" + r[2];
  else var e = "/original/" + r[0] + "/" + r[1];
  ga("send", "pageview", "/" + e, {
    title: "Descargando " + $(this).attr("title")
  }), $(this).attr("href", e), ajaxflagoff()
}

到目前为止，我的代码可以找到正确的链接：

import urllib.request as urlRequest
from bs4 import BeautifulSoup

# Subtitles for a specific TV show
urlpage = 'https://www.tusubtitulo.com/season/4674/1'
# pretend to be a chrome 47 browser on a windows 10 machine
headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko)"
}
req = urlRequest.Request(urlpage, headers=headers)
# open the url
url = urlRequest.urlopen(req)
# get the source code
source_code = url.read()

# parse the html using beautiful soup and store in variable 'soup'
soup = BeautifulSoup(source_code, 'html.parser')

results = []
for lang in soup.findAll("td", class_="language"):
    # only interested in the spanish language
    if "Español (España)" in str(lang):
        for element in lang.parent.findAll("a", class_="bt_descarga"):
            results.append(element)

缺少的是下载部分:( 我该怎么做？

提前致谢。

Answer 1

您可以在Python 中实现JS 事件函数并创建下载URLs。

最后，使用URLs，就可以下载字幕了。

以下是仅获取西班牙潜艇的方法：

from shutil import copyfileobj

import requests
from bs4 import BeautifulSoup

base_url = "https://www.tusubtitulo.com"
season = "/season/4674/1"

headers = {
    "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:95.0) Gecko/20100101 Firefox/95.0",
    "Referer": f"{base_url}{season}",
}


def get_rel_attributes(page: str):
    return [
        a["rel"][0] for a in
        BeautifulSoup(page, "lxml").select(".bt_descarga")[1::3]  # this gets only the Spanish subs
    ]


# This is the JS function translated to Python that's responsible for
# building the subtitle download urls.
def get_download_urls(rel_attributes: list):
    src_urls = []
    for item in rel_attributes:
        elements = item.split(",")
        one, two, three = elements
        if len(elements) == 3:
            src_urls.append(f"{base_url}/updated/{one}/{two}/{three}")
        else:
            src_urls.append(f"{base_url}/original/{one}/{two}")
    return src_urls


def downloader(target_url: str, conn: requests.Session):
    response = conn.get(target_url, headers=headers, stream=True)
    file_name = (
        response.headers["Content-Disposition"]
        .split("=", -1)[-1]
        .replace('"', "")
        .encode('latin-1')
        .decode('utf-8')
    )
    print(f"Fetching {file_name}...")
    with open(file_name, "wb") as output:
        copyfileobj(response.raw, output)


if __name__ == "__main__":
    with requests.Session() as connection:
        source_page = connection.get(f"{base_url}{season}", headers=headers).text

    for url in get_download_urls(get_rel_attributes(source_page)):
        downloader(url, connection)

您应该在运行脚本的文件夹中看到此输出和 10 个文件：

Fetching Invasion (2021) 1x01 - Last Day (Español (España)).srt...
Fetching Invasion (2021) 1x02 - Crash (Español (España)).srt...
Fetching Invasion (2021) 1x03 - Orion (Español (España)).srt...
Fetching Invasion (2021) 1x04 - The King is Dead (Español (Latinoamérica)).srt...
Fetching Invasion (2021) 1x05 - Going Home (Español (España)).srt...
Fetching Invasion (2021) 1x06 - Home Invasion (Español (España)).srt...
Fetching Invasion (2021) 1x07 - Hope (Español (España)).srt...
Fetching Invasion (2021) 1x08 - Contact (Español (España)).srt...
Fetching Invasion (2021) 1x09 - Full of Stars (Español (España)).srt...
Fetching Invasion (2021) 1x10 - First Day (Español (España)).srt...