如何加快请求 Python答案

【问题标题】：How to speed up requests Python如何加快请求 Python
【发布时间】：2021-04-22 08:15:01
【问题描述】：

所以我有这段代码可以抓取 javascript 内容：

from requests_html import HTMLSession

#create the session
session = HTMLSession()

#define our URL
url = 'https://partalert.net/product.js?asin=B08L8LG4M3&price=%E2%82%AC702.07&smid=A3JWKAKR8XB7XF&tag=partalertde-21&timestamp=16%3A33+UTC+%2821.4.2021%29&title=ASUS+DUAL+NVIDIA+GeForce+RTX+3070+OC+Edition+Gaming+Grafikkarte+%28PCIe+4.0%2C+8+GB+GDDR6+Speicher%2C+HDMI+2.1%2C+DisplayPort+1.4a%2C+Axial-tech+L%C3%BCfterdesign%2C+Dual+BIOS%2C+Schutzr%C3%BCckwand%2C+GPU+Tweak+II%29&tld=.de'

#use the session to get the data
r = session.get(url)

#Render the page, up the number on scrolldown to page down multiple times on a page
r.html.render(sleep=0, keep_page=True, scrolldown=0)

#take the rendered html and find the element that we are interested in
links = r.html.find('#href')

#loop through those elements extracting the text and link
for item in links:
    link = {
        'link': item.absolute_links
    }
print(link)

但是它需要 2-3 秒，这对我来说太长了。有没有办法加快速度？

【问题讨论】：

如果您查看该 URL 中的脚本的作用，它只会解析您传入的查询字符串中的数据，然后根据该脚本生成一个 Amazon URL。您应该自己解析查询字符串（参见urllib.parse），然后形成链接。

标签： python python-requests request python-requests-html

【解决方案1】：

根本不需要抓取网站。查看源代码时，您可以看到 javascript 正在从输入 url 生成 Amazon url：

document.getElementById(
          "href"
        ).href = `https://www.amazon${tld}/dp/${asin}?tag=${tag}&linkCode=ogi&th=1&psc=1&smid=${smid}`;

这意味着您只需在python 中复制此功能即可生成您的网址。可以通过urllib.parse获取url参数的值，然后使用字符串格式化生成新的url：

from urllib.parse import urlsplit, parse_qs

url = 'https://partalert.net/product.js?asin=B08L8LG4M3&price=%E2%82%AC702.07&smid=A3JWKAKR8XB7XF&tag=partalertde-21&timestamp=16%3A33+UTC+%2821.4.2021%29&title=ASUS+DUAL+NVIDIA+GeForce+RTX+3070+OC+Edition+Gaming+Grafikkarte+%28PCIe+4.0%2C+8+GB+GDDR6+Speicher%2C+HDMI+2.1%2C+DisplayPort+1.4a%2C+Axial-tech+L%C3%BCfterdesign%2C+Dual+BIOS%2C+Schutzr%C3%BCckwand%2C+GPU+Tweak+II%29&tld=.de'
query = urlsplit(url).query
params = parse_qs(query)
amazon_url = f"https://www.amazon{params['tld'][0]}/dp/{params['asin'][0]}?tag={params['tag'][0]}&linkCode=ogi&th=1&psc=1&smid={params['smid'][0]}"

结果：

https://www.amazon.de/dp/B08L8LG4M3?tag=partalertde-21&linkCode=ogi&th=1&psc=1&smid=A3JWKAKR8XB7XF

【讨论】：

非常感谢您的帮助！
我一直在尝试对 dropentry dropsentry.net/… 做同样的事情，但我无法找到所需的 var。
ibb.co/g7M9f73 这是我在 dropentry 中找到的唯一东西
那是另一种情况。这里的亚马逊 url 在 url 中编码为 base64。试试import base64，然后是base64.b64decode(url_variable_here.split('?')[1]).decode("utf-8")。
from urllib.parse import urlsplit, parse_qs import base64 url = 'http://dropsentry.net/product.html...' result = base64.b64decode(url.split('?')[1]).decode("utf-8") print(result) 好的，所以我编写了这段代码，并将其作为输出%7B%22n%22%3A%22Gigabyte%20AORUS%20GeForce%20RTX%203070%20MASTER%20-%20Scheda%20grafica%20da%208%20GB%22%2C%22s%22%3A%22Amazon.it%22%2C%22p%22%3A%22%u20AC737.71%22%2C%22u%22%3A%22https%3A//www.amazon.it/dp/B08LNY8P5L%3Ftag%3Ddropsentry01-21%26linkCode%3Dogi%26th%3D1%26psc%3D1%26smid%3DA11IL2PNWYJU7H%22%2C%22t%22%3A1619036563866%7D