如何从 requests.get() 获取更多标签

Question

我正在尝试从这里获取图像。

https://www.google.co.kr/search?q=dog&rlz=1C1MSIM_koKR747KR747&source=lnms&tbm=isch&sa=X&ved=0ahUKEwjSn7Hm053YAhWLmpQKHXRpAh4Q_AUICigB&biw=1440&bih=794

您可以使用 Chrome 开发者工具检查是否有很多“img”标签。但是，当我运行下面的代码时，我很失望地看到数字 21。如何增加它？

r=requests.get(url)
soup=bsp(r.text,'lxml')
len(soup.find_all('img'))

Answer 1

问题在于 Google 提供的静态页面实际上并不包含任何图像搜索结果。如果您获取结果中包含的图像，您会发现 Google 徽标和一些结构图像 - 可能是他们用来建议搜索优化的标签。

实际的图像是由 Javascript 代码延迟加载的，使用请求来获取这些图像是非常困难的 - 这意味着您通过使用浏览器的工具检查页面的客户端代码正在对服务器执行哪些请求，并且而是模仿那些。这可能可行，也可能不可行，因为谷歌很容易插入一些令牌和预先计算到惰性请求中，这很难从页面上的 Javascript 进行逆向工程。此外，它很可能违反了 Google 图片搜索的使用条款。

您可以尝试切换到Selenium 进行网络搜索，而不是请求。由于它使用真正的浏览器，它将运行 Javascript 并发出实际的延迟请求。它会起作用 - 但您仍然会违反网站的使用条款，而且您可能很快就会在结果中获得验证码。

因此，执行此类操作的正确方法是检查服务提供商（在本例中为 Google 图片搜索）是否具有可用于执行搜索的公共 API。在搜索时，您会发现 Google Image Search API 已被弃用，现在可以使用 Google Search API 来查找图像。目前，他们每天允许 100 次免费搜索，之后会收取服务费。

这是他们了解 API 的链接，并且可能是注册

https://developers.google.com/custom-search/json-api/v1/overview?csw=1

在您了解并获得 API 密钥后，有一个用于 API 的 Python 包装器，它可以让您省去很多麻烦，并且可能会为您提供图像的 URL：

https://pypi.python.org/pypi/Google-Search-API

Answer 2

您可以使用content-type: image/png 查询参数解析缩略图图像。请注意，它只会抓取 20 张图像。如果设置为更多，它将引发错误。详细了解MIME types。

import requests
from bs4 import BeautifulSoup

params = {
    "q": "dog",
    "tbm": "isch", 
    "content-type": "image/png",
}

html = requests.get("https://www.google.com/search", params=params)
soup = BeautifulSoup(html.text, 'html.parser')

for img in soup.select("img"):
  print(img["src"])

------
'''
https://encrypted-tbn0.gstatic.com/images?q=tbn:ANd9GcQq4fZiRGnMloOFKfYsBu91lbGkvT5RcLoa-ExEwbe6LDP3jl7zfZORPaprKA&s
https://encrypted-tbn0.gstatic.com/images?q=tbn:ANd9GcQAxU74QyJ8jn8Qq0ZK3ur_GkxjICcvmiC30DWnk03DEsi7YUgS8XXksdyybXY&s
...
'''

---

如果您想抓取原始分辨率图像 URL，您需要使用regex 并且为了从<script> 标签中提取URL 以匹配、提取和解码它们。与selenium 相比，这将是一个更快的抓取时间。

查找所有脚本标签：

soup.select('script')

其次，使用正则表达式匹配所需的模式：

# one of the regex patterns to find original size URL
re.findall(r"(?:'|,),\[\"(https:|http.*?)\",\d+,\d+\]", SOME_VARIABLE)

第三，遍历匹配，逐个提取和解码每个 URL：

for SOME_VARIABLE in SOME_VARIABLE:
    # it needs to be decoded twice.
    # otherwise Unicode characters will be still present after the first decode.
    # yes, it is stupid. 
    original_size_img_not_fixed = bytes(fixed_full_res_image, 'ascii').decode('unicode-escape')
    original_size_img = bytes(original_size_img_not_fixed, 'ascii').decode('unicode-escape')

---

同样下载图片的代码和full example in the online IDE：

import requests, lxml, re, json, urllib.request
from bs4 import BeautifulSoup

headers = {
    "User-Agent":
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36 Edge/18.19582"
}

params = {
    "q": "dog",     # query
    "tbm": "isch",  # img results
    "hl": "en",     # language 
    "ijn": "0",     # batch of 100 imgaes
}

html = requests.get("https://www.google.com/search", params=params, headers=headers)
soup = BeautifulSoup(html.text, 'lxml')


def get_images_data():

    print('\nGoogle Images Metadata:')
    for google_image in soup.select('.isv-r.PNCib.MSM1fd.BUooTd'):
        title = google_image.select_one('.VFACy.kGQAp.sMi44c.lNHeqe.WGvvNb')['title']
        source = google_image.select_one('.fxgdke').text
        link = google_image.select_one('.VFACy.kGQAp.sMi44c.lNHeqe.WGvvNb')['href']
        print(f'{title}\n{source}\n{link}\n')

    # this steps could be refactored to a more compact
    all_script_tags = soup.select('script')

    # # https://regex101.com/r/48UZhY/4
    matched_images_data = ''.join(re.findall(r"AF_initDataCallback\(([^<]+)\);", str(all_script_tags)))
    
    # https://kodlogs.com/34776/json-decoder-jsondecodeerror-expecting-property-name-enclosed-in-double-quotes
    # if you try to json.loads() without json.dumps it will throw an error:
    # "Expecting property name enclosed in double quotes"
    matched_images_data_fix = json.dumps(matched_images_data)
    matched_images_data_json = json.loads(matched_images_data_fix)

    # https://regex101.com/r/pdZOnW/3
    matched_google_image_data = re.findall(r'\[\"GRID_STATE0\",null,\[\[1,\[0,\".*?\",(.*),\"All\",', matched_images_data_json)

    # https://regex101.com/r/NnRg27/1
    matched_google_images_thumbnails = ', '.join(
        re.findall(r'\[\"(https\:\/\/encrypted-tbn0\.gstatic\.com\/images\?.*?)\",\d+,\d+\]',
                   str(matched_google_image_data))).split(', ')

    print('Google Image Thumbnails:')  # in order
    for fixed_google_image_thumbnail in matched_google_images_thumbnails:
        # https://stackoverflow.com/a/4004439/15164646 comment by Frédéric Hamidi
        google_image_thumbnail_not_fixed = bytes(fixed_google_image_thumbnail, 'ascii').decode('unicode-escape')

        # after first decoding, Unicode characters are still present. After the second iteration, they were decoded.
        google_image_thumbnail = bytes(google_image_thumbnail_not_fixed, 'ascii').decode('unicode-escape')
        print(google_image_thumbnail)

    # removing previously matched thumbnails for easier full resolution image matches.
    removed_matched_google_images_thumbnails = re.sub(
        r'\[\"(https\:\/\/encrypted-tbn0\.gstatic\.com\/images\?.*?)\",\d+,\d+\]', '', str(matched_google_image_data))

    # https://regex101.com/r/fXjfb1/4
    # https://stackoverflow.com/a/19821774/15164646
    matched_google_full_resolution_images = re.findall(r"(?:'|,),\[\"(https:|http.*?)\",\d+,\d+\]",
                                                       removed_matched_google_images_thumbnails)


    print('\nFull Resolution Images:')  # in order
    for index, fixed_full_res_image in enumerate(matched_google_full_resolution_images):

        # https://stackoverflow.com/a/4004439/15164646 comment by Frédéric Hamidi
        original_size_img_not_fixed = bytes(fixed_full_res_image, 'ascii').decode('unicode-escape')
        original_size_img = bytes(original_size_img_not_fixed, 'ascii').decode('unicode-escape')
        print(original_size_img)


get_images_data()

---------------
'''
Google Images Metadata:
How dogs contribute to your health and happiness
medicalnewstoday.com
https://www.medicalnewstoday.com/articles/322868
...

Google Image Thumbnails:
https://encrypted-tbn0.gstatic.com/images?q=tbn:ANd9GcT4Uiy-5lDV8xPcpy32U3gx7uqXLmHhi1tNt5qnJgaCCgc6RiCjiNePMTyVgVF4wrDqjgQ&usqp=CAU
...

Full Resolution Images:
https://post.medicalnewstoday.com/wp-content/uploads/sites/3/2020/02/322868_1100-800x825.jpg
...
'''

---

或者，您可以使用来自 SerpApi 的 Google Images API 来实现相同的目的。这是一个带有免费计划的付费 API。

您的情况的不同之处在于，您不必弄清楚这些 URL 位于 <script> 标记中，并了解如何为它们编写 regex 或在 HTML 中发生更改时随着时间的推移进行维护，相反，您需要遍历结构化 JSON 并获取所需的数据。

要集成的代码：

import os, json # json for pretty output
from serpapi import GoogleSearch


def get_google_images():
    params = {
      "api_key": os.getenv("API_KEY"),
      "engine": "google",
      "q": "dog",
      "tbm": "isch"
    }

    search = GoogleSearch(params)
    results = search.get_dict()

    print(json.dumps(results['images_results'], indent=2, ensure_ascii=False))

get_google_images()

---------------
'''
[
  {
    "position": 100,   # img number
    "thumbnail": "https://encrypted-tbn0.gstatic.com/images?q=tbn:ANd9GcR1somXixBxVLINMTbvzPfhX62xC2DimRvL-Q&usqp=CAU",
    "source": "hakaimagazine.com",
    "title": "The Dogs That Grew Wool and the People Who Love Them | Hakai Magazine",
    "link": "https://www.hakaimagazine.com/features/the-dogs-that-grew-wool-and-the-people-who-love-them/",
    "original": "https://www.hakaimagazine.com/wp-content/uploads/shiba-inu-wool-dogs.jpg",
    "is_product": false
  }
]
...
'''

P.S - 我写了一篇更深入的博文，介绍如何从 Google Images 抓取和下载图像。

免责声明，我为 SerpApi 工作。