如果不在视口中，则会延迟加载的 Web 抓取图像

Question

我正在尝试从这个无限滚动的网站：https://www.grailed.com/designers/jordan-brand/hi-top-sneakers 中抓取所有提要项图像。我已经看到了许多其他相关问题的答案并尝试了解决方案，但到目前为止对我没有任何帮助。
另一个非常相似的问题：How to handle lazy-loaded images in selenium?

这个网站的问题是，除非提要项目在视口中，否则单个项目的 img 标签通常被替换为带有类名“lazyload-placeholder”。我尝试使用 selenium 滚动到每个页面的底部并抓取图像，但它没有奏效，因为页面顶部的图像再次延迟加载。
我尝试过的另一种可行的解决方案是逐步向下滚动并在每次滚动后进行刮擦，但是，对于大约 10k 元素执行此操作花费了太长时间。我想知道是否有更快/更有效的方法来做到这一点？

这就是我当前滚动的方式：

driver.find_element_by_tag_name('body').send_keys(Keys.PAGE_DOWN)

我正在使用 bs4 从每个项目中抓取数据：（项目是提要中的一个元素）

try:
    img = item.find('img')['src']
except TypeError:
    img = "N/A"
    print(item)

Answer 1

为此使用 BeautifulSoup 或 Selenium 远远超出了要求，而且，我相信您已经自己发现，尝试用于这个特定用例非常麻烦。

更简单、更简洁的做法是：如果您打开浏览器的网络流量记录器，并且只查看 XHR (XmlHttpRequest) 请求，您会看到每次向下滚动并开始加载新产品时，您的浏览器向此 API 发出 HTTP POST 请求：https://mnrwefss2q-dsn.algolia.net/1/indexes/*/queries

如果您只是简单地模仿该 API 的 POST 请求，使用相同的查询字符串和 POST 表单数据，您可以获得您可能想要的所有产品信息，包括产品图片的 URL - 并且全部采用 JSON。

无论出于何种原因，API 都不关心请求标头，但这没关系。它只关心查询字符串和 POST 表单数据。您还可以更改 hitsPerPage 键值对以更改请求的产品数量。默认情况下，它似乎每次加载 40 个新产品，但您可以将该数字更改为任何您想要的：

def main():

    import requests
    from urllib.parse import urlencode

    url = "https://mnrwefss2q-dsn.algolia.net/1/indexes/*/queries"

    params = {
        "x-algolia-agent": "Algolia for JavaScript (3.35.1); Browser; react (16.13.1); react-instantsearch (6.6.0); JS Helper (3.1.2)",
        "x-algolia-application-id": "MNRWEFSS2Q",
        "x-algolia-api-key": "a3a4de2e05d9e9b463911705fb6323ad"
    }

    post_json = {
        "requests":[
            {
                "indexName": "Listing_production",
                "params": urlencode({
                    "highlightPreTag": "<ais-highlight-0000000000>",
                    "highlightPostTag": "</ais-highlight-0000000000>",
                    "maxValuesPerFacet": "100",
                    "hitsPerPage": "40",
                    "filters": "",
                    "page": "4",
                    "query": "",
                    "facets": "[\"designers.name\",\"category_path\",\"category_size\",\"price_i\",\"condition\",\"location\",\"badges\",\"strata\"]",
                    "tagFilters": "",
                    "facetFilters": "[[\"category_path:footwear.hitop_sneakers\"],[\"designers.name:Jordan Brand\"]]",
                    "numericFilters": "[\"price_i>=0\",\"price_i<=99999\"]"
                })
            }
        ]
    }

    response = requests.post(url, params=params, json=post_json)
    response.raise_for_status()

    results = response.json()["results"]
    items = results[0]["hits"]

    for item in items:
        print(f"{item['title']} - price: ${item['price']}")
        print(f"Image URL: \"{item['cover_photo']['url']}\"\n")

    return 0


if __name__ == "__main__":
    import sys
    sys.exit(main())

输出：

Air Jordan 13 Retro Grey Toe 2014 - price: $150
Image URL: "https://cdn.fs.grailed.com/api/file/HZfvq06fSYOZWvB6OTxA"

Air Jordan 5 Retro Grape 2013 - price: $300
Image URL: "https://cdn.fs.grailed.com/api/file/nwQmfUzITOSVa2Qg5gCt"

Air Jordan 11 BG Legend Blue - price: $243
Image URL: "https://cdn.fs.grailed.com/api/file/AKYdESePSdK1XqDEqYMr"

Air Jordan 12 Retro Cool Grey 2012 - price: $200
Image URL: "https://cdn.fs.grailed.com/api/file/oAl5cxdCSPyCQaXUKPRm"

Air Jordan 11 Retro GS Space Jam 2009 - price: $162
Image URL: "https://cdn.fs.grailed.com/api/file/oRAMFOMTeu9fkWp5l640"

Jordan 1 Flight Mens High Tops Shoes - Size 10.5 White - price: $50
Image URL: "https://cdn.fs.grailed.com/api/file/hRZFggEyRImxzi1T623p"

Air Jordan 1 Retro High OG Royal Toe Sz 8.5 - price: $400
Image URL: "https://cdn.fs.grailed.com/api/file/MWwBxHyNRDCuDZ4Pc2LC"

Air Jordan 14 GS Black Toe 2014 Size 4Y (5.5 Womans) - price: $58
Image URL: "https://cdn.fs.grailed.com/api/file/FzV1GrMGSRqPjV0dPFcB"

Air Jordan 5 Retro Fire Red 2020 - price: $250
Image URL: "https://cdn.fs.grailed.com/api/file/vGda4X6qReBq42muTatG"

Air Jordan 6 Retro All Star Chameleon - price: $70
Image URL: "https://cdn.fs.grailed.com/api/file/wk1ySwZDQqWHtCqvi9I0"

Air Jordan 5 Retro GS Oreo - price: $64
Image URL: "https://cdn.fs.grailed.com/api/file/rWrcrhdiSBG4hvRZ53aS"

Air Jordan 11 Retro GS Bred 2012 - price: $76
Image URL: "https://cdn.fs.grailed.com/api/file/ceUppZDSo6YgvPM6OoMg"

Air Jordan 1 Mid (GS) 6Y (5.5 Uk) - price: $115
Image URL: "https://cdn.fs.grailed.com/api/file/cRFIYqE8TKOSAiKaV2uA"

Air Jordan 7 Retro GS Pure Money 3.5Y - price: $87
Image URL: "https://cdn.fs.grailed.com/api/file/uXZIVZMQQain0pUSyJIe"

Air Jordan 5 Retro Olympic 2011 - price: $120
Image URL: "https://cdn.fs.grailed.com/api/file/D7E4GvaJSiywv3vaz3A5"

J12 Grey/University Blue - price: $117
Image URL: "https://cdn.fs.grailed.com/api/file/0T9GNBSUTDSGQsqrg4IS"

1994 Jordan 1 Bred - price: $801
Image URL: "https://cdn.fs.grailed.com/api/file/rfQ68e8PRDOpKN6zwwyw"

Nike Air Jordan Retro 13 He Got Game HGG - price: $90
Image URL: "https://cdn.fs.grailed.com/api/file/k9N12MAQnKcKk3Vl0dek"

Nike Air Jordan Retro 13 XIII Grey Toe He Got Game Playoff - price: $90
Image URL: "https://cdn.fs.grailed.com/api/file/wo1hKnUQeKt0lioebLgG"

Nike Air Jordan 1 Retro Countdown Pack 2008 Vintage - price: $429
Image URL: "https://cdn.fs.grailed.com/api/file/ktAH2VmbTgaG6zdtzHrw"

Air Jordan 1 Retro High OG Bred Toe - price: $495
Image URL: "https://cdn.fs.grailed.com/api/file/a41NTCSXTgGm7KyqkTqB"

Air Jordan 10 Retro 2018 Orlando 2018 - price: $121
Image URL: "https://cdn.fs.grailed.com/api/file/dMAbTBYVSYSX6KNqVvQA"

Air Jordan 12 Retro Winterized Triple Black 2018 - price: $84
Image URL: "https://cdn.fs.grailed.com/api/file/Jg69Af0QrelcZxWM2OEW"

Air Jordan 6 Retro Diffused Blue 2018 - price: $145
Image URL: "https://cdn.fs.grailed.com/api/file/bVPC1SomTKO3yPLCm0rC"

AIR JORDAN 7 RETRO "2005 CARDINAL" - price: $57
Image URL: "https://cdn.fs.grailed.com/api/file/QzvDCMVRqykY1CnkrMwC"

Air Jordan 5 Retro Camo 2017 - price: $220
Image URL: "https://cdn.fs.grailed.com/api/file/eLDeagz4TF6Yu5aItPAE"

Air Jordan 1 Retro High Grand Purple 2009 - price: $261
Image URL: "https://cdn.fs.grailed.com/api/file/XhhIFIQ5SyjNQkEXpQjK"

Air Jordan 6 Retro 2015 Maroon 2015 - price: $150
Image URL: "https://cdn.fs.grailed.com/api/file/5UTk9ctnTnSsZEixeYwW"

ORIGINAL 1985 WEARABLE Chicago OG Air Jordan 1 Last Dance - price: $2268
Image URL: "https://cdn.fs.grailed.com/api/file/CE4eMehmQvOYtpx16R3X"

Air Jordan 5 Retro Top 3 - price: $247
Image URL: "https://cdn.fs.grailed.com/api/file/B1A8oopSR226r2rmOjfR"

AIR JORDAN 4 METALLIC RED - price: $350
Image URL: "https://cdn.fs.grailed.com/api/file/ZHJVHw7fRC2jO5wDj7JF"

Air Jordan 1 Retro Mid Size 7 GS White Gym Red - price: $67
Image URL: "https://cdn.fs.grailed.com/api/file/XPMpEjlZSmCNikSeHjg4"

Air Jordan 5 Retro Metallic White 2015 Size 13 - price: $57
Image URL: "https://cdn.fs.grailed.com/api/file/LHvMnyxxQaOCyYiTSvHw"

Air Jordan 8 Retro C&C Trophy 2016 - price: $115
Image URL: "https://cdn.fs.grailed.com/api/file/7sh2iU42T6kUKjJaAyE8"

JIRDAN 7 UNIVERSITY BLUE - price: $120
Image URL: "https://cdn.fs.grailed.com/api/file/zJzKkd1QyKvMasw1ZWRB"

AIR JORDAN 3 5LAB3 - price: $70
Image URL: "https://cdn.fs.grailed.com/api/file/IJvMAIeMTPycUk7n0idU"

Jordan 6 rings UNC - price: $189
Image URL: "https://cdn.fs.grailed.com/api/file/nE6A02dKQBa7ZTSJWGu3"

Nike Air Jordan 5 Retro Top 3 - price: $255
Image URL: "https://cdn.fs.grailed.com/api/file/UfcVOOO0QJSM6csHc1cK"

Nike Air Jordan 13 White Pink Soar Aurora Green (GS) - price: $155
Image URL: "https://cdn.fs.grailed.com/api/file/58rVbApeR8K8Ojd1Zskg"

Jordan Thunder 4 Retro 2012 - price: $350
Image URL: "https://cdn.fs.grailed.com/api/file/ZJ1EuskfThGImbij5QkS"

>>> 

编辑 - 这是查看其他 API 的更新代码：

def main():

    import requests
    from urllib.parse import urlencode

    url = "https://mnrwefss2q-1.algolianet.com/1/indexes/Listing_production/browse"

    params = {
        "x-algolia-agent": "Algolia for JavaScript (3.35.1); Browser",
        "x-algolia-application-id": "MNRWEFSS2Q",
        "x-algolia-api-key": "a1c6338ffe41249d0284a5a1105eafe4"
    }

    post_json = {
        "params": "query=&" + urlencode({
            "offset": "0",
            "length": "100",
            "facetFilters": "[[\"category_path:footwear.hitop_sneakers\"], [\"designers.name:Jordan Brand\"]]",
            "filters": ""
        })
    }

    response = requests.post(url, params=params, json=post_json)
    response.raise_for_status()

    items = response.json()["hits"]

    # ...

    return 0


if __name__ == "__main__":
    import sys
    sys.exit(main())