如何从 url 列表中抓取图像并将其下载到本地文件夹？答案

【问题标题】：How to I scrape images from a list of urls and download it to local folder?如何从 url 列表中抓取图像并将其下载到本地文件夹？
【发布时间】：2020-06-06 18:33:30
【问题描述】：

我正在遍历图像的 url 列表。

import requests

list_url = ["www.abc.com/def.jpg", " www.abc.com/def1.jpg",... "www.abc.com/def100000.jpg"]

correct_img_list = []

for img in list_url:

    request = requests.get(img)
    if request.status_code == 200:
        correct_img_list.append(img)
        continue
    else:
        continue
i = 1 

for img in correct_img_list:
    urllib.request.urlretrieve(img, "local_image file_" + str(i))
    i += 1

我希望它通过列表，获取 url，然后将图像下载到本地目录。

我看到的错误是这样的：

Traceback (most recent call last):
  File "c:/Users/liuh9/Desktop/google chrome folder/image-downloader.py", line 714, in <module>
    urllib.request.urlretrieve(img, "local_image file " + str(i))
  File "C:\Users\liuh9\AppData\Local\Programs\Python\Python37\lib\urllib\request.py", line 247, in urlretrieve
    with contextlib.closing(urlopen(url, data)) as fp:
  File "C:\Users\liuh9\AppData\Local\Programs\Python\Python37\lib\urllib\request.py", line 222, in urlopen
    return opener.open(url, data, timeout)
  File "C:\Users\liuh9\AppData\Local\Programs\Python\Python37\lib\urllib\request.py", line 531, in open
    response = meth(req, response)
  File "C:\Users\liuh9\AppData\Local\Programs\Python\Python37\lib\urllib\request.py", line 641, in http_response
    'http', request, response, code, msg, hdrs)
  File "C:\Users\liuh9\AppData\Local\Programs\Python\Python37\lib\urllib\request.py", line 569, in error
    return self._call_chain(*args)
  File "C:\Users\liuh9\AppData\Local\Programs\Python\Python37\lib\urllib\request.py", line 503, in _call_chain
    result = func(*args)
  File "C:\Users\liuh9\AppData\Local\Programs\Python\Python37\lib\urllib\request.py", line 649, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 404: Not Found

非常感谢！

【问题讨论】：

图片网址好像不存在。您确定列表中的所有 URL 都有效吗？如果是这样，那么您应该在调用 URLretrieve 之前打印 URL，以确保您正在检索正确的 url
@chatax 我如何获得 url 的 http 状态
您可以在浏览器中访问该 URL。我会在检索 URL 之前运行您的代码并打印请求的 URL。比尝试访问给您找不到 404 的 URL
for img in list_url: request = requests.get(img) if request.status_code == 200: continue else: print(img) print('网站不存在') list_url.remove( img)
您从该错误消息中了解/不了解什么？

标签： python python-3.x web-scraping urllib

【解决方案1】：

你得到的响应码是404，表示该URL不存在。

作为一种好的做法，请确保检查响应代码并仅在响应代码为 200 时执行进一步操作。我还建议在 try except 块中添加操作，以便程序不会在第一个无效 URL 上崩溃.

看下面的例子

import urllib.request


url_list = ["https://cdn.pixabay.com/photo/2016/09/07/11/37/tropical-1651426__340.jpg", "https://image.shutterstock.com/image-photo/long-exposure-soft-colorful-sunset-260nw-142504771.jpg"]
filename = 1

for url in url_list:
    try:
        urllib.request.urlretrieve(url, f'{filename}.jpg')
        filename += 1
    except Exception as exc:
        print(f"Exception occued while downloading image from url {url} {str(exc)}")

【讨论】：

for img in list_url: request = requests.get(img) if request.status_code == 200: continue else: print(img) print('网站不存在') list_url.remove( img)