python3 URLError 未知 url 类型 http

Question

我正在尝试将 urllib.request.urlretrieve 与多处理模块一起使用来下载一些文件并对它们进行一些处理。但是，每次我尝试运行我的程序时，它都会给我一个错误：

multiprocessing.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/usr/lib/python3.4/multiprocessing/pool.py", line 119, in worker
    result = (True, func(*args, **kwds))
  File "/usr/lib/python3.4/multiprocessing/pool.py", line 44, in mapstar
    return list(map(*args))
  File "./thumb.py", line 13, in download_and_convert
    filename, headers = urlretrieve(url)
  File "/usr/lib/python3.4/urllib/request.py", line 186, in urlretrieve
    with contextlib.closing(urlopen(url, data)) as fp:
  File "/usr/lib/python3.4/urllib/request.py", line 161, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib/python3.4/urllib/request.py", line 463, in open
    response = self._open(req, data)
  File "/usr/lib/python3.4/urllib/request.py", line 486, in _open
    'unknown_open', req)
  File "/usr/lib/python3.4/urllib/request.py", line 441, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.4/urllib/request.py", line 1252, in unknown_open
    raise URLError('unknown url type: %s' % type)
urllib.error.URLError: <urlopen error unknown url type: http>
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "./thumb.py", line 27, in <module>
    pool.map(download_and_convert, enumerate(csvr))
  File "/usr/lib/python3.4/multiprocessing/pool.py", line 260, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/usr/lib/python3.4/multiprocessing/pool.py", line 599, in get
    raise self._value
urllib.error.URLError: <urlopen error unknown url type: http>

它似乎窒息的网址是http://phytoimages.siu.edu/users/vitt/10_27_06_2/Equisetumarvense.JPG。这是我的代码：

#!/usr/bin/env python3

from subprocess import Popen
from sys import argv, stdin
import csv
from multiprocessing import Pool
from urllib.request import urlretrieve

def download_and_convert(args):
    num, url_list = args
    url = url_list[0]
    try:
        filename, headers = urlretrieve(url)
    except:
        print(url)
        raise
    Popen(["convert", filename, "-resize", "250x250",\
           str(num)+'.'+url.split('.')[-1]])

if __name__ == "__main__":
    csvr = csv.reader(open(argv[1]))

    if(len(argv) > 2): nprocs = argv[2]
    else: nprocs = None

    pool = Pool(processes=nprocs)
    pool.map(download_and_convert, enumerate(csvr))

我不知道为什么会出现这个错误。可能是因为我正在使用多处理？如果有人可以帮助我，将不胜感激。

编辑：这是它尝试处理的第一个 url，如果我更改它，它不会更改错误。

Answer 1

检查此代码sn-p

>>> import urllib.parse
>>> urllib.parse.quote(':')
'%3A'

如您所见，urllib 奇怪地解释了 ':' 字符。巧合的是，您的程序挂断了您。

试试urllib.parse.urlencode()，应该会让你走上正轨。

Answer 2

在 cmets 的一些帮助下，我找到了解决方案。问题似乎是 csv 模块在字节顺序标记 (BOM) 上被绊倒。我可以通过encoding='utf-8-sig' 建议here 打开文件来修复它。