使用 urllib / sock.sendall(data) OSError 上传大文件（约 3GB）答案

【问题标题】：Large file(about 3GB) upload with urllib / sock.sendall(data) OSError使用 urllib / sock.sendall(data) OSError 上传大文件（约 3GB）
【发布时间】：2016-05-05 23:30:49
【问题描述】：

环境：Mac OS X El Capitan / python 3.5.1

我想上传大约 3GB 大小的文件。

def read_in_chunks(file_object, chunk_size=4096):
    while True:
        data = file_object.read(chunk_size)
        if not data:
            break
        yield data

with open('3GB.mov', 'br') as f:
    data = b''.join([chunk for chunk in read_in_chunks(f)])

req = urllib.request.Request(url, data, headers)
response = urllib.request.urlopen(req)
the_page = response.read()

问题是..

Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/requests/packages/urllib3/connectionpool.py", line 578, in urlopen
    chunked=chunked)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/requests/packages/urllib3/connectionpool.py", line 362, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 1083, in request
    self._send_request(method, url, body, headers)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 1128, in _send_request
    self.endheaders(body)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 1079, in endheaders
    self._send_output(message_body)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 913, in _send_output
    self.send(message_body)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 885, in send
    self.sock.sendall(data)
OSError: [Errno 22] Invalid argument

你能给我一些建议吗？

【问题讨论】：

相关：Python: HTTP Post a large file with streaming
无关：您不应该将整个文件加载到内存中。但如果你这样做，那么使用data = f.read() 而不是data = b"".join(...)
无关：stackoverflow.com/questions/11662960/… 我不能在 python3 的大文件（大约
我确实说过在这种情况下你不应该一次加载整个文件:) 原因可能是相同的（OSError: [Errno 22] Invalid argument 在 OSX 上请求具有大数据块的 IO）。正如第一个链接所建议的那样，这很容易解决：response = urlopen(Request(url, f, {'Content-Length': os.path.getsize(filename)}))
好的，我明白了。太感谢了。我会试试的！！

标签： python urllib large-file-upload

【解决方案1】：

import requests
url = 'http://domain.com/api/upload'
with open('3GB.mov', 'br') as f:
    for chunk in read_in_chunks(f):

        offset = index + len(chunk)
        headers['Content-Type'] = 'application/octet-stream'
        headers['Content-length'] = content_size
        headers['Content-Range'] = 'bytes %s-%s/%s' % (index, offset, content_size)
        index = offset
        try:
            r = requests.post(url, data=chunk, headers=headers)
            print "r: %s, Content-Range: %s" % (r, headers['Content-Range'])
        except Exception, e:
            print e

【讨论】：

你可以检查req.get_method()，嗯，也许你是对的
这是个好主意。但我无法在我的 api 服务器中使用范围请求。谢谢你的建议，Janom。