使用 python 从 Intranet 下载文件

Question

我想从我的 Intranet 下载一系列 pdf 文件。我可以毫无问题地在我的网络浏览器中查看文件，但是当尝试通过 python 自动提取文件时，我遇到了问题。通过我办公室设置的代理交谈后，我可以很容易地通过answer从互联网下载文件：

url = 'http://www.sample.com/fileiwanttodownload.pdf'

user = 'username'
pswd = 'password'
proxy_ip = '12.345.56.78:80'
proxy_url = 'http://' + user + ':' + pswd + '@' + proxy_ip
proxy_support = urllib2.ProxyHandler({"http":proxy_url})
opener = urllib2.build_opener(proxy_support,urllib2.HTTPHandler)
urllib2.install_opener(opener)

file_name = url.split('/')[-1]
u = urllib2.urlopen(url)
f = open(file_name, 'wb')
f.close()

但无论出于何种原因，如果 url 指向我的 Intranet 上的某些内容，它将无法正常工作。返回以下错误：

Traceback (most recent call last):

  File "<ipython-input-13-a055d9eaf05e>", line 1, in <module>
    runfile('C:/softwaredev/python/pdfwrite.py', wdir='C:/softwaredev/python')

  File "C:\Anaconda\lib\site-packages\spyderlib\widgets\externalshell\sitecustomize.py", line 585, in runfile
    execfile(filename, namespace)

  File "C:/softwaredev/python/pdfwrite.py", line 26, in <module>
    u = urllib2.urlopen(url)

  File "C:\Anaconda\lib\urllib2.py", line 127, in urlopen
    return _opener.open(url, data, timeout)

  File "C:\Anaconda\lib\urllib2.py", line 410, in open
    response = meth(req, response)

  File "C:\Anaconda\lib\urllib2.py", line 523, in http_response
    'http', request, response, code, msg, hdrs)

  File "C:\Anaconda\lib\urllib2.py", line 442, in error
    result = self._call_chain(*args)

  File "C:\Anaconda\lib\urllib2.py", line 382, in _call_chain
    result = func(*args)

  File "C:\Anaconda\lib\urllib2.py", line 629, in http_error_302
    return self.parent.open(new, timeout=req.timeout)

  File "C:\Anaconda\lib\urllib2.py", line 410, in open
    response = meth(req, response)

  File "C:\Anaconda\lib\urllib2.py", line 523, in http_response
    'http', request, response, code, msg, hdrs)

  File "C:\Anaconda\lib\urllib2.py", line 448, in error
    return self._call_chain(*args)

  File "C:\Anaconda\lib\urllib2.py", line 382, in _call_chain
    result = func(*args)

  File "C:\Anaconda\lib\urllib2.py", line 531, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)

HTTPError: Service Unavailable

在下面的代码中使用requests.py，我可以成功地从互联网上下载文件，但是当我试图从我的办公室内部网中提取一个pdf文件时，我只是收到一个以html形式发回给我的连接错误。运行以下代码：

import requests

url = 'www.intranet.sample.com/?layout=attachment&cfapp=26&attachmentid=57142'

proxies = {
  "http": "http://12.345.67.89:80",
  "https": "http://12.345.67.89:80"
}

local_filename = 'test.pdf'
r = requests.get(url, proxies=proxies, stream=True)
with open(local_filename, 'wb') as f:
    for chunk in r.iter_content(chunk_size=1024): 
        print chunk
        if chunk:
            f.write(chunk)
            f.flush()

还有返回的html：

Network Error (tcp_error) 

A communication error occurred: "No route to host"
The Web Server may be down, too busy, or experiencing other problems preventing it from responding to requests. You may wish to try again at a later time.

For assistance, contact your network support team.

是否可能有一些网络安全设置阻止了网络浏览器环境之外的自动请求？

Answer 1

在 urllib2 中安装 opener 不会影响请求。您需要使用请求自己的代理支持。将它们在proxies 参数中传递给get 就足够了，或者您可以设置HTTP_PROXY 和HTTPS_PROXY 环境变量。见http://docs.python-requests.org/en/latest/user/advanced/#proxies

import requests

proxies = {
  "http": "http://10.10.1.10:3128",
  "https": "http://10.10.1.10:1080",
}

requests.get("http://example.org", proxies=proxies)

Answer 2

您是否尝试过在内网上不使用代理来下载文件？

你可以在 python2 中尝试这样的事情

from urllib2 import urlopen

url = 'http://intranet/myfile.pdf'

with open(local_filename, 'wb') as f:
    f.write(urlopen(url).read())