读取 CONNECT 标头

Question

我正在使用代理服务 (proxymesh)，它将有用的信息放入响应 CONNECT 请求而发送的标头中。无论出于何种原因，Python's httplib doesn't parse them：

> CONNECT example.com:443 HTTP/1.1
> Host: example.com:443
>
< HTTP/1.1 200 Connection established
< X-Useful-Header: value  # completely ignored
<

requests 模块在内部使用httplib，因此它也会忽略它们。如何从CONNECT 请求中提取标头？

Answer 1

Python 的httplib 在创建隧道时实际上会忽略这些标头。这很 hacky，但您可以拦截它们并将“标头”行与实际 HTTP 响应的标头合并：

import socket
import httplib
import requests

from requests.packages.urllib3.connection import HTTPSConnection
from requests.packages.urllib3.connectionpool import HTTPSConnectionPool
from requests.packages.urllib3.poolmanager import ProxyManager

from requests.adapters import HTTPAdapter


class ProxyHeaderHTTPSConnection(HTTPSConnection):
    def __init__(self, *args, **kwargs):
        super(ProxyHeaderHTTPSConnection, self).__init__(*args, **kwargs)
        self._proxy_headers = []

    def _tunnel(self):
        self.send("CONNECT %s:%d HTTP/1.0\r\n" % (self._tunnel_host, self._tunnel_port))

        for header, value in self._tunnel_headers.iteritems():
            self.send("%s: %s\r\n" % (header, value))

        self.send("\r\n")

        response = self.response_class(self.sock, strict=self.strict, method=self._method)
        version, code, message = response._read_status()

        if version == "HTTP/0.9":
            # HTTP/0.9 doesn't support the CONNECT verb, so if httplib has
            # concluded HTTP/0.9 is being used something has gone wrong.
            self.close()
            raise socket.error("Invalid response from tunnel request")

        if code != 200:
            self.close()
            raise socket.error("Tunnel connection failed: %d %s" % (code, message.strip()))

        self._proxy_headers = []

        while True:
            line = response.fp.readline(httplib._MAXLINE + 1)

            if len(line) > httplib._MAXLINE:
                raise LineTooLong("header line")

            if not line or line == '\r\n':
                break

            # The line is a header, save it
            if ':' in line:
                self._proxy_headers.append(line)

    def getresponse(self, buffering=False):
        response = super(ProxyHeaderHTTPSConnection, self).getresponse(buffering)
        response.msg.headers.extend(self._proxy_headers)

        return response


class ProxyHeaderHTTPSConnectionPool(HTTPSConnectionPool):
    ConnectionCls = ProxyHeaderHTTPSConnection


class ProxyHeaderProxyManager(ProxyManager):
    def _new_pool(self, scheme, host, port):
        assert scheme == 'https'

        return ProxyHeaderHTTPSConnectionPool(host, port, **self.connection_pool_kw)


class ProxyHeaderHTTPAdapter(HTTPAdapter):
    def proxy_manager_for(self, proxy, **proxy_kwargs):
        if proxy in self.proxy_manager:
            manager = self.proxy_manager[proxy]
        else:
            proxy_headers = self.proxy_headers(proxy)
            manager = self.proxy_manager[proxy] = ProxyHeaderProxyManager(
                proxy_url=proxy,
                proxy_headers=proxy_headers,
                num_pools=self._pool_connections,
                maxsize=self._pool_maxsize,
                block=self._pool_block,
                **proxy_kwargs)

        return manager

然后您可以将适配器安装到会话中：

session = requests.Session()
session.mount('https://', ProxyHeaderHTTPAdapter())

response = session.get('https://example.com', proxies={...})

代理的标头将与响应标头合并，因此它应该表现得好像代理直接修改了响应标头。