使用 python tor 的一般 SOCKS 服务器故障，但在 tor 浏览器中工作

Question

我想使用 python 中的 tor 来自动化请求。 我用一个页面做了一个测试来检查IP，它可以工作。

然后我指向了我想要的站点，显然他们避免了 tor 端点，因为（请参阅下面的堆栈跟踪） - 但它可以在 tor 浏览器中使用。

有更好的方法来调试浏览器的响应吗？ （例如连接被拒绝）

从python而不是浏览器查询我缺少哪些东西？

我正在尝试类似的东西：

socks.setdefaultproxy(socks.PROXY_TYPE_SOCKS5 , "127.0.0.1", 9150, True)
socket.socket = socks.socksocket

socks.wrapmodule(requests)
url = "myexample.com"
# r = requests.Session()
s = requests.get(url)
print s


    Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 127, in urlopen
    return _opener.open(url, data, timeout)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 404, in open
    response = self._open(req, data)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 422, in _open
    '_open', req)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 382, in _call_chain
    result = func(*args)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 1222, in https_open
    return self.do_open(httplib.HTTPSConnection, req)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 1181, in do_open
    h.request(req.get_method(), req.get_selector(), req.data, headers)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 973, in request
    self._send_request(method, url, body, headers)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 1007, in _send_request
    self.endheaders(body)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 969, in endheaders
    self._send_output(message_body)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 829, in _send_output
    self.send(msg)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 791, in send
    self.connect()
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 1172, in connect
    self.timeout, self.source_address)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/socket.py", line 562, in create_connection
    sock.connect(sa)
  File "/Library/Python/2.7/site-packages/socks.py", line 747, in connect
    negotiate(self, dest_addr, dest_port)
  File "/Library/Python/2.7/site-packages/socks.py", line 419, in _negotiate_SOCKS5
    CONNECT, dest_addr)
  File "/Library/Python/2.7/site-packages/socks.py", line 494, in _SOCKS5_request
    raise SOCKS5Error("{0:#04x}: {1}".format(status, error))
socks.SOCKS5Error: 0x01: General SOCKS server failure

Answer 1

即使您使用 Tor 某些网站正在寻找用户代理。尝试将用户代理标头添加到您的请求中。我遇到了同样的问题。

这对我有用

from urllib.request import Request, urlopen
from bs4 import BeautifulSoup
import random
from fake_useragent import UserAgent
from torrequest import TorRequest
import time, socks, socket
from stem import Signal
from stem.control import Controller


ua = UserAgent()


with Controller.from_port(port = 9051) as controller:
    controller.authenticate(password = 'YourPasswordHere')
    socks.setdefaultproxy(socks.PROXY_TYPE_SOCKS5, "127.0.0.1", 9050)
    socket.socket = socks.socksocket   

    controller.signal(Signal.NEWNYM)
    if controller.is_newnym_available() == False:
        print("Waitting time for Tor to change IP: "+ str(controller.get_newnym_wait()) +" seconds")
        time.sleep(controller.get_newnym_wait())
    req = Request(url)
    req.add_header('User-Agent', ua.random)
    req_doc= urlopen(req)#.read().decode('utf8')
    print(req_doc)