如何在 selenium chromedriver python 中设置具有身份验证的代理？

Question

我正在创建一个脚本来抓取一个网站以收集一些数据，但问题是他们在太多请求后阻止了我，但使用代理我可以发送比目前更多的请求。我已将代理与 chrome 选项集成在一起 --proxy-server

options.add_argument('--proxy-server={}'.format('http://ip:port'))

但我使用的是付费代理，因此它需要身份验证，如下图所示，它提供了用户名和密码的警报框

然后我尝试用用户名和密码来使用它

options.add_argument('--proxy-server={}'.format('http://username:password@ip:port'))

但它似乎也不起作用。我正在寻找一个解决方案，并在下面找到了解决方案，我将它与 chrome 扩展 proxy auto auth 一起使用，而没有 chrome 扩展

proxy = {'address': settings.PROXY,
             'username': settings.PROXY_USER,
             'password': settings.PROXY_PASSWORD}

capabilities = dict(DesiredCapabilities.CHROME)
capabilities['proxy'] = {'proxyType': 'MANUAL',
                             'httpProxy': proxy['address'],
                             'ftpProxy': proxy['address'],
                             'sslProxy': proxy['address'],
                             'noProxy': '',
                             'class': "org.openqa.selenium.Proxy",
                             'autodetect': False,
                             'socksUsername': proxy['username'],
                             'socksPassword': proxy['password']}
options.add_extension(os.path.join(settings.DIR, "extension_2_0.crx")) # proxy auth extension

但以上都不能正常工作，它似乎工作，因为在上面的代码之后代理身份验证警报消失了，当我通过谷歌搜索我的 IP 并确认它不起作用时。

请任何可以帮助我在 chromedriver 上验证代理服务器的人。

Answer 1

我一直在寻找相同的答案，但只针对 Java 代码，所以这是我的 @itsmnthn Python 代码变体。

不要忘记将 MainTest 类的 String 字段更改为您的 ip、端口、登录名、密码和 chromeDriver 路径。

import org.openqa.selenium.WebDriver;
import org.openqa.selenium.chrome.ChromeDriver;
import org.openqa.selenium.chrome.ChromeOptions;

import java.io.*;
import java.util.zip.ZipEntry;
import java.util.zip.ZipOutputStream;

public class MainTest {
    private static final String PROXY_HOST = "127.0.0.1";
    private static final String PROXY_PORT = "8080";
    private static final String PROXY_USER = "login";
    private static final String PROXY_PASS = "password";
    private static final String CHROMEDRIVER_PATH = "chromeDriverPath";
    private static final String PROXY_OPTION_TEMPLATE = "--proxy-server=http://%s";

    public static void main(String[] args) throws IOException {
        System.setProperty("webdriver.chrome.driver", CHROMEDRIVER_PATH);
        ChromeOptions options = new ChromeOptions();
        String manifest_json = "{\n" +
                "  \"version\": \"1.0.0\",\n" +
                "  \"manifest_version\": 2,\n" +
                "  \"name\": \"Chrome Proxy\",\n" +
                "  \"permissions\": [\n" +
                "    \"proxy\",\n" +
                "    \"tabs\",\n" +
                "    \"unlimitedStorage\",\n" +
                "    \"storage\",\n" +
                "    \"<all_urls>\",\n" +
                "    \"webRequest\",\n" +
                "    \"webRequestBlocking\"\n" +
                "  ],\n" +
                "  \"background\": {\n" +
                "    \"scripts\": [\"background.js\"]\n" +
                "  },\n" +
                "  \"minimum_chrome_version\":\"22.0.0\"\n" +
                "}";

        String background_js = String.format("var config = {\n" +
                "  mode: \"fixed_servers\",\n" +
                "  rules: {\n" +
                "    singleProxy: {\n" +
                "      scheme: \"http\",\n" +
                "      host: \"%s\",\n" +
                "      port: parseInt(%s)\n" +
                "    },\n" +
                "    bypassList: [\"localhost\"]\n" +
                "  }\n" +
                "};\n" +
                "\n" +
                "chrome.proxy.settings.set({value: config, scope: \"regular\"}, function() {});\n" +
                "\n" +
                "function callbackFn(details) {\n" +
                "return {\n" +
                "authCredentials: {\n" +
                "username: \"%s\",\n" +
                "password: \"%s\"\n" +
                "}\n" +
                "};\n" +
                "}\n" +
                "\n" +
                "chrome.webRequest.onAuthRequired.addListener(\n" +
                "callbackFn,\n" +
                "{urls: [\"<all_urls>\"]},\n" +
                "['blocking']\n" +
                ");", PROXY_HOST, PROXY_PORT, PROXY_USER, PROXY_PASS);

        FileOutputStream fos = new FileOutputStream("proxy_auth_plugin.zip");
        ZipOutputStream zipOS = new ZipOutputStream(fos);

        createFile("manifest.json", manifest_json);
        createFile("background.js", background_js);

        File file = new File("proxy_auth_plugin.zip");
        writeToZipFile("manifest.json", zipOS);
        writeToZipFile("background.js", zipOS);
        zipOS.close();
        fos.close();
        options.addExtensions(file);

        WebDriver driver = new ChromeDriver(options);
        try {
            driver.get("https://2ip.ru");
        } finally {
            driver.close();
        }

    }

    public static void writeToZipFile(String path, ZipOutputStream zipStream) throws FileNotFoundException, IOException {
        System.out.println("Writing file : '" + path + "' to zip file");
        File aFile = new File(path);
        FileInputStream fis = new FileInputStream(aFile);
        ZipEntry zipEntry = new ZipEntry(path);
        zipStream.putNextEntry(zipEntry);
        byte[] bytes = new byte[1024];
        int length;
        while ((length = fis.read(bytes)) >= 0) {
            zipStream.write(bytes, 0, length);
        }
        zipStream.closeEntry();
        fis.close();
    }

    public static void createFile(String filename, String text) throws FileNotFoundException {
        try (PrintWriter out = new PrintWriter(filename)) {
            out.println(text);
        }
    }


}

Answer 2

我遇到了同样的问题 - 是否可以将 selenium-wire 函数与选项中的无头函数结合起来 - 例如，这段代码对我来说是有效的 - 有什么问题吗？

from seleniumwire import webdriver
from selenium.webdriver.chrome.options import Options
import os, sys, time
from dotenv import load_dotenv, find_dotenv

path = os.path.abspath (os.path.dirname (sys.argv[0]))
cd = '/chromedriver.exe'
load_dotenv(find_dotenv()) 
PROXY_CHEAP_USER = os.environ.get("PROXY_CHEAP_USER")
PROXY_CHEAP_PW= os.environ.get("PROXY_CHEAP_PW")
PROXY_HOST = 'proxyhost.com'  # rotating proxy or host
PROXY_PORT = port # port
PROXY_USER = PROXY_CHEAP_USER # username
PROXY_PASS = PROXY_CHEAP_PW # password

options = Options()
options.add_argument('--headless')
options.add_argument("--window-size=1920x1080")
options.add_argument('--no-sandbox')
options.add_argument('--disable-gpu')

options_seleniumWire = {
    'proxy': {
        'https': f'https://{PROXY_USER}:{PROXY_PASS}@{PROXY_HOST}:{PROXY_PORT}',
    }
}
 
driver = webdriver.Chrome (path + cd, options=options, seleniumwire_options=options_seleniumWire)
driver.get("https://ifconfig.co/")

我认为这个解决方案也适用于无头模式。

Answer 3

使用selenium-wire。

文档中的示例代码：

options = {
   'proxy': {
        'http': 'socks5://user:pass@192.168.10.100:8888',
        'https': 'socks5://user:pass@192.168.10.100:8888',
        'no_proxy': 'localhost,127.0.0.1'
    }
}
driver = webdriver.Chrome(seleniumwire_options=options)

Answer 4

在与您遇到的相同问题进行了数小时的挖掘之后，我发现了这个网站https://botproxy.net/docs/how-to/setting-chromedriver-proxy-auth-with-selenium-using-python/。我对其进行了测试并完美地为我工作。

import os
import zipfile

from selenium import webdriver

PROXY_HOST = 'x.botproxy.net'  # rotating proxy
PROXY_PORT = 8080
PROXY_USER = 'proxy-user'
PROXY_PASS = 'proxy-password'


manifest_json = """
{
    "version": "1.0.0",
    "manifest_version": 2,
    "name": "Chrome Proxy",
    "permissions": [
        "proxy",
        "tabs",
        "unlimitedStorage",
        "storage",
        "<all_urls>",
        "webRequest",
        "webRequestBlocking"
    ],
    "background": {
        "scripts": ["background.js"]
    },
    "minimum_chrome_version":"22.0.0"
}
"""

background_js = """
var config = {
        mode: "fixed_servers",
        rules: {
          singleProxy: {
            scheme: "http",
            host: "%s",
            port: parseInt(%s)
          },
          bypassList: ["localhost"]
        }
      };

chrome.proxy.settings.set({value: config, scope: "regular"}, function() {});

function callbackFn(details) {
    return {
        authCredentials: {
            username: "%s",
            password: "%s"
        }
    };
}

chrome.webRequest.onAuthRequired.addListener(
            callbackFn,
            {urls: ["<all_urls>"]},
            ['blocking']
);
""" % (PROXY_HOST, PROXY_PORT, PROXY_USER, PROXY_PASS)


def get_chromedriver(use_proxy=False, user_agent=None):
    path = os.path.dirname(os.path.abspath(__file__))
    chrome_options = webdriver.ChromeOptions()
    if use_proxy:
        pluginfile = 'proxy_auth_plugin.zip'

        with zipfile.ZipFile(pluginfile, 'w') as zp:
            zp.writestr("manifest.json", manifest_json)
            zp.writestr("background.js", background_js)
        chrome_options.add_extension(pluginfile)
    if user_agent:
        chrome_options.add_argument('--user-agent=%s' % user_agent)
    driver = webdriver.Chrome(
        os.path.join(path, 'chromedriver'),
        chrome_options=chrome_options)
    return driver

def main():
    driver = get_chromedriver(use_proxy=True)
    #driver.get('https://www.google.com/search?q=my+ip+address')
    driver.get('https://httpbin.org/ip')

if __name__ == '__main__':
    main()

Answer 5

Selenium Chrome 代理身份验证

使用 Python 使用 Selenium 设置 chromedriver 代理

如果您需要使用带有 python 的代理和带有 chromedriver 的 Selenium 库，您通常使用以下代码（无需任何用户名和密码：

chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument('--proxy-server=%s' % hostname + ":" + port)
driver = webdriver.Chrome(chrome_options=chrome_options)

除非代理需要身份验证，否则它工作正常。如果代理要求您使用用户名和密码登录，它将不起作用。在这种情况下，您必须使用下面解释的更棘手的解决方案。顺便说一句，如果您将来自代理提供商或服务器的服务器 IP 地址列入白名单，它不应该询问代理凭据。

在 Selenium 中使用 Chromedriver 进行 HTTP 代理身份验证

要设置代理身份验证，我们将生成一个特殊文件并使用以下代码将其动态上传到 chromedriver。此代码使用 chromedriver 配置 selenium 以使用需要使用用户/密码对进行身份验证的 HTTP 代理。

import os
import zipfile

from selenium import webdriver

PROXY_HOST = '192.168.3.2'  # rotating proxy or host
PROXY_PORT = 8080 # port
PROXY_USER = 'proxy-user' # username
PROXY_PASS = 'proxy-password' # password


manifest_json = """
{
    "version": "1.0.0",
    "manifest_version": 2,
    "name": "Chrome Proxy",
    "permissions": [
        "proxy",
        "tabs",
        "unlimitedStorage",
        "storage",
        "<all_urls>",
        "webRequest",
        "webRequestBlocking"
    ],
    "background": {
        "scripts": ["background.js"]
    },
    "minimum_chrome_version":"22.0.0"
}
"""

background_js = """
var config = {
        mode: "fixed_servers",
        rules: {
        singleProxy: {
            scheme: "http",
            host: "%s",
            port: parseInt(%s)
        },
        bypassList: ["localhost"]
        }
    };

chrome.proxy.settings.set({value: config, scope: "regular"}, function() {});

function callbackFn(details) {
    return {
        authCredentials: {
            username: "%s",
            password: "%s"
        }
    };
}

chrome.webRequest.onAuthRequired.addListener(
            callbackFn,
            {urls: ["<all_urls>"]},
            ['blocking']
);
""" % (PROXY_HOST, PROXY_PORT, PROXY_USER, PROXY_PASS)


def get_chromedriver(use_proxy=False, user_agent=None):
    path = os.path.dirname(os.path.abspath(__file__))
    chrome_options = webdriver.ChromeOptions()
    if use_proxy:
        pluginfile = 'proxy_auth_plugin.zip'

        with zipfile.ZipFile(pluginfile, 'w') as zp:
            zp.writestr("manifest.json", manifest_json)
            zp.writestr("background.js", background_js)
        chrome_options.add_extension(pluginfile)
    if user_agent:
        chrome_options.add_argument('--user-agent=%s' % user_agent)
    driver = webdriver.Chrome(
        os.path.join(path, 'chromedriver'),
        chrome_options=chrome_options)
    return driver

def main():
    driver = get_chromedriver(use_proxy=True)
    #driver.get('https://www.google.com/search?q=my+ip+address')
    driver.get('https://httpbin.org/ip')

if __name__ == '__main__':
    main()

函数 get_chromedriver 返回可在应用程序中使用的已配置 selenium webdriver。这段代码已经过测试并且运行良好。

详细了解 Chrome 中的 onAuthRequired 事件。

Answer 6

在此过程中，在更新中，使用扩展的解决方案不起作用（至少在 windows 下），而 mac 和 linux 可以。 我认为 chromedriver v2.44 是最后一个带有扩展的工作版本

Answer 7

这是一个快速、有创意的解决方案，不需要修改 selenium 的选项或将文件上传到 chromedriver。它利用 pyautogui（可以使用任何模拟按键的 python 包）来输入代理身份验证详细信息。它还使用线程来处理 chrome 身份验证弹出窗口，否则会暂停脚本。

import time
from threading import Thread
import pyautogui
from selenium.webdriver.chrome.options import Options
from selenium import webdriver

hostname = "HOST_NAME"
port = "PORT"
proxy_username = "USERNAME"
proxy_password = "PASSWORD"

chrome_options = Options()
chrome_options.add_argument('--proxy-server={}'.format(hostname + ":" + port))
driver = webdriver.Chrome(options=chrome_options)


def enter_proxy_auth(proxy_username, proxy_password):
    time.sleep(1)
    pyautogui.typewrite(proxy_username)
    pyautogui.press('tab')
    pyautogui.typewrite(proxy_password)
    pyautogui.press('enter')


def open_a_page(driver, url):
    driver.get(url)


Thread(target=open_a_page, args=(driver, "http://www.example.com/")).start()
Thread(target=enter_proxy_auth, args=(proxy_username, proxy_password)).start()

注意：对于任何严肃的项目或测试套件，我建议选择更强大的解决方案。但是，如果您只是在尝试并需要快速有效的解决方案，那么这是一个选择。