Setting Up Selenium 4 with Proxy Authentication

Setting Up Selenium with Proxy Authentication

In this article, we will explore how to use Selenium WebDriver to open multiple URLs while routing the requests through a proxy server that requires authentication. We’ll walk through the prerequisites, the setup process, and the detailed explanation of the provided code.

Full Code

from selenium import webdriver
from selenium.webdriver.common.proxy import Proxy, ProxyType
import time

# List of URLs to open
urls = ['https://example.com', 'https://google.com', 'https://stackoverflow.com']

# Proxy settings
proxy_host = "vodafone.trial.prestigeproxies.com"
proxy_port = 55401
proxy_username = "yawer"
proxy_password = "mIHVrT85ux"

# Create a Proxy object
proxy = Proxy()
proxy.proxy_type = ProxyType.MANUAL
proxy.http_proxy = f"{proxy_host}:{proxy_port}"
proxy.ssl_proxy = f"{proxy_host}:{proxy_port}"
proxy.socks_proxy = f"{proxy_host}:{proxy_port}"
proxy.ftp_proxy = f"{proxy_host}:{proxy_port}"
proxy.no_proxy = ""

# Add authentication
capabilities = webdriver.DesiredCapabilities.CHROME
proxy.add_to_capabilities(capabilities)
capabilities['proxy']['username'] = proxy_username
capabilities['proxy']['password'] = proxy_password

# Create a new instance of the Chrome driver with the proxy
driver = webdriver.Chrome(desired_capabilities=capabilities)

# Open each URL
for url in urls:
    driver.get(url)

# Wait for 30 seconds
time.sleep(30)

# Close the browser
driver.quit()

Prerequisites

Either you can follow this tutorial. or do these simple steps.

1. Python: Ensure that you have Python installed on your machine. You can download it from python.org.
2. Selenium WebDriver: Install the Selenium package using pip:

   pip install selenium

3. WebDriver for Chrome: Download the ChromeDriver from chromedriver.chromium.org and ensure it is accessible in your system’s PATH.
4. Google Chrome Browser: Ensure that you have the Google Chrome browser installed.

Overview of the Code

The provided Python script uses Selenium to automate the following tasks:

1. Setting up a proxy server with authentication.
2. Configuring Selenium to use this proxy server.
3. Opening a series of URLs.
4. Waiting for a specified time.
5. Closing the browser.

Let’s break down the code step-by-step.

Step-by-Step Explanation

Importing Required Libraries

from selenium import webdriver
from selenium.webdriver.common.proxy import Proxy, ProxyType
import time

• selenium.webdriver: The main Selenium package for controlling web browsers.
• selenium.webdriver.common.proxy: Contains classes for configuring proxy settings.
• time: Standard Python library for adding delays.

Defining URLs to Open

urls = ['https://example.com', 'https://google.com', 'https://stackoverflow.com']

• A list of URLs that you want to open sequentially.

Setting Up Proxy Details

proxy_host = "vodafone.trial.prestigeproxies.com"
proxy_port = 55401
proxy_username = "yawer"
proxy_password = "mIHVrT85ux"

• Proxy host, port, username, and password for authentication.

Creating and Configuring the Proxy Object

proxy = Proxy()
proxy.proxy_type = ProxyType.MANUAL
proxy.http_proxy = f"{proxy_host}:{proxy_port}"
proxy.ssl_proxy = f"{proxy_host}:{proxy_port}"
proxy.socks_proxy = f"{proxy_host}:{proxy_port}"
proxy.ftp_proxy = f"{proxy_host}:{proxy_port}"
proxy.no_proxy = ""

• Proxy(): Initializes a new proxy object.
• proxy.proxy_type: Sets the proxy type to manual.
• Configures the proxy settings for various protocols (HTTP, SSL, SOCKS, FTP).

Adding Authentication to Capabilities

capabilities = webdriver.DesiredCapabilities.CHROME
proxy.add_to_capabilities(capabilities)
capabilities['proxy']['username'] = proxy_username
capabilities['proxy']['password'] = proxy_password

• webdriver.DesiredCapabilities.CHROME: Retrieves the desired capabilities for Chrome.
• proxy.add_to_capabilities(capabilities): Adds the proxy configuration to the capabilities object.
• Adds the username and password for proxy authentication to the capabilities.

Initializing the WebDriver with Proxy Settings

driver = webdriver.Chrome(desired_capabilities=capabilities)

• webdriver.Chrome: Creates a new instance of the Chrome WebDriver with the specified capabilities, including proxy settings.

Opening URLs Sequentially

for url in urls:
    driver.get(url)

• Iterates through the list of URLs and opens each one in the browser.

Adding a Delay

time.sleep(30)

• Waits for 30 seconds to allow time for interactions or observation.

Closing the Browser

driver.quit()

• Closes the browser and ends the WebDriver session.

Conclusion

This script is useful for scenarios where you need to route your web traffic through a proxy server with authentication, such as for web scraping or automated testing under different network conditions. By following the steps outlined, you can set up and execute the script on your machine.

Make sure to handle any exceptions and add necessary logging for a more robust implementation. Additionally, ensure that the proxy service you are using is reliable and legal for your use case.