How to Use a Proxies in Python Requests?

In this article, we will explore how to use the Python Requests library behind a proxy server. Developers use proxies to achieve anonymity, security, and sometimes even employ multiple proxies to prevent websites from blocking their IP addresses. Proxies also offer other benefits such as bypassing filters and censorship.

Prerequisites and Installation

Step 1: Install the Requests Library

First, ensure that the Requests library is installed in your Python environment. If it is not already installed, you can install it using pip:

pip install requests

For further reading: Crawling Web Articles via HTTP Proxy in Python

Step 2: Set Up the Proxy Server

Using a proxy in Requests is straightforward. You just need to create a dictionary containing the protocol and address of the proxy server you want to use. For example:

proxies = {
    "http": "http://10.10.1.10:3128",
    "https": "https://10.10.1.10:1080",
}

Here, the `http` key corresponds to the proxy server address for HTTP requests, and the `https` key corresponds to the proxy server address for HTTPS requests. Note that you need to replace the proxy server's IP address and port number with your actual settings.

Step 3: Send a Request

Once the proxy is set up, you can pass it as a parameter when sending a request with Requests. Here is an example of sending a GET request using a proxy:

import requests

url = 'http://example.com'
proxies = {
    "http": "http://10.10.1.10:3128",
    "https": "https://10.10.1.10:1080",
}

response = requests.get(url, proxies=proxies)
print(response.text)

Step 4: Handle Exceptions

When using proxies, you may encounter network connection issues or problems with the proxy server. To make your application more robust, you should catch and handle possible exceptions:

import requests
from requests.exceptions import RequestException

url = 'http://example.com'
proxies = {
    "http": "http://10.10.1.10:3128",
    "https": "https://10.10.1.10:1080",
}

try:
    response = requests.get(url, proxies=proxies)
    print(response.text)
except RequestException as e:
    print(f"Request failed: {e}")

Step 5: Use an Authenticated Proxy Server

If your proxy server requires authentication, you can include the username and password in the proxy address. For example:

proxies = {
    "http": "http://user:[email protected]:3128",
    "https": "https://user:[email protected]:1080",
}

Make sure to replace `user` and `password` with your proxy authentication details.

By following these steps, you can use the Requests library in Python to send requests through a proxy. This is very useful for web scraping, testing, and many other scenarios.

Building on this tutorial, we will further explore how to manage multiple proxy servers using Python's Requests library and implement a simple proxy pool. This is particularly useful for situations where frequent IP address changes are needed to avoid being blocked by the target website.

Create a Proxy Pool

The basic idea of a proxy pool is to maintain a list of proxies and randomly select one to distribute the source of the requests. Here’s how to implement a basic proxy pool:

Store the Proxy List: You can store the proxies in a list or use a more advanced storage system like a database or caching system.

Select a Proxy: Randomly choose a proxy from the list to make a request.

Validate Proxy Effectiveness: Regularly check the validity of the proxies and remove those that are invalid or slow to respond.

import requests
import random

# Proxy list
proxies_list = [
    {"http": "http://10.10.1.10:3128", "https": "https://10.10.1.10:1080"},
    {"http": "http://10.10.2.10:3128", "https": "https://10.10.2.10:1080"},
    {"http": "http://10.10.3.10:3128", "https": "https://10.10.3.10:1080"}
]

def get_random_proxy():
    return random.choice(proxies_list)

# Send request using a random proxy
url = 'http://example.com'
proxy = get_random_proxy()
response = requests.get(url, proxies=proxy)
print(response.text)

• Python
• Proxies

How to Use a Proxies in Python Requests? Review FAQ

Best Telegram Proxie...

As a secure and fast instant m...

Best Pinterest Proxi...

Pinterest has become an essent...

Best Gmail Proxies

Gmail has become a crucial com...

Understanding Static...

Static residential proxies and...