Python Requests Module

Python’s “requests” module makes HTTP requests to web services or APIs. It provides a simple and convenient way to send HTTP requests and handle responses, making it easier to interact with web services than using the built-in urllib library.

Requests module allows you to send HTTP/1.1 requests exceptionally easily. There’s no need to manually add query strings to your URLs or to form-encode your PUT & POST data — but nowadays, just use the json method!

Download and Install the Requests Module

To get started with the requests library, you must install it first, as it is not included in the Python standard library:

python -m pip install requests

Syntax

requests.methodname(params)

Creating a request

# Importing the requests module
import requests

# Defining the URL for the request
url = 'https://api.example.com/data'

# Sending a GET request to the specified URL
response = requests.get(url)

# Checking if the request was successful (HTTP status code 200)
if response.status_code == 200:
  # Printing the response content as text
  print(response.text)
else:
  print(f'Request failed with status code {response.status_code}')

In addition, to GET requests, the requests module supports other HTTP methods like POST, PUT, DELETE, and more:

# Sending a POST request with JSON data

data = {'key': 'value'}
headers = {'Content-Type': 'application/json'}
response = requests.post(url, json=data, headers=headers)

Requests methods

The requests module provides many other features, such as handling timeouts, managing cookies, handling redirects, and more.

Response object

Here are the main attributes and methods of the Response object:

1. .text: The response content is decoded as a string using the character encoding specified by the response headers (or UTF-8 if None is specified).
2. .content: The raw response content as bytes.
3. .status_code: The HTTP status code returned by the server.
4. .headers: A dictionary-like object containing the response headers.
5. .cookies: A RequestsCookieJar object representing the cookies set or returned in the response.
6. .url: The URL of the response.
7. .encoding: The character encoding used to decode the response content to produce the .text attribute. It can be manually set to change the encoding.
8. .reason: The HTTP reason phrase returned by the server (e.g., “OK” for a 200 status code).
9. .elapsed: A timedelta object representing the duration of time taken to complete the request.

Authentication using Python Requests

Authentication is a crucial aspect of many web services. The requests library in Python provides built-in mechanisms to handle various types of Authentication.

Basic Authentication

Basic Authentication requires a username and password. It sends the credentials as a Base64-encoded string in the request headers.

import requests
from requests.auth import HTTPBasicAuth

response = requests.get('https://api.example.com/data', 
                          auth=HTTPBasicAuth('username', 'password'))

# Alternatively, you can use shorthand:
response = requests.get('https://api.example.com/data', auth=('username', 'password'))

Digest Authentication

Digest Authentication is a challenge-response mechanism that’s more secure than Basic Authentication.

from requests.auth import HTTPDigestAuth

response = requests.get('https://api.example.com/data', 
                         auth=HTTPDigestAuth('username', 'password'))

Bearer Token Authentication (e.g., OAuth2.0)

For APIs that require token-based Authentication, such as OAuth2.0:

import requests

headers = {
  'Authorization': 'Bearer YOUR_ACCESS_TOKEN'
}

response = requests.get('https://api.example.com/data', headers=headers)

SSL Certification Verification

In the context of the requests library in Python, SSL certificate verification is enabled by default for all HTTPS requests. This is a security feature to prevent potential man-in-the-middle attacks.

Behavior in requests:

Default Behavior

By default, requests will verify the SSL certificate of the server.

The request usually proceeds if the certificate is valid and issued by a trusted Certificate Authority (CA). If the certificate is invalid, self-signed, expired, or issued by an untrusted CA, requests will raise an SSLError.

Disabling Verification

For development or when connecting to a server with a self-signed certificate, you might want to disable certificate verification. However, this is not recommended for production use.

response = requests.get('https://example.com', verify=False)

When you do this, you will typically see a warning like:

InsecureRequestWarning: Unverified HTTPS request is being made. 
                        Adding certificate verification is strongly advised.

To suppress this warning (though not recommended), you can do the following:

import urllib3

urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)

Session Objects

In the requests library, a Session object allows you to persist specific parameters across requests. This can be particularly useful when you’re making several requests to the same server and want to reuse specific attributes like headers, cookies, proxies, or authentication details.

Using a Session can also improve performance as it can reuse the underlying TCP connection for multiple requests to the same host rather than creating a new connection for each request.

In summary, if you’re making multiple requests to the same server or require specific attributes to persist across requests, using a Session object can simplify your code and improve performance.

import requests

s = requests.Session()

s.get('https://httpbin.org/cookies/set/sessioncookie/123456789')

op = s.get('https://httpbin.org/cookies')

print(op.text)

Output

This output suggests that the cookie named sessioncookie with the value 123456789 was successfully set in the session and retrieved in the subsequent request.

That’s it!

Krunal

Krunal Lathiya is a seasoned Computer Science expert with over eight years in the tech industry. He boasts deep knowledge in Data Science and Machine Learning. Versed in Python, JavaScript, PHP, R, and Golang. Skilled in frameworks like Angular and React and platforms such as Node.js. His expertise spans both front-end and back-end development. His proficiency in the Python language stands as a testament to his versatility and commitment to the craft.