The requests library is the main standard for making HTTP requests in Python. Requesting with Python requests library is very simple. It abstracts the complexities of making HTTP requests behind the beautiful, simple API so that you can concentrate on interacting with services and consuming the data in your app.
HTTP GET Request
HTTP(HyperText Transfer Protocol) methods such as GET and POST decide which action you are trying to perform when making the HTTP request. Besides GET and POST, there are several other standard methods you’ll use later in this ExampleExample.
One of the most standard HTTP methods is GET. The GET method indicates that you are trying to get or retrieve the data from a specified resource.
Install requests Python module
You can install the requests library by running the following command.
python3 -m pip install requests # OR pip install requests
If you wish to use Pipenv for managing Python packages, you can run the following.
pipenv install requests
Once the requests module is installed, you can use it in your application. For example, importing requests looks like the below.
import requests
Python requests get
To create a GET request in Python, use the requests.get() method. The get() method takes three parameters and returns a response with a status code. The requests get() method sends a GET request to the specified URL.
Syntax
requests.get(url, params={key: value}, args)
The args means zero or more of the named arguments in the parameter table below. Example:
requests.get(url, timeout=2.50)
Parameters
| Parameter | Description |
|---|---|
| url | Required. The url of the request. |
| params | Optional. It is a dictionary, a list of tuples or bytes to send as the query string. Default None |
| allow_redirects | Optional. A Boolean to enable/disable redirection. Default True (allowing redirects) |
| auth | Optional. A tuple to enable a secure HTTP authentication. Default None |
| cert | Optional. It is a String or Tuple specifying the cert file or key. Default None |
| cookies | Optional. It is a dictionary of cookies to send to the specified url. Default None |
| headers | Optional. It is a dictionary of HTTP headers to send to the specified url. Default None |
| proxies | Optional. It is a dictionary of the protocol to the proxy url. Default None |
| stream | Optional. It is a Boolean indication if the response should be immediately downloaded (False) or streamed (True). Default False |
| timeout | Optional. It is a number, or a tuple, indicating how many seconds to wait for a client to make the connection and send a response. Default None means that the request will continue until the connection is closed. |
| verify | Optional. A Boolean or a String indication to test the servers’ TLS certificate or not. Default True |
We will call the Github API.
import requests
data = requests.get('https://api.github.com/users/KrunalLathiya')
print(data.json()['login'])
Output
KrunalLathiya
We get the response in the form of data, and then we have used the json() to parse the data.
So, we get the data in the form of a dictionary, Then, we need to call the key of the value, and we will get the value.
Response object
The requests.get() method returns the response that is a powerful object for inspecting the request results. Again, let’s see the above ExampleExample, but this time, we won’t parse the json.
import requests
data = requests.get('https://api.github.com/users/KrunalLathiya')
print(data)
Output
<Response [200]>
In this ExampleExample, you have captured the return value of get(), the instance of response, and stored it in the variable called a response. You can now use a response to see a lot of information about the results of your GET request.
We got the status code in the response, which is 200. So that means the page is there.
Let’s send a GET request in which we don’t get 200 in the status code.
import requests
data = requests.get('https://api.github.com/KrunalLathiya')
print(data)
Output
<Response [404]>
That means the page does not exist, and we got the page not found error whose status code is 404.
Status Codes
The first information we can gather from the response is the status code. A status code notifies you of the status of the request.
For instance, a 200 OK status means that your request was successful, whereas a 404 status means that the resource you were looking for was not found.
In the above examples, we got the 200 and 404 status codes.
By accessing the .status_code, you can see the status code that the server returned.
import requests
data = requests.get('https://api.github.com/KrunalLathiya')
print(data.status_code)
Output
404
Sometimes, you have to use the status code information to make decisions in your code. For instance, put the if-else statement based on the returned code.
if response.status_code == 200:
print('You got the success!')
elif response.status_code == 404:
print('Page not Found.')
You can simplify the above ExampleExample by rewriting the if statement.
if response:
print('You got the success!')
else:
print('Something went wrong.')
Raise an exception based on status code
You don’t need to check the response’s status code in an if statement. Instead, you have to raise an exception if the request was unsuccessful. You can do the using .raise_for_status().
See the following code.
from requests import get, HTTPError
for url in ['https://api.github.com/users/KrunalLathiya', 'https://api.github.com/data']:
try:
response = get(url)
# If the response was successful, no Exception will be raised
response.raise_for_status()
except HTTPError as http_err:
print(f'HTTP error occurred: {http_err}')
except Exception as err:
print(f'Other error occurred: {err}')
else:
print('Successfull GET Request')
In this ExampleExample, we have just imported the required methods to import from the requests module.
We have imported get and HTTPError.
Then we have used them for…in loop and use the Exception handling mechanism.
If the response of a request is successful, then no exception will be raised; otherwise, the HTTPError exception will be raised.
In the above code example, we have passed two list items. One is valid, and another is invalid.
See the output.
Successfull GET Request HTTP error occurred: 404 Client Error: Not Found for url: https://api.github.com/data
You can see that the first GET request is successful, and the second GET request is failed and returned the 404 Client Error.
If you invoke .raise_for_status(), the HTTPError will be raised for certain status codes. If the status code indicates the successful request, the program will proceed without exception being raised.
Content
The GET Request often has some valuable information, known as a payload, in the message body. Using the attributes and methods of Response, you can see the payload in various formats.
from requests import get
res = get('https://api.github.com/users/KrunalLathiya')
print(res.content)
Output
b'{"login":"KrunalLathiya","id":14403830,"node_id":"MDQ6VXNlcjE0NDAzODMw",
"avatar_url":"https://avatars1.githubusercontent.com/u/14403830?v=4",
"gravatar_id":"","url":"https://api.github.com/users/KrunalLathiya",
"html_url":"https://github.com/KrunalLathiya",
"followers_url":"https://api.github.com/users/KrunalLathiya/followers",
"following_url":"https://api.github.com/users/KrunalLathiya/following{/other_user}",
"gists_url":"https://api.github.com/users/KrunalLathiya/gists{/gist_id}",
"starred_url":"https://api.github.com/users/KrunalLathiya/starred{/owner}{/repo}",
"subscriptions_url":"https://api.github.com/users/KrunalLathiya/subscriptions",
"organizations_url":"https://api.github.com/users/KrunalLathiya/orgs",
"repos_url":"https://api.github.com/users/KrunalLathiya/repos",
"events_url":"https://api.github.com/users/KrunalLathiya/events{/privacy}",
"received_events_url":"https://api.github.com/users/KrunalLathiya/received_events",
"type":"User",
"site_admin":false,"name":"Krunal","company":"Freelancer",
"blog":"https://appdividend.com","location":"Rajkot",
"email":null,"hireable":true,"bio":"Web Developer and Author",
"twitter_username":null,"public_repos":121,"public_gists":0,
"followers":546,"following":13,"created_at":"2015-09-20T19:22:19Z",
"updated_at":"2020-06-07T13:10:25Z"}'
The content property of the response object gives the raw bytes of the payload. You will often have to convert them into the string using a character encoding such as UTF-8.
The response will do that for you when you access .text.
from requests import get
res = get('https://api.github.com/users/KrunalLathiya')
print(res.text)
Convert Serialized JSON to Dictionary
If you check the response content closely, you can see that it is a serialized json. You can convert the json to the dictionary using .json() method.
from requests import get
res = get('https://api.github.com/users/KrunalLathiya')
print(res.json())
Output
{'login': 'KrunalLathiya', 'id': 14403830, 'node_id': 'MDQ6VXNlcjE0NDAzODMw',
'avatar_url': 'https://avatars1.githubusercontent.com/u/14403830?v=4',
'gravatar_id': '', 'url': 'https://api.github.com/users/KrunalLathiya',
'html_url': 'https://github.com/KrunalLathiya',
'followers_url': 'https://api.github.com/users/KrunalLathiya/followers',
'following_url': 'https://api.github.com/users/KrunalLathiya/following{/other_user}',
'gists_url': 'https://api.github.com/users/KrunalLathiya/gists{/gist_id}',
'starred_url': 'https://api.github.com/users/KrunalLathiya/starred{/owner}{/repo}',
'subscriptions_url': 'https://api.github.com/users/KrunalLathiya/subscriptions',
'organizations_url': 'https://api.github.com/users/KrunalLathiya/orgs',
'repos_url': 'https://api.github.com/users/KrunalLathiya/repos',
'events_url': 'https://api.github.com/users/KrunalLathiya/events{/privacy}',
'received_events_url': 'https://api.github.com/users/KrunalLathiya/received_events',
'type': 'User', 'site_admin': False, 'name': 'Krunal',
'company': 'Freelancer', 'blog': 'https://appdividend.com',
'location': 'Rajkot', 'email': None,
'hireable': True, 'bio': 'Web Developer and Author',
'twitter_username': None, 'public_repos': 121,
'public_gists': 0, 'followers': 546, 'following': 13,
'created_at': '2015-09-20T19:22:19Z',
'updated_at': '2020-06-07T13:10:25Z'}
You can see that I have got the data in the form of a Dictionary.
Now, I can access the one-by-one value of the Dictionary using its key.
from requests import get
res = get('https://api.github.com/users/KrunalLathiya')
print(res.json()['login'])
Output
KrunalLathiya
The type of return value of the .json() method is the dictionary so that you can access the values in the object by key.
You can do a lot with status codes and message bodies. But, if you need more information, like metadata about the response itself, you will need to look at the response’s headers.
Headers
The response headers can give us helpful information, such as the content type of the response payload and the time limit on how long to cache the response. To view these headers, access .headers.
from requests import get
res = get('https://api.github.com/users/KrunalLathiya')
print(res.headers)
Output
{'date': 'Thu, 18 Jun 2020 08:06:29 GMT', 'content-type': 'application/json; charset=utf-8', 'server': 'GitHub.com', 'status': '200 OK', 'cache-control': 'public, max-age=60, s-maxage=60', 'vary': 'Accept, Accept-Encoding, Accept, X-Requested-With, Accept-Encoding', 'etag': 'W/"cd0a64256ac419c62d91c62feebf0f95"', 'last-modified': 'Sun, 07 Jun 2020 13:10:25 GMT', 'x-github-media-type': 'github.v3; format=json', 'access-control-expose-headers': 'ETag, Link, Location, Retry-After, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval, X-GitHub-Media-Type, Deprecation, Sunset', 'access-control-allow-origin': '*', 'strict-transport-security': 'max-age=31536000; includeSubdomains; preload', 'x-frame-options': 'deny', 'x-content-type-options': 'nosniff', 'x-xss-protection': '1; mode=block', 'referrer-policy': 'origin-when-cross-origin, strict-origin-when-cross-origin', 'content-security-policy': "default-src 'none'", 'content-encoding': 'gzip', 'X-Ratelimit-Limit': '60', 'X-Ratelimit-Remaining': '54', 'X-Ratelimit-Reset': '1592469405', 'Accept-Ranges': 'bytes', 'Content-Length': '513', 'X-GitHub-Request-Id': '9E0F:3387:F368EE:140B639:5EEB2084'}
The .headers return a dictionary-like object, allowing you to access header values by key.
from requests import get
res = get('https://api.github.com/users/KrunalLathiya')
print(res.headers['server'])
Output
GitHub.com
There is something special about this dictionary-like header object, though. The HTTP spec defines headers to be case-insensitive, which means we can access these headers without worrying about their capitalization.
Request Headers
To customize headers, you pass the dictionary of HTTP headers to get() using the headers parameter.
For instance, you can change your previous search request to highlight the matching search terms in the results by specifying a text-match media type in the Accept header.
from requests import get
response = get(
'https://api.github.com/search/repositories',
params={'q': 'requests+language:php'},
headers={'Accept': 'application/vnd.github.v3.text-match+json'},
)
# View the new `text-matches` array which provides information
# about your search term within the results
json_response = response.json()
repository = json_response['items'][0]
print(f'Text matches: {repository["text_matches"]}')
Output
Text matches: [{'object_url': 'https://api.github.com/repositories/117567334', 'object_type': 'Repository', 'property': 'description', 'fragment': 'Easily build Eloquent queries from API requests', 'matches': [{'text': 'requests', 'indices': [39, 47]}]}]
The Accept header tells the server what content types your application can handle.
Since you are expecting the matching search terms to be highlighted, you are using the header value application/vnd.Github.v3.text-match+json is the proprietary GitHub Accept header where the content is in a particular JSON format.
Conclusion
In this Python requests get ExampleExample, we have seen how to send GET requests to a server, handle the response, convert data from json to dictionary, and request headers.
Finally, Python Requests get() Example is over.
