A POST request is a particular type of HTTP method used when we send data to services on the web. We use them on web sites that use forms - when we login, when we send messages or post an image. Applications also use POST requests to interact with other services to send data with JSON being a common data format for exchange.
The Requests library is one of the most popular HTTP client libraries for Python. It currently has over 45k stars on Github, with downloads on PyPI of 115M a month! It makes sending POST requests much simpler programmatically than having to send data via a headless browser. With a headless browser we'd need to write scripts to navigate to a site, move between fields, populate each of them with the desired data before finally submitting the data. By sending a POST request, we skip straight to the final submission step. Requests also is a much, much smaller library than a browser resulting in better performance and memory usage.
In this article we'll cover how to construct a POST request using Requests and how it can make the process much simpler for us.
Building a JSON POST Request with Requests
As an example, lets start by building a JSON POST request the hard way. Don't worry Requests will simplify this for us later! We're using the httpbin.org service, which returns a JSON response detailing the content that was sent.
1. Set the Request Method to POST
Requests has a really simple HTTP verb based design, meaning that to get a response we call it's .post() method supplying our URI as an argument. It also provides similar methods for GET, PUT and PATCH requests.
import requests
r = requests.post("https://httpbin.org/post")
2. Set the POST data
To actually send some data, we supply a data argument. By default data is sent using as a HTML form as we would get by submitting any form on the web. Requests sets the content type to 'application/x-www-form-urlencoded', so it isn't necessary to set any headers.
import requests
import json
r = requests.post("https://httpbin.org/post",
data={"key": "value"},
)
Inspecting the response from the httpbin.org service, we can see what was sent as form data under the "form" key.
>>> r.text
'{\n "args": {}, \n "data": "", \n "files": {}, \n "form": {\n "key": "value"\n }, \n "headers": {\n "Accept": "*/*", \n "Accept-Encoding": "gzip, deflate", \n "Content-Length": "9", \n "Content-Type": "application/x-www-form-urlencoded", \n "Host": "httpbin.org", \n "User-Agent": "python-requests/2.25.1", \n "X-Amzn-Trace-Id": "Root=1-60df1a04-0384d3ce7d9ac00b5855064b"\n }, \n "json": null, \n "origin": "**.***.**.***", \n "url": "https://httpbin.org/post"\n}\n'
If we want to send JSON, we can supply a JSON formatted string. In our example, we're using the json module to convert a dictionary to JSON formatted string ready to send.
import requests
import json
r = requests.post("https://httpbin.org/post",
data=json.dumps({"key": "value"}),
)
3. Set the POST Headers
Our JSON example isn't going to work as it is now. With many services, we'll likely get a 400 (Bad Request) HTTP status code. To prevent this we also need to inform the service we're calling that our data is JSON so it can be handled correctly. To do so, we set the 'Content-Type' to 'application/json' in the request headers:
import requests
import json
r = requests.post(
"https://httpbin.org/post",
data=json.dumps({"key": "value"}),
headers={"Content-Type": "application/json"},
)
4. POST JSON Data
If you think our JSON examples so far look a bit complex - you're totally right. Requests makes it very easy to reduce all this down to a much simpler call. We can just supply a 'json' argument with our data. Requests will correctly set our headers and encode the JSON formatted string for us automatically.
import requests
r = requests.post('https://httpbin.org/post', json={'key':'value'})
Reading JSON Responses
By inspecting the response from our service, we can see the data returned is a JSON formatted string too. This is only text, so we'd need to parse it ourselves to use it within our Python script.
>>> r.text
'{\n "args": {}, \n "data": "{\\"key\\": \\"value\\"}", \n "files": {}, \n "form": {}, \n "headers": {\n "Accept": "*/*", \n "Accept-Encoding": "gzip, deflate", \n "Content-Length": "16", \n "Content-Type": "application/json", \n "Host": "httpbin.org", \n "User-Agent": "python-requests/2.25.1", \n "X-Amzn-Trace-Id": "Root=1-60df0aaa-3105fc35436042571335fa22"\n }, \n "json": {\n "key": "value"\n }, \n "origin": "**.***.**.***", \n "url": "https://httpbin.org/post"\n}\n'
Alternatively, if we want to access the response as a JSON object we can also use Requests built in JSON decoder by calling .json() on the response object.
>>> r.json()
{'args': {}, 'data': '{"key": "value"}', 'files': {}, 'form': {}, 'headers': {'Accept': '*/*', 'Accept-Encoding': 'gzip, deflate', 'Content-Length': '16', 'Content-Type': 'application/json', 'Host': 'httpbin.org', 'User-Agent': 'python-requests/2.25.1', 'X-Amzn-Trace-Id': 'Root=1-60deee08-4e9c76b6457826d5001b76fa'}, 'json': {'key': 'value'}, 'origin': '**.***.**.***', 'url': 'http://httpbin.org/post'}
Making POST requests within a Session
As a final example, lets login to Hacker News using Requests and submit some form data to login in combination with a requests.Session() object. Any cookies we receive will be stored in the object, and will be reused in later .get() calls to the session.
For our example, we'll search the response text of the news page to confirm if we're shown the logout link indicating we're being served a logged in page.
import requests
session = requests.Session()
r = session.post("https://news.ycombinator.com/login", data={
"acct": "username",
"pw": "***not-telling***"
})
r = session.get("https://news.ycombinator.com/news")
logout_pos = r.text.find("logout")
print(r.text[logout_pos-20:logout_pos+20])
# <a id='logout' href="logout
Great! Our link is in the response indicating we're logged in. We can now continue to interact with the site using the session object via Requests.
💡 Love web scraping in Python? Check out our expert list of the Best Python web scraping libraries.
Conclusion
You've seen just how simple Requests makes it to post data to sites or services and how we can reduce much of the common code in our applications by using it. This simple design has certainly contributed to it's success in the Python community, making it a firm favorite with developers. Before you reach for a full blown headless browser to send your data, you should definitely consider using Requests.
You will often need proxies for your web scraping projects, don't hesitate to checkout this article on how to use proxies with Python Requests .
