cat /dev/brain

Use a Session

If you're largely talking to the same service (or set of services), connection pooling is your friend. If you're doing something like:

import requests

requests.get(url, ...)
requests.post(url, ...)

A lot, then what you're effectively doing each time is:

# requests.get(url, ...)
s = requests.Session()
s.get(url, ...)
s.close()
# requests.post(url, ...)
s = requests.Session()
s.post(url, ...)
s.close()

This can be fine for prototyping or debugging, but in production applications where performance is important, it's very wasteful, especially because for every requests.Session you create two urllib3.PoolManager instances, etc.

If you instead create a session that you share across these places, it would look like:

s = requests.Session()
s.get(url, ...)
s.post(url, ...)

And if the connection can be kept open after the get, then you can reuse that connection on your post. That avoids:

1. A DNS lookup
2. Socket creation, including finding a DNS address that accepts the connection within your connection timeout setting
3. A TLS handshake
4. And a few other less costly things.

Be Smart About Data

This goes for both sending and receiving data.

s.post(url, files={"myformfieldname": open("large_file", "rb")})

import requests
from requests_toolbelt.multipart.encoder import MultipartEncoder

s = requests.Session()
m = MultipartEncoder(
    fields={'field0': 'value', 'field1': 'value',
            'field2': ('filename', open('file.py', 'rb'), 'text/plain')}
    )

r = s.post(url, data=m, headers={'Content-Type': m.content_type})

Use Timeouts

A lot of times people can get away without specifying timeouts for python-requests and often times when they start specifying them they get tripped up by one big thing. So let's start there:

s.get(url_that_trickles_data_slowly, timeout=5)

for some_bytes in data:
   write(response, some_bytes)
   sleep(3)

duration_of_connection_in_ms
+ number_of_chunks_sent_by_server * 3000

2135 + 150 * 3000 = 452135
452135 / ( 60 * 1000 ) = 7.535583333

connect_timeout_s: float = ...
read_timeout_s: float = ...
s.get(url, timeout=(connect_timeout_s, read_timeout_s))

Be Careful With Threads and gevent

There are ways around making sure python-requests is thread-safe. If you use a requests.Session per thread, then this isn't a concern for you. You'll still have connection pooling but per thread, so you'll want to ensure you track which remote services have been requested via which threads.

Most people don't use threads, though, but many do use gevent. You'll want to ensure that before you import anything else, you import gevent and monkey patch the world. For example, urllib3 uses a last-in-first-out queue to pool connections but because we rely on being able to set the class attribute at import time, if gevent monkey-patches the standard library's queue.LifoQueue after urllib3 is imported, then this doesn't get patched properly.

TLS Tricks and Trip-ups

As I mentioned above in Use a Session, urllib3 and python-requests will pool connections for you that can be kept alive. The beauty of it is that it improves how quickly you can send the next request to the remote service if a connection is already open to it. One of the reasons that may be faster is not having to renegotiate a TLS handshake.

That, of course, may also be a downside. Let's say that you have two code paths that do similar things with the same service but that service is a little... unconventional. Those paths may look like:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

import requests

def f1(s: requests.Session, data: bytes) -> requests.Response:
   url: str = 'https://example.local/f1'
   return s.post(
      url,
      data=data,
      headers={'Content-Type': 'application/vnd.mycustom+json'},
      verify=True,
   )

def f2(s: requests.Session, data: bytes) -> requests.Response:
   url: str = 'https://example.local/f2'
   return s.post(
      url,
      data=data,
      headers={'Content-Type': 'application/vnd.othercustom+json'},
      verify=False,
   )

def something_else(s: requests.Session):
   r1 = f2(s, b'...')
   r2 = f1(s, b'...')

The problem, however, is that depending on the order you call these you get behaviour you do not want. In a call to something_else above, we create a new connection to our endpoint with TLS verification disabled. Then, we make a second call where we explicitly want it enabled. If we're able to reuse the first connection, though, then we will not re-establish a TLS connection in order to verify the chain of trust in the second call.

This affects a number of old versions of requests but the fix is in the next release.

Customize User-Agent Header

One thing that people often seem to run into is being bucketed into a group with abusive users of python-requests. Many servers now look for the default python-requests User-Agent string and change the behaviour of their service. Regardless of whether this applies to you, or you're only talking to other services you own or are internal to your company, you should create a User-Agent header that distinguishes you from someone else.

These are by no means meant as authorization or authentication, but they will help with potentially finding hints as to what's happen.

Personally, I recommend the name of your library/service and the version of it that makes sense to you. If you distribute a library to others to use to interact with your service and you're on the hook for providing support for that library, I also include suggest including the version of critical dependencies. Luckily, there's a utility that makes this easier.

Use Redirects Well

By default, python-requests has redirect handling and it has had this support for quite a while. python-requests also limits how many redirects it will follow by default to 30 redirects. If we allowed unlimited redirects, someone could easily create a server that has infinite redirects which tie up a client indefinitely creating a denial of service.

My advice here, is that you probably want to set a lower limit than that. Personally, I always lower that to 3 as a starting point (or if I know I should never see a redirect, I disable them). I also deliberately combine this with constrained timeouts. I constrain my timeouts because, as I mentioned above, a server could trickle data to you without tripping a read timeout. So if it trickles data to you but is also redirecting you after it finishes writing all that data, then a single request can take as long as the amount of time it takes to connect and read all of that data multiplied by the number of redirects.

Learn Just Enough About Networking

Finally, the team behind python-requests is very small and we have full-time jobs and families. Many times we get people who file issues "urgently" seeking help with an exception they're encountering that they could otherwise learn about themselves. Unfortunately for them, there is no guarantee of support at all from python-requests and definitely no urgent support available.

If instead you learn a little bit about what goes into HTTP, HTTPS, etc. (for example, you could start by reading a zine) then you'll better be able to help yourself. The more you can get a basic understanding of what's happening, the better able you'll be to find information online to help yourself.

If you want an outline, here are some good starting points:

• Learn what TLS is (and why it is sometimes still called SSL):
  • Keep in mind that when establishing a TLS connection, things can get in the way
  • Learn a little about certificates and how to get information about them, e.g., using openssl s_client.
• Learn a little about TCP and DNS:
  • Know that a connection can go through intermediaries and so sometimes you get a vague message about a "Peer".
  • Know that DNS is the backbone of much of the infrastructure you need to care about. (Bonus points: Learn how to use a tool to inspect DNS, like dig)
  • Know that sometimes performance is caused by having to break up large messages across multiple TCP packets
• Learn a little about network layers
  • Know that there are seven different layers of the network
  • Know how "appliances" operating at different layers of the network may affect your usage of any HTTP client
    • For example, know that an "appliance" (e.g., load balancer) operating at layer 4 cannot modify the HTTP request or response bodies, but can affect the TCP packets. (This is useful to know if you care about preserving aspects of the clients information. See also [4] [5])
    • Also know that at Layer 7, the "appliance" is what you establish TLS with and it may not establish TLS with the service behind it that you are hoping to securely communicate with. Also Layer 7 allows the "appliance" to modify aspects of the request and response.

Remember python-requests is not a Browser

While a lot of the time, python-requests looks to browsers and curl to determine what the "right" thing might be to do, it is not in fact a browser. It will not do the following:

• HTTP Strict Transport Security, a.k.a., HSTS
• Render/Execute JavaScript
• Save login sessions across executions of the Python interpreter
• Follow redirects in the <head> section of an HTML page
• Or do any other number of things that you might expect of a browser like Firefox or Chrome.

Understand the Limitations of Python

Additionally, there are some things that python-requests can not do because of Python. For example, exposing the TLS chain (verified or not) to the end user is not something that is possible because that's not something that is available anywhere in Python's ssl module. Likewise, we cannot do Online Certificate Status Protocol (a.k.a., OCSP) verification to check for revoked certificates because there's no way to do that with ssl today during the current connection handshake. If you want to know that you received a 100 Continue informational response, and wait for that, python-requests effectively relies on http.client which does not support that.