Creating a boto3 client is super expensive and benefits hugely from being re-used. I see lots of code, like the examples in the post, that re-create them over and over again.
4ms is an incredibly long time to construct an object. Under the hood it’s dynamically creating the client from JSON IDL files IIRC.
At even 10 RPS it’s going to be taking a non-trivial amount of time, and I believe the connection pooling is done at the client level so you’re going to be paying for a new connection to s3 each time.
7 comments
[ 2.5 ms ] story [ 14.7 ms ] threadEdit: especially compared to any interactions with s3
At even 10 RPS it’s going to be taking a non-trivial amount of time, and I believe the connection pooling is done at the client level so you’re going to be paying for a new connection to s3 each time.