when you say parallel you mean the work is distributed across multiple cores? e.g. multiple python worker processes. Or is all work running single threaded in a single process (concurrent IO, but not true parallelism)?
Why not distribute the concurrent work over multiple processes to get the most out of multiple cores? More event loops, better performance, and maybe more text preprocessing.
That would be a good idea, but it's non-trivial in Python as multi-processing async support is patchy. If you have to choose multi-processing vs async for HTTP requests, from experience, async is much faster (as your not CPU bound but IO bound) and easier to use.
Look into Go, specifically the colly pkg. Learnt Go just so I could use this library after wrestling with highly parralelized Python web scraping. The 1k/requests/sec/core figure is true and frankly deadly, and Go borrows a lot of useful stuff from Python like array slicing.
It was pretty remarkable to go from 32 cores being maxed for 100 concurrent workers to... A 10-15% usage spike on a 4 core XPS :)
Really it's better to use a fast language like Go or Java for huge loads like this.
Our typical CRUD Spring backends can handle ~15,000 requests a second with no optimization.
I appreciate all the effort that's gone into making Python faster but I think it's still the wrong tool for the job at this point if heavy load is involved
Is vanilla Python networking is still dog slow, on the order of 1k requests per second per core on mid-range consumer hardware no matter how you wire it all together?
Oh, handy. This would have solved the exact problem I had last month (which I solved with just-good-enough fiddling with async python, which I'm not comfortable in yet)
Granted that example is more lines of code than the README for this library. But the core of it is fairly straightforward:
rsps = []
with concurrent.futures.ThreadPoolExecutor(max_workers=N) as executor:
futures = [executor.submit(requests.get, url) for url in urls]
for future in concurrent.futures.as_completed(futures):
rsps.append(future.result)
Great point , adding more libraries should always be a last resort.
When using lambdas adding Python libraries are an absolute pain in the but, I'll use this example in the future
This wont work for a large number of requests as the throughput plateaus quit rapidly with a large number of threads due to the GIL and blocking I/O. This is the main reason async was introduced into Python.
Also `many-requests` has further logic to deal with retries and bad responses.
Its not simple to just "swap out" the requests library with an async one. This is why I wrote this package :) It deals with a lot of the headaches one would run into by trying just that.
Do you have empirical evidence for this? Even theoretically, threads can perform better because pythom does infact release the Gil for i/o bound operations and also, not blocking an event loop when a cpu bound task is running.
In my experience at least, asyncio was almost always slower than threads, in terms of just raw throughput (as seen from nethogs)
I do actually - just dug it out: https://imgur.com/a/0uIQW6f This was an experiment I did against a very slow API. It's 5 years old and so I was using Python 2.7. The graph shows number of completed requests per second against number of threads. As you can see at first the throughput linearly increases with threads but then plateaus at around 12 threads.
I haven't done the same experiment with async-io but it would be quite easy using this package `many-requests`
How do you know the bottleneck is python's threads, and not the server or your network? If its the latter, no amount of async programming will improve this graph.
many-requests is the name the package was registered with on pypi.org. So you install with: pip install many-requests.
That installation creates a folder named many_requests inside the site-packages directory on your PYTHON_PATH. It uses underscore rather than hyphen because modules/packages must be legal Python names. A hyphen would be parsed as subtraction.
24 comments
[ 3.0 ms ] story [ 56.3 ms ] threadIt has been tested in the wild with more than 10 million requests. Automatically handles errors and executes retries.
Also provides helper functions for executing embarrassingly parallel async coroutines.
It was pretty remarkable to go from 32 cores being maxed for 100 concurrent workers to... A 10-15% usage spike on a 4 core XPS :)
https://github.com/gocolly/colly
Our typical CRUD Spring backends can handle ~15,000 requests a second with no optimization.
I appreciate all the effort that's gone into making Python faster but I think it's still the wrong tool for the job at this point if heavy load is involved
There is a space for go and other languages, but also for libraries like these.
Is vanilla Python networking is still dog slow, on the order of 1k requests per second per core on mid-range consumer hardware no matter how you wire it all together?
Here's a complete, similar example from the Python docs: https://docs.python.org/3/library/concurrent.futures.html#th....
Granted that example is more lines of code than the README for this library. But the core of it is fairly straightforward:
Also `many-requests` has further logic to deal with retries and bad responses.
But I think requests already handles retries on its own.
I haven't done the same experiment with async-io but it would be quite easy using this package `many-requests`
That installation creates a folder named many_requests inside the site-packages directory on your PYTHON_PATH. It uses underscore rather than hyphen because modules/packages must be legal Python names. A hyphen would be parsed as subtraction.
And inversely, PyPI normalizes the package name (the name you give to `pip install`) to use hyphens instead of underscores. https://www.python.org/dev/peps/pep-0503/#normalized-names
What is the current best practice way to parallelize reading many parquet files on S3 using pyarrow? Deep down, it's just lots of HTTP requests.