It's a shame, httpx has so much potential to be the default Python http library. It's crazy that there isn't one really. I contributed some patches to the project some years ago now and it was a nice and friendly process. I was expecting a v1 release imminently. It looks like the author is having some issues which seem to afflict so many in this field for some reason. I notice they've changed their name since I last interacted with the project...
Always remember that open-source is an author’s gift to the world, and the author doesn’t owe anything to anyone. Thus, if you need a feature that for whatever reason can’t or won’t go upstream, forking is just about the only viable option. Fingers crossed!
What is it about Python that makes developers love fragmentation so much? Sending HTTP requests is a basic capability in the modern world, the standard library should include a friendly, fully-featured, battle-tested, async-ready client. But not in Python, stdlib only has the ugly urllib.request, and everyone is using third party stuff like requests or httpx, which aren't always well maintained. (See also: packaging)
I'm not a lawyer, but are there any potential trademark issues? AFAIK in general you HAVE to change the name to something clearly different. I consider it morally OK, and it's probably fine, but HTTPXYZ is cutting it close. It's too late for a rebrand, but IMO open-source people often ignore this topic a bit too much.
The lack of a well-maintained async HTTP client in Python's stdlib has been a pain point for a while. Makes sense someone eventually took it into their own hands
Do you see yourself taking over httpcore as well as it's likely to have the same maintainership problem? It would certainly instill more confidence that this is a serious fork.
This certainly wouldn't be the first time an author of a popular library got a little too distracted on the sequel to their library that the current users are left to languish a bit.
This sounds like an ideal use case for modshim [0]
One of its intended use cases is bridging contribution gaps: while contributing upstream is ideal, maintainers may be slow to merge contributions for various reasons. Forking in response creates a permanent schism and a significant maintenance burden for what might be a small change. Modshim would allow you to create a new Python package containing only the fixes for your bugbears, while automatically inheriting the rest from upstream httpx.
> The fix was ignored and there was never any release since November 2024. Me, and others, asked repeatedly for a release containing my fix. I sent email to the author personally. I got response when I added that I was considering forking. The author replied “1.0 development is on course”.... I do understand about maintainer burnout, and preferring to work on ‘next’, and that there is life outside of Python, but I think not doing anything for maintenance and also not letting other people help out in maintaining, for such a high profile module, is problematic.
I feel like it's counterproductive in situations like this to mention forking. It will come across like a threat, when there isn't really anything intrinsically aggressive about it. So just do it; and when you have a decent amount of separate development, you can decide whether to make PRs back, advertise your fork, etc.
It's been a pleasure to use, has a httpx compatibility layer for gradually migrating to its API, and it's a lot more performant (right now, I think it's the most performant Python http client out there: https://github.com/MarkusSintonen/pyreqwest/blob/main/docs/b...)
I wrote a framework, link below, which uses them all. You can compare each to verify crawling speed. Some sites can be cleanly crawled with a one particular framework.
Having read the article I am in a pain. I do break things while development. I rewrite stuff. Maybe some day I will find a way to develop things "stable". One thing I try to keep in good shape is 'docker' image. I update it once everything seems to be quite stable.
26 comments
[ 2.7 ms ] story [ 44.1 ms ] threadI am trying to resolve what you've seen. For years of hard work.
Just a small headsup: clicking on the Leiden Python link in your About Me page give not the expected results.
And a small nitpick: it's "Michiel's" in English (where it's "Michiels" in Dutch).
Thanks for devoting time to opensource... <3
Always remember that open-source is an author’s gift to the world, and the author doesn’t owe anything to anyone. Thus, if you need a feature that for whatever reason can’t or won’t go upstream, forking is just about the only viable option. Fingers crossed!
Loved that little detail, reminds me of the old interwebs :)
Node.js got its production version in 2023.
Rust doesn't include an HTTP client at all.
Even for stdlib that have a client, virtually none support HTTP/3, which is used for 30% of web traffic. [1]
--
HTTP (particularly 2+) is a complex protocol, with no single correct answers for high-level and low-level needs.
[1] https://radar.cloudflare.com/adoption-and-usage
"So what is the plan now?" - "Move a little faster and not break things"
This certainly wouldn't be the first time an author of a popular library got a little too distracted on the sequel to their library that the current users are left to languish a bit.
One of its intended use cases is bridging contribution gaps: while contributing upstream is ideal, maintainers may be slow to merge contributions for various reasons. Forking in response creates a permanent schism and a significant maintenance burden for what might be a small change. Modshim would allow you to create a new Python package containing only the fixes for your bugbears, while automatically inheriting the rest from upstream httpx.
[0] https://github.com/joouha/modshim
I feel like it's counterproductive in situations like this to mention forking. It will come across like a threat, when there isn't really anything intrinsically aggressive about it. So just do it; and when you have a decent amount of separate development, you can decide whether to make PRs back, advertise your fork, etc.
It's been a pleasure to use, has a httpx compatibility layer for gradually migrating to its API, and it's a lot more performant (right now, I think it's the most performant Python http client out there: https://github.com/MarkusSintonen/pyreqwest/blob/main/docs/b...)
- httpx
- curl cffi
- httpmorph
- httpcloak
- stealth crawler
I wrote a framework, link below, which uses them all. You can compare each to verify crawling speed. Some sites can be cleanly crawled with a one particular framework.
Having read the article I am in a pain. I do break things while development. I rewrite stuff. Maybe some day I will find a way to develop things "stable". One thing I try to keep in good shape is 'docker' image. I update it once everything seems to be quite stable.
https://github.com/rumca-js/crawler-buddy