A faster version of black is not a strong enough value proposition for an entire company. Ruff should remain an open source product. What is the point of making every half decent Developer tool a whole startup?
I’m not against folks getting paid. My question is how does one build an entire company selling a slightly better version of a widely used open source tool? Ruff is not strong enough to build a company from.
Python has become a hugely critical language for science, AI/ML, finance, etc. and the current state of tooling has lagged the language's importance. I can think of lots of ways a company that solves that problem could add adjacent products and monetize them. Enterprise support, tooling for building and deploying custom lint rules, supply-chain security, managed builds, etc.
I mean, these are the famous last words of a lot of non-visionaries. I'm not saying that Ruff is some kind of unicorn, but there are a lot of cases where a seemingly small improvement on an existing technology resulted in a very successful enterprise. Docker, for example. There are others that I'm sure people will chime in with.
I don't know their plan, so I can't speak for them.
What I would do is build an entire ecosystem of that quality that would include a tool to solve the Python distributions problem.
Either you help with deployment on the server, and you offer hosting.
Or you help with making an installer, and you offer nodes to build installers for multiple OS and upload to multiple app stores, manage updates, cdn, permissions...
You can even start small and just help with a service for cross-compiling C extensions and scale from that.
Or provide machine learning analysis of the quality of your code and make companies pay for it.
Or go full Continuum.
They are good enough that they can pick and choose whatever they want, really.
When you solve pain, people pay. If readthedoc managed to survive by being a static rst site, astral has a shot provided they keep the business side of things in mind as nicely as they build their user stories.
A lot of companies would pay actual money for some semblance of supply-chain security. Hosted, verified, certified Python dependencies. This is how Red Hat made all their money in Linux. Something like "use our vetted and secure pypi instead of the free-for-all full of typo squaters and package takeovers that the public pypi.org offers."
Starting with some nice developer tooling and going from there doesn't seem crazy at all.
I agree, but I also don't make the decision. The free market decides whether their value proposition is enough, so the company's success is dependent on the developer demand.
I agree with you. On the other hand, if you build something cool you may want to make some money...
The problem imo, is changing the speech in the middle of the way, aham openai
this is too narrowly focused on what Ruff does today and not enough on what it could do for the Python ecosystem as a whole. his focus and clear execution has built an incredible wedge and brand, and his next product will probably be well received, and the next, and the next.
the opportunity to bring speed and sanity to the whole Python Ecosystem tooling is large (if you dont feel the pain, you don't do enough python) and honestly i cant belive anyone has been (crazy enough) to try this since Anaconda.
I'm inclined to agree and when they eventually die, what happens to the tool they built?
I've been using https://rome.tools and really love the work they put into it. It's clear they had people working fulltime on it. But now, what? The code is open-source, there are people working on it, but development has mostly dropped off. I guess that's okay? It just adds a lot of doubt into the longevity of the project.
I'd be wary adding these tools into your stack because their progress relies pretty heavily on VC funding and a tight runway to profitability.
It's much more noticeable when running locally. Going from something like black + pylint + mypy running in a pre-commit hook to black + ruff + mypy has been wonderful for me.
It lets me actually set up another terminal session to run ruff on every file change - where pylint would take seconds, ruff is essentially instant.
Side note: I really hope mypy can get the same treatment; it runs quickly once its cache is established, but it's terribly slow running from scratch.
I'm currently speeding up Mypy + Jedi in Rust. I'm pretty far already and it's definitely a lot faster. Tests are currently running 500 times faster than Mypy.
These don't have to be run on every file in the repo, only the ones that have changed. At least not often. If it takes seconds, there is something wrong.
I love Ruff and I'm glad that Charlie and the rest of the team are able to work on such tools full-time. I'm also happy to see that the author of Maturin (Rust+Python interop) is involved, as Maturin is a fantastic project with great ease-of-use.
For those who aren't familiar, Ruff is a very fast Python linter that supersedes a variety of tools like isort, flake8, and perhaps eventually Black [1]. My understanding is that since Ruff is an all-in-one tool, it can parse the underlying Python files just once -- and since this is the most expensive phase of these kinds of tools, adding any extra functionality on top (such as linting, formatting, ...) is very cheap. And that's to say nothing of the fact that Ruff is built on Rust, with the usual performance wins that Rust projects tend to have.
I'm a little apprehensive of how development will look given a VC raise with the expectation of VC returns. But for now, I'm delighted and can't wait to see what the team comes up with.
if it's any consolation, Accel is in my mind one of the "good ones". they have a record of backing "good" devtool companies that manage open source and commercial concerns well - Vercel, Sentry, etc. (i'm sure theres more, those are just the two that i'm closest to)
tbh i care more about individual investors than the fund brand, but its a fallback whenever meeting someone new
edit: https://overcast.fm/+t_0aYbn-o/12:00 listen to steve krouse talk about how accel encouraged him for val.town even tho he didnt initially believe in it himself. thats not normal investor behavior. look out for people like that, by definition they are rare
I thought Black was an auto-formatter, similar to clang-format, but for Python. Ruff seems to be only doing linting, not formatting. Am I missing something?
Pretty bold to try to replace Flake8 and Black with one tool. I hope they succeed. Would be great to have a tool that does both and is substantially faster.
I do hope they'll expose Ruff as a Python module / API in the future. I'm currently using Black to format Python code that's in-memory (never gets written out to disk).
With Black (as an imported Py module), it's just a matter of passing in a string and getting one back. With Ruff as it is now, I'd have to write that out to disk, spawn Ruff process, then read the formatted file back in. Do that for many files and the speed advantages disappear, and actually it's slower.
Substituting bad.py and good.py for in-memory os.pipe objects for your data. This still involves spawning a subprocess, which can add up, but you're not having to read/write to disk at least.
* run ruff command many times (with different input file each time)
* write all the files to "disk" (tempfs actually so it doesn't involve disk access) and run ruff once
Currently with black I avoid that by importing it as a module once so I get the benefits of both (and is simple to boot).
In other comments here I've read that a person heavily involved in Python-Rust interop is also part of the team, so there's hope they'll expose ruff as a module too and magically whisk my dilemma away :-)
How did Hashi corp make money (and an eventual billion dollar plus IPO!) developing open source tools like vagrant, consul and terraform...
Clearly the creator of ruff is going to expand from an open source tool into a company focusing on Python tooling and developer experience, services, etc. There is a market for stuff like that just based on pycharm and jet brains' success in the space alone.
im sure they have a ton of ideas. my bet is that they might launch something in the CI/CD space. like building python packages/images fast, speeding up python testing or tackling something in ML inference.
Support contracts. Custom features built to order. Other commercial products based on the open-source libraries for parsing and analysis. Running a hosted service.
All this while remaining a team of 2-3, hopefully, with all company-running stuff minimized and outsourced.
I think we're going to see a shift in developers expectations towards the cost of their tools, as a side-effect of Copilot & co. If you already pay $100/mo in subscriptions for IDE, its plugins, and services it integrates with, then forking additional $5 for a linter doesn't look as absurd as it used to...
just starting to dabble in Rust, and I've seen PyO3 mentioned a number of times for writing rust bindings. What's the difference between Maturin and PyO3?
PyO3 is the library that enables Python <-> Rust bindings, Maturin is a build tool for packaging PyO3 Rust libraries (which export Python APIs) as Python packages!
The main reason pylint is slow is it tries to infer the types of the variables, and it predates type annotations so it can go through multiple functions/control paths across multiple files to figure these out. This approach is obviated by type annotations and type checkers.
Ruff is built for speed from the start and doesn't look at type annotations because you should run a type checker alongside.
that's dodging the question a little bit? let's say Ruff is faster because it's so good, how much faster than type checking alternatives would this great tool be if it did type checking?
Without building the tool to compare, it's really hard to tell. In my experience of rewriting some Python and Ruby code into compiled versions, you can expect 10-20x improvements. But it's both really dependant on the code itself and on how you're going to reimplement it - most of the time it doesn't make sense to do perfect 1:1 rewrites.
That depends on how you do type checking and more specifically type inference. Mypy infers types of local variables but not function boundaries as it expects you to annotate function arguments and return values.
In mypy a variable taking values of two types is an error (unless explicitly annotated) which means you can assume the type from the first assignment statement and then use that assumption every other time the variable is used/assigned. In pylint it allows variables to take multiple types and only emits errors when an operation is invalid against one of the types. E.g. `x[0]` is valid for lists and tuples so you could assign it a list or a tuple depending on some condition.
The number of possibilities it considers quickly multiplies. Eg if your function is of the form `if blue_condition: x = blue_action(x)` that's a doubling of the possibilities and ten flags -> 1024 possibilities. (pylint has some heuristics on when to give up.)
The advantage of pylint is that it doesn't require your code to fit a certain style. But nowadays people have type annotations and they prefer to use them even if it means changing their code style a bit here and there.
Other type inferers with different approaches are pyright, Jedi, Pyre, pytype etc with different tradeoffs.
I actually cursed Maturin a few years ago, as it refused to compile on OpenBSD and hence broke some lib I planned to use.
Python is becoming a bit too reliant on Rust. Rust is good but, in term of portability, is just not as mature as C. If your lib relies on Rust, please please please test it on something beyond Linux and Mac.
> please test it on something beyond Linux and Mac
That’s asking a lot. If your platform is poorly supported by Rust, then maybe that’s where your efforts should be directed. If you fix that, lots of interesting stuff beyond one library is unlocked.
That email (from Dec 2017) ends with “Such ecosystems come with incredible costs. For instance, rust cannot even compile itself on i386 at present time because it exhausts the address space.”. I presume Theo is actually complaining about using more than 3 GB (?) of memory, but still it really shows the different cost-versus-benefit decisions that we all make.
Yes. My recollection is that around that time, llvm and/or rustc itself would sometimes go over that. I don't know off the top of my head what memory usage looks like right now.
Is it actually reasonable to expect compiler toolchains to work on old or underpowered hardware? Or is the real problem that cross-compilation is still a special case, rather than being the only way compilers work? If it wasn't still normal to depend on your target environment being the same as your build host, would anyone really want to do serious development work directly on their RPi/Gameboy/watch/whatever, just because that is the target runtime environment?
I am wildly guessing that Theo’s beef is more that rust uses a lot of memory (paraphrase: 640kb should be enough for anybody). OpenBSD does integration builds on a variety of different systems, and maybe Theo noticed the OpenBSD/386 build failing due to lack of necessary memory?
That seems a reasonable guess, but it brings me straight back to wondering why such a diversity of build environments is necessary or useful. And my wild guess is that it's mostly because cross compilation is still a second class citizen in most languages today. Though I guess it could be a kind of cultural expectation that you should be able to compile the whole OS on the hardware you're running it on.
My understanding is that the policy of the project is that the base system must not be cross compiled. My understanding of why is that they want to be able to fully bootstrap from base itself.
I guess at the end of the day this is all that really matters. If there are specific goals that Theo has around this, then there's no point in me second-guessing whether those are "reasonable"; it's a matter of values and preferences and whatever.
Whereas if this conversation was about something with broader stewardship, like _Linux_, I'd be saying this is silly, you shouldn't be compromising other things just so you can build your RPi kernel on an RPi.
I think latchin onto this example misses the crux of his raitonale:
> In OpenBSD there is a strict requirement that base builds base. So we cannot replace any base utility, unless the toolchain to build it is in the base.
> Such ecosystems come with incredible costs.
Basically, the cost of adding rust to the OpenBSD base system currently far outweights the proposed benefits (reimplement fileutils), especially considering people will probably not want to pour in the effort needed to rewrite those with strict POSIX compliance (which is another requirement).
He's not saying rust isn't useful but that it wouldn't be a net benefit to have a hard dependency on it in the base system. In the BSD world folks take a very strict and conservative view of what can go in base (and for good reason IMHO).
This is different from the GP comments about just making it possible to use rust programs/libraries in OpenBSD, so we're definitely on a tangent here.
> please please please test it on something beyond Linux and Mac
Why? Simply because you use something that even 95% of the unix folk do not?
I can totally see how they wouldn’t be concerned with that. Just as I’m not concerned with making my JS lib work for those weirdos that still run IE 5.5.
>If your lib relies on Rust, please please please test it on something beyond Linux and Mac.
No, thank you, I'll just add Windows. Your choice to use platforms that nobody else does save for a few specialists does not constitute a need on my part to support your favorite. In the same way that if I package for Debian, Ubuntu and RHEL, it's your problem that it's not on Arch or unsupported by your custom Hannah Montana Linux. Feel free to submit PRs though.
Ruff is great, but it lacks a lot of the rules[1]. As far as I understand it reads every file in parallel, if that is still the case there are categories of issues it cannot detect. In my case ruff cannot fully replace my existing tools, but makes for a great companion.
Having written a bunch of Flake8 plugins, including custom plugins for internal use at a company, and using 5-10 popular plugins on every Python project I work on... rewriting the entire linting ecosystem in one monolithic Rust tool doesn't feel like the best solution. There's no good story for building an ecosystem around this yet, and I think that's a big hurdle to overcome.
Ruff is fast, sure, but the benchmarking seems a little disingenuous, as I believe their number includes caching, but doesn't necessarily include caching for other tools. In fairness, not all the other tools have caching, but it is common to run them through pre-commit and therefore only on the current git diff, which speeds them up by orders of magnitude.
There's no reason your custom bespoke plugins couldn't be called by ruff as necessary. It's silly to burden the happy path of 99% of users who just need common sense python linting with those edge cases and custom needs.
- Supporting flake8 plugins, using the existing community, and sacrificing the speed.
- Requiring new plugins, in Python or another scripting language, sacrificing the community progress and goodwill, and sacrificing some of the speed.
Neither of these options is good. The Python linting ecosystem is a mature one with a lot of investment into the existing tools, and rather than try to speed those up (which could be done in a number of ways), Ruff started from scratch.
It doesn't feel like the right decision for an ecosystem that is as community focused as Python, and the engineering reasons feel like a toss up at best.
There is no downside to adding your bespoke flake8 plugins. For people that don't use them (99% of people) they get the benefit of blazing speed. For your custom plugins you live with the tradeoff of slower execution to do those AST passes with flake8/python tools. Even if ruff didn't exist you would still be burdened with your slow flake8 plugins speed. There is zero downside to you and only upside to people that aren't you.
Kinda just sounds like you're grousing because somebody moved the cheese.
It's not just "custom" plugins, it's all third-party plugins though right? If someone wants to publish a new linter for something, right now they can, and others can use it easily, but Ruff centralises that and makes it harder.
You're right that it will probably still be faster overall because "most" linting will be done with Ruff and any extras would be done externally, but now you've either got 2 tools when you had 1 before, or you've got to shell out to Python which adds overhead, or you've got to rewrite plugins in a Ruff-compatible format, or something else.
> Kinda just sounds like you're grousing because somebody moved the cheese.
I'm just disappointed that someone looked at slow linting and decided the answer was their own new tool, rather than participating in the existing community. Now the effort has forked, it'll take more work overall in the community, and we were already lacking engineering resource.
I'm disappointed the flake8 community hasn't prioritized performance and has led to python linting being much less widely utilized. I'm looking forward to tools like ruff giving much faster and more usable linting.
Yes the point isn't esbuild will have magic pixie dust that makes nodejs AST processing magically faster--that's objectively impossible. The point is you can migrate to the new system over time without losing critical plugin functionality right now. It's not to live in a steady state with old slow nodejs plugins.
Having now switched several projects from webpack+babel to esbuild, with a bunch of plugins to replace various bits of resolution magic, SCSS compilation, and more, the performance gains are consistently astronomical. Simple builds go from taking several seconds to consistently completing in an unnoticeable amounts of time. Full rebuilds are still significantly faster than incremental (--watch) builds used to be.
Setting this up takes a bit more work than for webpack, and it's easier to run into limits on what's reasonably possible, but I don't intend to ever go back.
> Ruff is fast, sure, but the benchmarking seems a little disingenuous, as I believe their number includes caching, but doesn't necessarily include caching for other tools.
Those progress bars with time of other linters - is it a joke or it's a data from stupidly enourmous codebase? Or slow (>1s) linters is something that is normal in python?
It's linting the entire CPython codebase and standard library, which is quite large (possibly one of the largest python codebases). But yes, pure python linters will be inherently quite a bit slower than a native implementation. The point of ruff is that it's so fast you forget or never even notice it's running. Every change, every little modification, etc. should always be linted with near instant feedback.
I worked on a 50k-100k loc python codebase where the linters took a full minute to run if you deleted your cache. And I did try to optimize it. I wouldn't be surprised if that's normal.
If you have to do things like parsing or AST walking a lot, you run into the slow performance of the core bytecode interpreter pretty quickly. For Python, this is a compute-heavy workload.
I think it's actually basically the same. Being a clean, organized cook is probably positively correlated with making good food, but it's not strictly necessary. If someone is writing sloppy code but shipping a project of this overall quality without leaking the "low quality" of their implementation, then I wouldn't presume to call them a "bad dev" - I would probably conclude that they're more efficient than I am at prioritizing where to spend organizational energy.
In this analogy, the food is the entire project, and a dev does much more than coding. You can do one thing right with luck, you can't do most things right by luck alone.
The guy is fluent in 2 languages, one being known to be hard to master, create a tool that replace several others, get adopted in months by half the community, welcome 172 contributors on a project that is parsing stuff, a hard problem. Also the doc is good.
As a professional dev, I never get all those right. Never.
So yes, the food is good, and the chef is excellent to get all that stuff right.
then you probably dont know that i'm actually more supportive of charlie than most and are reading negative intent from a perfectly neutral/innocent question that is perhaps badly phrased :)
i mean i kinda like that HN de-emphasizes usernames, conversation should stand on its own if its to be judged by own merit. but i do have 14k karma haha
No it was not. The question was directed at the commenter about assessing skill, never made any statement disparaging said skill.
Also you seem to be unaware of the relationship of this commenter - you may want to examine their Twitter from before this story was submitted here. Or go on and prove yourself a prat, I won’t stop you.
You're out of line, no need to name-call. I may not want to examine the website and twitter feeds of every commenter on here. Perhaps it wasn't rude, but given how pedantic many people here are, I took the original comment to be a criticism of praising the coder, not a genuine question of evaluating code quality.
> I may not want to examine the website and twitter feeds of every commenter on here.
While that sounds fair enough, the commenter had already noted their relationship within the original thread here by the time you decided to make that completely dismissive reply.
If you’re going to just tritely counter a statement it seems rather foolish not to do a bear minimum of observation.
When I worked at Khan Academy, Charlie was an intern, and for a few months we worked together on building the KA app for Android. I can vouch that he was a damn good developer then, and I'm sure he's a much better one now.
That's a very good point. We love to talk about speed in dev tools, probably because it's an easy metric to track, but usability and good features are really the key points for making a tool that people want to use and keep using. Too many open source projects neglect the product angle in favor of the technical.
Perfect timing, I’m excited to what Astral will bring.
The python ecosystem keeps building momentum as “doing things with data” becomes bigger, more accessible, and also more (near) real-time and I think the ecosystem would really thrive with better, unambiguous tools that become de facto to the community of builders instead of plenty of suboptimal ones to choose from.
In the last few weeks I've been using Python to play with some LLMs locally, and at first I thought there was some bug in VS Code because the linting was so slooooooow.
This seems long overdue! Looks like a great project :)
Speed is a non existent problem for linters. Pyflake is quick enough. I spend more time thinking than writing code. I just write in one file at a time, which can be linted in sub-second time.
How does it compare with Pyflake8 in error messages? Does it find more errors? Does it have less false positives? Does it integrate well with other developers tools? Does it have sane defaults?
These are the really important questions that aren't answered in the site.
I disagree about the speed. I don't have a problem with pyflake, but running `time ruff -s .` in a project with 1,236 Python files took 78ms. At that speed, it could re-check the file I'm working on in an editor after every keystroke with no noticeable latency. It's not just a little bit faster. It's freakishly, ridiculously, gone-plaid faster.
Edit: for comparison, flake8 took 8.19s and found approximately the same number of issues. pyflakes took 4.49s and found fewer.
That’s not true unless you have a dependency map between all modules. (Note: that’s what I did in pytest-fastest to only retest modules that had changed, or that imported modules that had changed.) Otherwise, if you rename a function, you wouldn’t know what all broke.
1. Fast enough to be live updating as you type in an IDE.
2. Slow enough that running a linter has to be a separate action you take, but fast enough that you don't go do something else while it's running.
3. So slow that it's an asynchronous task that you launch and then come back to later.
Ruff is in the first category, while most other python linters are in the second. This level of performance enables a qualitative difference in how you interact with the tool. If you are invoking it as a separate task, then going from 500ms to 50ms is indeed not very interesting, though.
> Ruff supports over 500 lint rules, many of which are inspired by popular tools like Flake8, isort, pyupgrade, and others. ... By default, Ruff enables Flake8's E and F rules. Ruff supports all rules from the F category, and a subset of the E category, omitting those stylistic rules made obsolete by the use of an autoformatter, like Black.
A VC raise gives the team cash to work on this full-time, that's great news.
What worries me is what possible possible path can exist that gives VC-returns that is not at odds with customer happiness? Customers being Python developers.
I'm glad people are continuing the promise of Rome. High quality, high performance programming language tooling is a great mission and very much an unsolved problem. And tbh, the Python ecosystem is a great space. There's plenty to be built.
Agreed. I have a policy of _never_ clicking on a link if I don't have at least _some_ idea of what's behind it, and that includes HN. It's the top link on HN right now and I came to the comments just to see what it was even about.
And yes, I have regretted clicking on HN links before.
Banning is a little extreme, but I agree that title is unhelpful. Better title would be "Ruff is a fast Python linter written in Rust" (which is a slight paraphrase from one of the subheads).
especially when it doesnt seem to be the ruff founder who submitted this particular post. maybe dang or someone will rename it something more appropriate but from hn comments its pretty clear what the context is
I wonder if the eventual goal of this is similar to deno, where it's about building really good, no-hassle tooling/ecosystem for a language and then making it very simple to deploy/host (if I'm misspeaking about deno please correct me, I'm not well versed on it).
One thing I like most about Go is actually the Go tool; having a ubiquitous linter, formatter, test framework, dependency management, etc, all built in and not having to install various tools is huge imo. I think a lot of languages are missing this ease of tooling. I think (?) this is what deno/astral is trying to address for javascript/python and then the business model is once you're using it, it's simple to host with them. Curious what other think
- happy because Ruff deserves full time focus and having a team that can focus on tooling as their main product (not a nights&weekends hobby) is a clear win for everyone;
but also
- concerned because VC is not charity, and this must surely come with strings attached (in terms of future growth); I haven't seen many VCs aiming for "sustainable profitable business providing great value for the community" type exits; then again, if this turns into a Hashicorp-type story that wouldn't be too bad an outcome either
I cannot comment on the feasibility of building a profitable open source business around Python tooling, but I can say that I'm very very glad someone is taking that risk: both Charlie in career/business risks, and the VCs in taking on the financial risk. I just get to benefit - ruff is really nice and I anticipate big improvements to Python tooling as a result, at no cost to me.
Super excited about ruff improvements, but I gotta comment that the CSS for a clicked link in astral.sh looks like normal text. I clicked a link, hit back in Android Firefox, and couldn't find the link again.
heh, I would actually be curious about that environment claim when one balances the amount of power consumed by compiling rust; i.e. the savings for its users balanced against the horrific amount of wattage drawn on the developer's workstations or cloud or ci servers
I now have to compile Firefox overnight because I can no longer compile it and do something else with my (core i7) laptop simultaneously
Sure, as long as there are no tradeoffs involved. But there are. Opportunity cost for one.
Your saved energy calculation is the absolute upper bound possible, the real value is likely close to 0 or even negative (crypto).
Even the upper bound you've calculated is an infinitely small amount of energy compared with everything else we use energy for at the global scale. It's irrelevant.
If you want to save the world, do something that has a clear positive relationship towards it. Ditch your car, plant a tree, vote for the right politician.
> nowadays, it's cool to say this new toy is faster than the rest because it is written in Rust without providing further details?
What more details do you want? You run it, and it runs in milliseconds rather than seconds. I don't care if it's because it's written in Rust or because they sacrificed a goat to Baal, I care that it's fast.
> Ruff is a linter, not a type checker. It can detect some of the same problems that a type checker can, but a type checker will catch certain errors that Ruff would miss. The opposite is also true: Ruff will catch certain errors that a type checker would typically ignore. ...
247 comments
[ 2.1 ms ] story [ 237 ms ] threadI mean, paying people who work on it, for one?
I mean, these are the famous last words of a lot of non-visionaries. I'm not saying that Ruff is some kind of unicorn, but there are a lot of cases where a seemingly small improvement on an existing technology resulted in a very successful enterprise. Docker, for example. There are others that I'm sure people will chime in with.
Like sentry made a good logging library, then pivoted to an observability service.
And today I use sentry because I have a great history with their product.
It's smart and a positive way to make money.
I dig it.
PS: ruff is not replacing black (although it will probably in the end), but compete with flake8 and pylint.
Serious question, what is the path for a linter? Where else are people paying for linting as a service?
What I would do is build an entire ecosystem of that quality that would include a tool to solve the Python distributions problem.
Either you help with deployment on the server, and you offer hosting.
Or you help with making an installer, and you offer nodes to build installers for multiple OS and upload to multiple app stores, manage updates, cdn, permissions...
You can even start small and just help with a service for cross-compiling C extensions and scale from that.
Or provide machine learning analysis of the quality of your code and make companies pay for it.
Or go full Continuum.
They are good enough that they can pick and choose whatever they want, really.
When you solve pain, people pay. If readthedoc managed to survive by being a static rst site, astral has a shot provided they keep the business side of things in mind as nicely as they build their user stories.
Starting with some nice developer tooling and going from there doesn't seem crazy at all.
After the core-js debacle[0] earlier this year, it was evident that alot of companies actually do not care care about supply-chain security.
Those that do will happily roll their own hosted repositories that provide little to no guarantees.
[0] https://github.com/zloirock/core-js/blob/master/docs/2023-02...
The natural end state is yet another build service.
There's a FOSS SAST product for Python already, though, called Pysa: https://pyre-check.org/docs/pysa-basics/
the opportunity to bring speed and sanity to the whole Python Ecosystem tooling is large (if you dont feel the pain, you don't do enough python) and honestly i cant belive anyone has been (crazy enough) to try this since Anaconda.
I've been using https://rome.tools and really love the work they put into it. It's clear they had people working fulltime on it. But now, what? The code is open-source, there are people working on it, but development has mostly dropped off. I guess that's okay? It just adds a lot of doubt into the longevity of the project.
I'd be wary adding these tools into your stack because their progress relies pretty heavily on VC funding and a tight runway to profitability.
Having an automated lint run upon opening a PR seems like a minor expense, especially when you can work on other tickets while you wait.
It lets me actually set up another terminal session to run ruff on every file change - where pylint would take seconds, ruff is essentially instant.
Side note: I really hope mypy can get the same treatment; it runs quickly once its cache is established, but it's terribly slow running from scratch.
For those who aren't familiar, Ruff is a very fast Python linter that supersedes a variety of tools like isort, flake8, and perhaps eventually Black [1]. My understanding is that since Ruff is an all-in-one tool, it can parse the underlying Python files just once -- and since this is the most expensive phase of these kinds of tools, adding any extra functionality on top (such as linting, formatting, ...) is very cheap. And that's to say nothing of the fact that Ruff is built on Rust, with the usual performance wins that Rust projects tend to have.
I'm a little apprehensive of how development will look given a VC raise with the expectation of VC returns. But for now, I'm delighted and can't wait to see what the team comes up with.
[1]: https://github.com/charliermarsh/ruff/issues/1904
tbh i care more about individual investors than the fund brand, but its a fallback whenever meeting someone new
edit: https://overcast.fm/+t_0aYbn-o/12:00 listen to steve krouse talk about how accel encouraged him for val.town even tho he didnt initially believe in it himself. thats not normal investor behavior. look out for people like that, by definition they are rare
I do hope they'll expose Ruff as a Python module / API in the future. I'm currently using Black to format Python code that's in-memory (never gets written out to disk).
With Black (as an imported Py module), it's just a matter of passing in a string and getting one back. With Ruff as it is now, I'd have to write that out to disk, spawn Ruff process, then read the formatted file back in. Do that for many files and the speed advantages disappear, and actually it's slower.
* run ruff command many times (with different input file each time) * write all the files to "disk" (tempfs actually so it doesn't involve disk access) and run ruff once
Currently with black I avoid that by importing it as a module once so I get the benefits of both (and is simple to boot).
In other comments here I've read that a person heavily involved in Python-Rust interop is also part of the team, so there's hope they'll expose ruff as a module too and magically whisk my dilemma away :-)
Clearly the creator of ruff is going to expand from an open source tool into a company focusing on Python tooling and developer experience, services, etc. There is a market for stuff like that just based on pycharm and jet brains' success in the space alone.
But you might be able to do support contracts, like ansible. Tough road though.
All this while remaining a team of 2-3, hopefully, with all company-running stuff minimized and outsourced.
Ruff is built for speed from the start and doesn't look at type annotations because you should run a type checker alongside.
Also, does Ruff work well on codebase without type hints?
Ruff will work fine without type hints.
I believe it rightfully leaves it to mypy for those who want those features.
Mypy transpiles itself to c using mypyc and that can still take a while to complete when caches get invalidated.
Ruff works well regardless of whether you use type hints.
In mypy a variable taking values of two types is an error (unless explicitly annotated) which means you can assume the type from the first assignment statement and then use that assumption every other time the variable is used/assigned. In pylint it allows variables to take multiple types and only emits errors when an operation is invalid against one of the types. E.g. `x[0]` is valid for lists and tuples so you could assign it a list or a tuple depending on some condition.
The number of possibilities it considers quickly multiplies. Eg if your function is of the form `if blue_condition: x = blue_action(x)` that's a doubling of the possibilities and ten flags -> 1024 possibilities. (pylint has some heuristics on when to give up.)
The advantage of pylint is that it doesn't require your code to fit a certain style. But nowadays people have type annotations and they prefer to use them even if it means changing their code style a bit here and there.
Other type inferers with different approaches are pyright, Jedi, Pyre, pytype etc with different tradeoffs.
Python is becoming a bit too reliant on Rust. Rust is good but, in term of portability, is just not as mature as C. If your lib relies on Rust, please please please test it on something beyond Linux and Mac.
That’s asking a lot. If your platform is poorly supported by Rust, then maybe that’s where your efforts should be directed. If you fix that, lots of interesting stuff beyond one library is unlocked.
Which doesn't mean he hasn't, of course!
Whereas if this conversation was about something with broader stewardship, like _Linux_, I'd be saying this is silly, you shouldn't be compromising other things just so you can build your RPi kernel on an RPi.
> In OpenBSD there is a strict requirement that base builds base. So we cannot replace any base utility, unless the toolchain to build it is in the base.
> Such ecosystems come with incredible costs.
Basically, the cost of adding rust to the OpenBSD base system currently far outweights the proposed benefits (reimplement fileutils), especially considering people will probably not want to pour in the effort needed to rewrite those with strict POSIX compliance (which is another requirement).
He's not saying rust isn't useful but that it wouldn't be a net benefit to have a hard dependency on it in the base system. In the BSD world folks take a very strict and conservative view of what can go in base (and for good reason IMHO).
This is different from the GP comments about just making it possible to use rust programs/libraries in OpenBSD, so we're definitely on a tangent here.
Why? Simply because you use something that even 95% of the unix folk do not?
I can totally see how they wouldn’t be concerned with that. Just as I’m not concerned with making my JS lib work for those weirdos that still run IE 5.5.
No, thank you, I'll just add Windows. Your choice to use platforms that nobody else does save for a few specialists does not constitute a need on my part to support your favorite. In the same way that if I package for Debian, Ubuntu and RHEL, it's your problem that it's not on Arch or unsupported by your custom Hannah Montana Linux. Feel free to submit PRs though.
1: https://github.com/charliermarsh/ruff/issues/970
Ruff is fast, sure, but the benchmarking seems a little disingenuous, as I believe their number includes caching, but doesn't necessarily include caching for other tools. In fairness, not all the other tools have caching, but it is common to run them through pre-commit and therefore only on the current git diff, which speeds them up by orders of magnitude.
- Supporting flake8 plugins, using the existing community, and sacrificing the speed.
- Requiring new plugins, in Python or another scripting language, sacrificing the community progress and goodwill, and sacrificing some of the speed.
Neither of these options is good. The Python linting ecosystem is a mature one with a lot of investment into the existing tools, and rather than try to speed those up (which could be done in a number of ways), Ruff started from scratch.
It doesn't feel like the right decision for an ecosystem that is as community focused as Python, and the engineering reasons feel like a toss up at best.
There is no downside to adding your bespoke flake8 plugins. For people that don't use them (99% of people) they get the benefit of blazing speed. For your custom plugins you live with the tradeoff of slower execution to do those AST passes with flake8/python tools. Even if ruff didn't exist you would still be burdened with your slow flake8 plugins speed. There is zero downside to you and only upside to people that aren't you.
Kinda just sounds like you're grousing because somebody moved the cheese.
You're right that it will probably still be faster overall because "most" linting will be done with Ruff and any extras would be done externally, but now you've either got 2 tools when you had 1 before, or you've got to shell out to Python which adds overhead, or you've got to rewrite plugins in a Ruff-compatible format, or something else.
> Kinda just sounds like you're grousing because somebody moved the cheese.
I'm just disappointed that someone looked at slow linting and decided the answer was their own new tool, rather than participating in the existing community. Now the effort has forked, it'll take more work overall in the community, and we were already lacking engineering resource.
Setting this up takes a bit more work than for webpack, and it's easier to run into limits on what's reasonably possible, but I don't intend to ever go back.
It looks like the ruff benchmark is run with the `--no-cache` arg: https://github.com/charliermarsh/ruff/blob/main/CONTRIBUTING...
Also: speed is good, but when it comes to linting, that's not what I'd place first as a feature.
How good is the code at spotting my potential mistakes would come before speed for sure.
Hence: it's fast, but is it in any way better than the other solutions out there?
ruff is just really good.
It has sane defaults, it is easy to use and configure. It can replace not one but multiple tools.
From day one it had few bugs or integration problems.
Charlie Marsh is just a damn good developer, on top of driving a good product vision.
That's rare.
Lots of respect to that.
curious how you assessed this. did you inspect the code or are you just commenting from the user POV?
- "This chef is a great cook!"
- "How do you know, have you looked inside the kitchen?"
How you come to the conclusion he's "just a damn good developer" is a very fair question.
The guy is fluent in 2 languages, one being known to be hard to master, create a tool that replace several others, get adopted in months by half the community, welcome 172 contributors on a project that is parsing stuff, a hard problem. Also the doc is good.
As a professional dev, I never get all those right. Never.
So yes, the food is good, and the chef is excellent to get all that stuff right.
I think there's something to be said for praising the developer that is able to understand user needs so well that they create flawless experiences.
Just my opinion of course, but to be considered a "damn good developer," you have to do both.
Yes.
For the brief time he was at Cedar, he contributed a lot of high quality code
I mean, for your own sake, ideally you are not, but I think it’s reasonable.
If you can put your knee jerk emotions aside for a minute and reread that comment carefully - the competence of the dev was never in question.
Also you seem to be unaware of the relationship of this commenter - you may want to examine their Twitter from before this story was submitted here. Or go on and prove yourself a prat, I won’t stop you.
While that sounds fair enough, the commenter had already noted their relationship within the original thread here by the time you decided to make that completely dismissive reply.
If you’re going to just tritely counter a statement it seems rather foolish not to do a bear minimum of observation.
The python ecosystem keeps building momentum as “doing things with data” becomes bigger, more accessible, and also more (near) real-time and I think the ecosystem would really thrive with better, unambiguous tools that become de facto to the community of builders instead of plenty of suboptimal ones to choose from.
This seems long overdue! Looks like a great project :)
How does it compare with Pyflake8 in error messages? Does it find more errors? Does it have less false positives? Does it integrate well with other developers tools? Does it have sane defaults?
These are the really important questions that aren't answered in the site.
Edit: for comparison, flake8 took 8.19s and found approximately the same number of issues. pyflakes took 4.49s and found fewer.
Good time for a full check is when new feature is complete. Run all linters, typers, and full test suite, until clear. Then commit and push.
Ruff has an LSP too so it integrates well with editors unlike flake8 and similar which only work on save — and are really slow
1. Fast enough to be live updating as you type in an IDE.
2. Slow enough that running a linter has to be a separate action you take, but fast enough that you don't go do something else while it's running.
3. So slow that it's an asynchronous task that you launch and then come back to later.
Ruff is in the first category, while most other python linters are in the second. This level of performance enables a qualitative difference in how you interact with the tool. If you are invoking it as a separate task, then going from 500ms to 50ms is indeed not very interesting, though.
https://github.com/charliermarsh/ruff#rules
> Ruff supports over 500 lint rules, many of which are inspired by popular tools like Flake8, isort, pyupgrade, and others. ... By default, Ruff enables Flake8's E and F rules. Ruff supports all rules from the F category, and a subset of the E category, omitting those stylistic rules made obsolete by the use of an autoformatter, like Black.
You can see the current list of supported rules here: https://beta.ruff.rs/docs/rules/
There's also a checklist on this PR which tracks progress on implementing pylint compatibility: https://github.com/charliermarsh/ruff/issues/970
What worries me is what possible possible path can exist that gives VC-returns that is not at odds with customer happiness? Customers being Python developers.
I mean, "Astral" .... what's that ? The "astral.sh" domain doesn't tell me anything either.
I'm sure those in the know automatically know what it is, but for the rest of us, the title is completely and utterly meaningless.
And yes, I have regretted clicking on HN links before.
One thing I like most about Go is actually the Go tool; having a ubiquitous linter, formatter, test framework, dependency management, etc, all built in and not having to install various tools is huge imo. I think a lot of languages are missing this ease of tooling. I think (?) this is what deno/astral is trying to address for javascript/python and then the business model is once you're using it, it's simple to host with them. Curious what other think
- happy because Ruff deserves full time focus and having a team that can focus on tooling as their main product (not a nights&weekends hobby) is a clear win for everyone;
but also
- concerned because VC is not charity, and this must surely come with strings attached (in terms of future growth); I haven't seen many VCs aiming for "sustainable profitable business providing great value for the community" type exits; then again, if this turns into a Hashicorp-type story that wouldn't be too bad an outcome either
countless CPU hours wasted by running dev tools written in slow as molasses languages now getting rewritten in Rust.
build / lint times * number of times builds taken * kWH = saved energy
I now have to compile Firefox overnight because I can no longer compile it and do something else with my (core i7) laptop simultaneously
Your saved energy calculation is the absolute upper bound possible, the real value is likely close to 0 or even negative (crypto).
Even the upper bound you've calculated is an infinitely small amount of energy compared with everything else we use energy for at the global scale. It's irrelevant.
If you want to save the world, do something that has a clear positive relationship towards it. Ditch your car, plant a tree, vote for the right politician.
When I visit the Bun project landing page[0], I get concise reasons as to why bun is faster than its peers.
[0] https://bun.sh/
What more details do you want? You run it, and it runs in milliseconds rather than seconds. I don't care if it's because it's written in Rust or because they sacrificed a goat to Baal, I care that it's fast.
It's usually true, too.
> Ruff is a linter, not a type checker. It can detect some of the same problems that a type checker can, but a type checker will catch certain errors that Ruff would miss. The opposite is also true: Ruff will catch certain errors that a type checker would typically ignore. ...