136 comments

[ 3.0 ms ] story [ 233 ms ] thread
This article implies that batch allocations are some kind of panacea and that raii has no place there.

Neither of these implications have any case made for them, just links to other publications that do the same.

It also alludes to downsides but fails to define them, and also misses documenting any downsides associated with alternatives.

Despite deep familiarity with the space and understanding of both sides, I can’t really see it saying anything much at all in fact

There’s a one-sentence implied claim in here that raii is the reason for c++ rejection and rejection of c++ implies rejection of raii.

The author is clearly unaware of the recent cleanup attribute infrastructure submitted on the c side of the kernel: https://github.com/torvalds/linux/blob/master/include/linux/...

Zig is a fun and interesting language in its own right, there’s really no need for this post-truth era style propaganda. Just focus on your own work

Basically the same conclusion I came to, I was hoping the article would really make a point at some stage but it just fizzled out without really asserting anything with any supporting evidence.

I particularly didn't enjoy how it completely failed to take into account the well-established practice of using RAII alongside arena allocation in C++...

Except Arenas are well expressible in Rust. What's more, the fact that there are lifetime constraints allows to have both safe and idiomatic abstractions for them.
> The fact that Rust developers who are interfacing with the Linux project seem completely unaware of the downsides of RAII, reminds me of when the US ambassador to Denmark thought that their collaborators biked to work because they were too poor to own a car.

I would imagine kernel developers working with Rust are quite aware of the downsides of RAII when it comes to large synchronous drops, and aware of arena allocation patterns. They are also likely aware of both arenas that run destructors on drop, and arenas that don't, and the relative merits of each. They are also likely aware of arenas that you can garbage collect over time, as enabled via generational index patterns.

The good news is that if profiling shows this to be a bottleneck, it is relatively easy to safely switch code that does allocations to using an arena instead (you'd have to add a lifetime parameter to everything tied to the arena).

RAII remains a great way to solve many real problems in systems programming, and banning tools which enable it on the off chance that you'd run into performance issues with synchronous drops, without any data to back it up, doesn't seem wise.

The author's line of thought seems to be concerned with the social aspect of RAII in my reading though. Like how you can write memory-safe C if you're a "good C programmer" (meaning most people evidently won't), these considerations for RAII might be subtle enough that it justifies banning its use altogether (like using memory unsafe constructions is in safe Rust).
A sibling comment [1] points out that the kernel recently gained some RAII-like functionality.

The typical alternative to RAII-based error handling in C tends to be goto statements. I'm sure those of us who have written C are used to writing goto statements in inverse order for their error handling -- and we've also made mistakes with them especially when there's a lot of branching paths (or at least I have). In comparison to that I personally appreciate RAII quite a lot.

Even with arena allocation, it may make sense to have a struct be composed of some memory that's borrowed from the arena and some that's owned by the struct. That works exactly as you'd expect in Rust.

[1] https://news.ycombinator.com/item?id=42291921

This doesn't quite address what I wrote about specifically.
I'm quite sympathetic to social effects of technical choices in general!

You said:

> The author's line of thought seems to be concerned with the social aspect of RAII in my reading though. Like how you can write memory-safe C if you're a "good C programmer" (meaning most people evidently won't), these considerations for RAII might be subtle enough that it justifies banning its use altogether (like using memory unsafe constructions is in safe Rust).

In response I gave you a link which (if you follow it) says:

  * The "goto error" pattern is notorious for introducing subtle resource
  * leaks. It is tedious and error prone to add new resource acquisition
  * constraints into code paths that already have several unwind
  * conditions. The "cleanup" helpers enable the compiler to help with
  * this tedium and can aid in maintaining LIFO (last in first out)
  * unwind ordering to avoid unintentional leaks.
So clearly this is about understanding that goto-based error handling doesn't really scale and adding RAII-like behavior to make it work better.
What you linked to / cite doesn't address what I was talking about either. I understand that you believe it does, so let me clarify why I think it doesn't.

On the top level, you wrote:

> I would imagine kernel developers working with Rust are quite aware of the downsides of RAII when it comes to large synchronous drops, and aware of arena allocation patterns. They are also likely aware of both arenas that run destructors on drop, and arenas that don't, and the relative merits of each.

Thing is, I'm not nearly familiar enough on a low level with constructs like RAII - however, I think this is actually working in my advantage in this case. Because to me what you wrote reads like this:

- RAII in certain scenarios should be avoided

- those scenarios are mindware (and we just trust that kernel devs can and will recall it as appropriate)

- arenas running destructors on drop in certain scenarios should be avoided

- those scenarios are also mindware (and we just trust that kernel devs can and will recall it as appropriate)

Crucially, none of this is about whether RAII is better than any alternative (relative value), but is about its properties (absolute value).

This is also not incompatible with what the article says, in fact, I believe is what the article was sort of peddling inadvertently.

And so my comment was about relating this to the memory management of C in principle. That the flow of the argument (I think) is similar. That the correctness is dependent on the programmer attention (and their correct usage of tooling), while also being high on the skill tree.

Now, I understand if this comes off as an extremely unrevelatory discovery, or if you feel different about the likelihoods involved. But I found it fascinating, if not equally fearsome.

Thank you for clarifying! That's helpful.

> - those scenarios are mindware (and we just trust that kernel devs can and will recall it as appropriate)

Ah, I see. I don't think they're "mindware" so much as they're good targets for making data-driven decisions about. Unlike C memory safety which no amount of measurement or careful thinking can solve, the relative merits of RAII/direct allocations vs arena allocations can be determined case-by-case through running a profiler and trying out things. Rust doesn't really lock you in to one approach, especially at the scale of an individual device driver.

I think there is space here for a more sophisticated critique of Rust, which is that trying out alternative strategies can take some effort and still deliver negative results. For example, I was recently doing some profiling on an application and found that deserializing Cargo's metadata format (and malloc within it) show up a bunch. Out of curiosity, I tried switching the cargo-metadata crate [1] to using zero-copy deserialization. It took a couple hours to do. The end result was around a 15% performance improvement in the deserialization process -- not nothing, but also not hugely impactful, and overshadowed by the act of generating and reading the JSON on Cargo's side. But negative results are still useful.

[1] https://crates.io/crates/cargo_metadata

Thing is, my impression is that running a profiler can get quite convoluted. I'm aware for example of different types of profilers / profiling techniques, but not in detail. I also know that I wouldn't pull out a profiler, unless I was experiencing unacceptable performance, which may not happen on my (or the reviewers') end at all.

... which again rings really similar to the memory management story in C (in my head anyways). You can use tooling, and the tooling does help, but certain tooling only helps in certain scenarios, and if you didn't bother to use the tooling for whatever reason, you're hosed, and you won't find out until potentially much later or even too late.

What I'm essentially getting at is that Rust doesn't provide guarantees that'd ensure the correct employment of RAII or arenas (according to both you and the article at least), meaning their pitfalls manifesting exist somewhere on this range from possible to likely, and then depending on who you ask, the answer may be different.

The article author appears to be of the opinion that this probability is closer to likely, while you - if I read it correctly - think it is closer to the other end of the spectrum. But another interpretation of the article is that the author is just remarking that it is a possibility now at all, and that that makes it a bad thing, regardless of likelihood. So ultimately, my point was to try and get engagement on this idea specifically rather than debating the likelihood itself, especially considering I stand no chance at appropriately gauging that likelihood myself.

Thanks for your response.

> I also know that I wouldn't pull out a profiler, unless I was experiencing unacceptable performance, which may not happen on my (or the reviewers') end at all.

At some level it doesn't matter in that case, right? We only have a finite amount of time in the world to do things, and if it's fast enough then time is better spent elsewhere.

For a GPU driver, I would imagine that unacceptable performance would be visible quite quickly in p99+ frame times, which the gaming community has figured out are important (they call them "1% lows").

My general philosophy having written Rust for a long time (and before that, C, Python and a bunch of other languages) is to nail correctness first, don't do anything too egregious performance-wise (so try and avoid quadratic or worse algorithms), then worry about performance only to the extent that you can measure average and outlier performance. RAII really, really helps in getting correctness right.

(Rust also helps with performance. Idiomatic Rust tends to be quite fast due to aggressive inlining, especially with PGO.)

> ... which again rings really similar to the memory management story in C (in my head anyways).

I think where I differ is that profilers are a fairly mature and well-understood technology. You can even put profiles of Linux or macOS programs directly into a web browser profiler view, and get a really nice idea of where time is being spent (e.g. [1]). By the way, try out samply or a similar tool some time, it's really easy to use.

I just don't think it's possible to get there with any amount of tooling on top of C, especially with existing codebases -- the language is just not rich enough to support the kinds of annotations required for a sound static analysis ("sound" here means all memory safety bugs are in code not otherwise marked unsafe).

So yes, they look formally similar, but the difference in practice is enormous.

[1] https://github.com/mstange/samply

Aggressive inlining is exactly what confuses most profilers in my (limited) experience.
In addition to that, I feel like pointing out that destructors generally compile down to nothing. I have mostly only seen issues with "heavy destructors" in old-school C++ code where you have lots of small allocations and complicated trees of object ownership--which is a model Rust discourages!

I think the anti-Rust people would be better off arguing that they just don't like RAII because it's "magic" (ie. too implicit about cleanup). It's harder to argue against because it's subjective, whereas these arguments against RAII are specious at best.

Agreed (I think arena-style "allocation at the top" makes Rust significantly nicer to write in general, and I borrow from a big chunk of the data at the top quite frequently), though in this case what Lina was describing sounded somewhat close to the big tree of allocations. Now, it could be any or all of:

* inherently required by the hardware, in which case all you can do is shrug

* good for a first-pass implementation and maybe optimized in the future

* simply not an issue in practice.

Without more information, I would trust that Lina made the right decisions and/or will revise her decisions over time based on data.

My impression was just that it was the cleanest way for her to map things on top of the existing kernel structures (which are probably tree-ish). And I highly doubt that there are any actual issues with this aspect of her design; it was just picked out as a contrived example by the blog post.

I suspect if Lina's DRM patches had not come from a Rust dev or invoked "RAII theory", and were only advertised as improvements for clarity and conformance with expected behavior, they probably would have been accepted far sooner. (She says as much in her Mastodon post.)

Why would you trust that Lina made the right decisions? Is it simply because of her past work? Even experienced engineers regularly make mistakes.

It seems like you show up in these threads to give Lina all the benefit of the doubt and her detractors none.

Oh interesting, in what other threads have I shown up?

I would trust her general judgment based on a complex list of factors, including my own personal experience and how my own understanding of the world aligns with hers. I'm exercising my own judgment honed over time.

Because it is working code. Not just talk.
> In addition to that, I feel like pointing out that destructors generally compile down to nothing.

Really? If that's true then RAII didn't do anything in the first place and you could do without it.

And the "everybody uses goto" is a "defer/errdefer" pattern rather than an RAII pattern.

So, in at least 2/3 of the cases, you don't need the full power of RAII and all the problems it brings.

> I think the anti-Rust people would be better off arguing that they just don't like RAII because it's "magic" (ie. too implicit about cleanup). It's harder to argue against because it's subjective, whereas these arguments against RAII are specious at best.

The big failure point is when your RAII object escapes local scope (ie. you pass it back or put it into a container data structure).

RAII means that the entity who gets charged with deallocation is different form the entity who created the allocation.

This is the exact same problem as the old C problem of "allocate an object in one function but expect a different function to deallocate it". C old timers excoriate anyone who does this for good reason. Sure, RAII in Rust means you don't core dump; your performance characteristics just become non-deterministic.

This kind of cleanup action at a distance is essentially "garbage collection" by any other name--only less well controlled.

Quite often the whole point of using a systems programming language is to avoid garbage collection non-determinism--that chunk of people are going to be very upset to see it come back via RAII.

I find it quite nice that I can return owned resources and don't have to pass in buffers for all my out params :)

I'm sensitive to action at a distance problems, but I don't really think this is one. In practice I think the ability to return owned resources leads to better, more scalable development.

It is not relatively easy to switch to arenas in rust at all. It is more time consuming compared to Zig for example. There is a reason almost no one implements it in rust. Even libraries like apache arrow and related query engines don’t implement it.
I've always wondered why not just fork Linux and add Rust there then keep it in sync with mainline and maintain it long time to demonstrate that it's a viable strategy and there's enough volunteers willing to actually spend their time instead of turning the Rust part of the kernel into abandonware when there's so few developers available, before trying to get it supported upstream?
Just fork one of the largest open source projects in the world? Linux has many contributors who are paid full time to improve the project. Something that is impossible to match in a sideshow project.
Exactly my point, if that cannot be done as a third party project, then shoehorning Rust into Linux and expecting unpaid third party volunteers to actually maintain it makes no sense. Unless a big corp steps up to fund such efforts, it's pointless.
It's Linus's project.

He wants to allow Rust. If you don't then you fork it into your own project lol ...

Linus has already decided to accept Rust! A fork would simply not be practical. Disagree and commit.
How much of that is the opinion of the corporate masters who fund the Linux Foundation?

Why Rust and not Ada for example? Rust is heavily promoted by web developers who call the shots these days because they bring in the advertising money.

I find it hard to believe that Rust is the end of all other languages, especially given its ugly syntax.

It turns out that Linus has repeatedly shared his personal opinion on these matters, so you can judge for yourself whether he sounds "forced by corporate masters", or whether that's conspiratorial FUD that exists only in your own mind.

https://www.youtube.com/watch?v=OvuEYtkOH88&t=6m07s

https://www.youtube.com/watch?v=OM_8UOPFpqE&t=8m30s

He's even given briefly opined that Rust looks much better than the "disaster" of Ada: https://www.infoworld.com/article/2247741/linux-at-25-linus-...

Rust has so much momentum behind it. It's become extremely popular among leading-edge systems and tool developers, for being just the right mix of approachable and geared towards sustainable correctness.

For example, the Jujutsu VCS is written in Rust. Of course it is — it was started in 2020 or so, and no other choice would make sense for software which needs to be fast while not compromising on correctness. And Jujutsu is not written by web developers. It is written by systems developers.

I've also been in private conversations to start a couple of new systems projects to fulfill some currently-unmet needs. Again, of course they would be in Rust; there was never any doubt or serious debate.

This by no means indicates that Rust is the end-all of languages. I hope Rust gets superseded at some point! But if you're making a decision today, right now, about systems projects you want to ship and deliver value with, then Rust it is.

Linux is not, by and large, maintained by "unpaid volunteers". And big corps are, in fact, funding the Rust for Linux effort.

Nothing is being "shoehorned". The project has the explicit backing of Linus and Greg KH. It's taking a long time because it's not being shoehorned.

There was a lot of money poured into Rust by large companies and they're looking for immediate and tangible impact (this helps to explain a lot of the negative feelings on all sides imo)

Investors aren't going to abide by a long-term experiment with undefined success outcomes.

I think part of the rustacian movement is to convert others instead of just doing their own thing.
Some work is already done that way but it's quite expensive to go years without upstreaming. Keeping branches in sync is a lot of work.
That is exactly what the Rust for Linux project did: they made a branch of Linux, they did enough implementation work to demonstrate the viability, they maintained it for a while, they had a large number of developers (both volunteer and paid) enthusiastically improving it and building upon it, they talked to upstream about getting it added, upstream agreed and merged it, and the Rust for Linux developers along with many others continue to improve it upstream. That's exactly how the process should work.
Right, but my point about long time maintainability still stands, the person in the lead of the Rust project in the kernel already threw in the towel: https://lore.kernel.org/lkml/20240828211117.9422-1-wedsonaf@...
One person involved in the project got tired of the ongoing obnoxious behavior of a few kernel developers (as partially documented in that mail), and decided to move on. That's a sad outcome, and reflects badly on the kernel developers in question. It doesn't reflect a maintainability issue with the Rust for Linux project (which has quite a few developers and many more contributors), except insofar as it continues to be an occupational hazard of participating in the Linux kernel community that you end up dealing with some number of obnoxious people.
The author of this is the VP of Community for Zig; his only professional experience, according to LinkedIn, is in similar advocacy roles. It's an incredibly bad look for the Zig community then, in my opinion, for him - somewhat sarcastically and disrespectfully ("oh no!", "I already have to waste way too much time for other slow software to load") - trying to school Asahi Lina on "writing performance-oriented software" when she's one of the most impressive software engineers in the public eye right now.

This is just a FUD-y hit piece on RAII. It, frankly, smacks of Dunning Kruger (I know, not statistically a real thing, but rhetorically useful): a writer way beyond his depth hears someone praise RAII, and out of ignorance and professional obligations launches an attack by linking some canned "RAII Bad" sources, not realizing the person they're attacking knows vastly more about...well, basically everything, than they do.

Oh, I didn't realize that. I guess I should add a disclaimer then: a mean and condescending tweet from him caused me to withdraw my sponsorship of Zig a couple of years ago.
(comment deleted)
This is not his best post, granted (there are some good ones).

That said, I think this is a very uncharitable read and resorting to ad hominom (he's not a real dev!) is uncalled for.

If you think the argument is wrong -say so. Refute it on its own merits. It gets no better or worse from it being said by him.

That also goes for Asahi lina - I find it so incredibly tiring when people's hero-worshipping of this admittedly great developer extends to thinking they are simply right about everything.

Carmack is a great dev, he likes his C and C++. Alexis King (Lexi Lambda) used to be a top lisper (likely still is), now a great haskeller. Rich Hickey and so on and so on. Point is - how do ypu reconcile the fact that these people hold wildly different ideas on what makes good programming and programning languages, some of them even changing opinions over their lifetimes.

Will you create a programmer godhood pantheon and claim Asahi Lina is the smartest person ever or write it off as "systems programming must be Rust"? I am curious.

Anyway, not aimed at singling you out, but the appeal to authority (Lina) as some way of saying " this is right, everyone else are wrong" is shoddy arguing.

> If you think the argument is wrong -say so. Refute it on its own merits. It gets no better or worse from it being said by him.

Thing is, one of their points is that the claims listed are unsupported. Challenging unsupported claims is an asymmetrical effort, and is of ill form - those making the claims should be the one providing evidence, the onus is on them after all.

I mean, him first? I definitely implied this, but admittedly others on this thread have made this more explicit: he doesn't really say anything here; he leans on other canned RAII-bad arguments without addressing Lina's points seriously and in good faith, presuming that she's "completely unaware of the downsides of RAII", that the code she's writing is going to have performance problems, that she - someone who's actually worked on the Linux kernel! - doesn't know about a completely imagined intolerance for RAII in the kernel. And as I mentioned, he was kind of a jerk about it too. I think I gave this post the response it deserved.

I know we want to live in a world where everyone rigorously logically defends every position all the time, but in life - especially now, in the world of the internet where everyone has a soapbox - we have to apply heuristics of credibility to make this manageable. "Some guy with limited experience who's paid to promote a programming language" has vastly less credibility than "Linux kernel developer who reversed engineered the M1 GPU" - he needs to come out with a very robust argument for us to take him seriously; instead he posted FUD.

I certainly don't think great devs are infallible. As a particular example - Hickey's a great dev. But I think Hickey's criticism of type systems in "Simple Made Easy" is a flawed argument. I've spent a long time thinking about why it's flawed. If I were to write a blog post about why it's flawed, though, I'd do my best to engage with it seriously; not just presume he doesn't know what he's talking about, presume his code is going to be riddled with type errors, link to some Haskell 101 and call it a day.

> presuming that she's "completely unaware of the downsides of RAII", that the code she's writing is going to have performance problems, that she - someone who's actually worked on the Linux kernel! - doesn't know about a completely imagined intolerance for RAII in the kernel. And as I mentioned, he was kind of a jerk about it too.

In short, and while I don't particularly like the term - he's a textbook mansplainer.

The fact that he thinks Rust forces mass discards shows how little he knows about Rust.

Looking at his social media, his first project with Rust was a month or two ago.

TBF, Asahi Lina's remarks on Zig weren't exactly a good look either (it essentially boiled down to 'Zig is bad because I don't like it') - that left a much worse taste in my mouth than anything coming out of the Zig community, especially since it came from a respected person who really should know better than to shit on other software projects.
I don't see anything in her language that can be interpreted as "Zig is bad", rather than "Rust fits my use case best" along with bullet points that explain that logic.
(comment deleted)
> RAII

Resource acquisition is initialization

The name is terrible, all the more so because we are talking about deacquisition and deinitialization most of the time.
RAII is really all about the deconstructors, not the constructors.
Imagine this: we have a group of friends and we’re having a good time. We all speak English. One or two guys invite some people who are always speaking in Klingon. They invite more and now a large amount of the group wants to have equal support in d&d and other activities for Klingon. They insist it’s a better language when it’s only just a different language, not better or worse. The English speakers are trying to stop the Klingon speakers from changing their group to something new that they don’t like (and not provably better).

Rust is Klingon. C is working fine and is the language everyone is already familiar with. If Rust was actually better, it would flow into the ground organically.

Being technically more memory safe doesn’t make people like it more. Proper tooling with C can prevent memory issues, taking away the main argument for rust.

In the end we don’t need any more of a reason to stop rust other than “we don’t like rust”.

> If Rust was actually better, it would flow into the ground organically.

What would Rust flowing into the Linux kernel organically look like in your opinion?

On a similar note, I work in a non-English speaking country, but have developed a real bad habit of sticking to English when it comes to technical discussions (work or non-work). Mind you, this is an international company we work at, proficiency in English is expected (and often lackluster).

My colleagues don't appreciate this, and will poke fun at it from time to time. They're convinced I'm being obnoxious on purpose, despite me just spending 90% of my day interacting with English in English. It is only natural then that I keep thinking about things in English.

What you'll discover is that these colleagues of mine have two things in common: they're not anywhere nearly as immersed and proficient in English as I am despite working here for a decade, and that they're a bit of an old-timer jackass types.

This was not metaphor.

I don’t know what it looks like, but it’s definitely not forcing people to accept it.

Rust right now just isn’t in a form that some (I personally think most) people can accept. Maybe it’s just that it’s so not-c-like.

… given your language examples, maybe the main issue is just that rust is just so different. I personally find the rust language to be non intuitive… but what’s intuitive for me is heavily informed by c/c++ or python

What I was kind of getting at is that I cannot imagine a new language being introduced without friction, regardless of how organically that happens, and that you probably don't either.

Obviously, there's room for a middle ground here, where we recognize that these are absolutes, and so we start debating whether what would be a less friction inducing way. This could then split into whether it's Rust the language that's causing the friction that exists, or the community on either side.

Personally, I don't see a less friction inducing way being possible, and according to the people involved, it's a mix of both the language and the communities that this friction is the way it is - and I find that to be pretty believable, honestly.

Not only that, and we're getting to very philosophical heights here I admit, but I think this is a pervasive issue with how people usually reason about hardships: that there must always exist a solution to problems. I don't think that's true. I think that things sometimes suck, and there's no way around them, you can only balance between the ways they suck (dilemma instead of a problem). I further think it is all too often this reach-for-a-solution is exploited in rhetoric (and of course, it wouldn't be life without examples for the exact opposite too). It's also a bit like being down. Sometimes it's justified, and there's no need to "correct" or change it.

> it’s definitely not forcing people to accept it

Wait, you mean you've worked on projects where you've achieved unanimous agreement on any large technical decision, ever?

No, sometimes people need to be forced to accept change. The whole "disagree and commit" thing. Humans are strange creatures; we often resist change for the sake of resisting change, regardless of whether or not the change is good for us.

When it comes to the Linux kernel, what Torvalds says is law. He certainly isn't going to go against what many/most of his trusted maintainers want, but ultimately he's going to decide what's best for the project, based on his own reasoning, and the choices are to get on board, or find another project to work on. It wouldn't be the first time he's "forced" people to accept something that he's decided on.

> but it’s definitely not forcing people to accept it.

No one is forced to accept it. The big whiny outbursts from the old guard was when the RfL guys asked them to specify their implementations; they were never asked to touch a LoC of Rust even with a 10ft pole.

> Proper tooling with C can prevent memory issues, taking away the main argument for rust.

This is not true, despite decades of research and billions of dollars poured into static analysis of C programs. Rust solves this by making programmers put pervasive annotations into their code, and even more importantly by making programmers like the fact that they have to put pervasive annotations into their code. Rust today, without additional tooling, delivers a better memory safety story than C static analysis researchers could ever dream about.

This is false; with formal analysis tools like CBMC, you can prove the memory safety properties of your C code. You can even statically prove lack of integer overflows and that asserts are unreachable. I'll agree this is uncommon in general, but CBMC is used for various C codebases where I work. Though this tooling is also available for Rust with Kani.
Is CBMC usable, directly, on any project? Or do you need to write your code in a certain way, using certain patterns, etc.? Or does the CBMC tooling need to be taught various things about how memory management works in your project?

Regardless, the fact that it's not used regularly/routinely with the Linux kernel (or any project that I'm aware of) suggests that it's either not suitable, doesn't catch enough issues to be worth the effort, or is just such a pain to deal with that no one has the patience to put up with it.

Rust gives you much (sure, not all) of what CBMC provides, in the compiler. That is a huge win, IMO.

Yet it's still not used in most C codebases. Where was it for Heartbleed?

We all understand that things can be done better. The problem is that they often are not.

Rust and other tools help by not giving people the choice to do sloppy work at all. History has proven many times that defaults matter.

That’s the crux of it. I think I can write safe C. So do many other people who, surely like me, cannot. I know I’ve written safe Rust because I had to fix my code until it was able to compile.
I think the issue with such formal analysis tools is that in order to make the analysis succeed you often end up writing code that doesn't look like normal C anymore and arguably at that point you're not writing C anymore.

---

As an aside, does CBMC check temporal memory safety and thread safety? It seems it checks for out-of-bound accesses, null pointer access and double-free, but I could not find a mention of use-after-free and data races.

CBMC is truly wonderful technology. Rust is still more powerful and easier to use at scale.

If your project has 100k lines of code, would you rather use CBMC or Rust? What about if your project has a million? 10 million? 100 million?

The fundamental issue is that SMT-based verification scales quite poorly. The optimal use of SMT-based verification is proving local properties of small chunks of code independently, and using the type system's encapsulation to scale up to global correctness.

> even more importantly by making programmers like the fact that they have to put pervasive annotations into their code

That’s the thing: many of us deeply dislike the annotations since they slow things down and look ugly.

Many developers dislike doing all sorts of useful things, like writing documentation in general, or tests.

Rust is an upfront investment of time which pays back in later stages when you're not spending 10 hours hunting down issues with valgrind or trying to explain codebase-specific lifetime semantics to junior developers. Yes the initial development is "slower", but you can then do massive refactors without fear of breaking some subtle rule that will blow things up, because such rule-breaking would result in a failure to compile.

I must write code very differently than most writing the actual code is not what takes my time. Sure you feel that time, but it ends up being a pretty small percentage of mine.
So? I feel like that backs up my point. Sure it may slow down development but it encodes useful information for reading and understanding the code later on, and preventing the most pernicious sorts of bugs (data races, UB, memory unsafety) from creeping in in the first place - and thus less need to spend time doing non-development tasks later.
Yes, I am agreeing. I am saying, a lot of these affordances that enter code faster or terser are not where I spend my time. I spend my time on the things you are talking about.
Rust is a continuous additional investment of time which will need to happen when writing the code, changing the code, doing the massive refactorings you mention and last but not least when reading the code.

There are benefits to that investment, but likely only in specific kinds of software (such as browsers, kernels, etc) which are relentlessly attacked and for specific kinds of programmers that like to specify things in much detail and don’t mind the associated costs.

The friction comes from pushing Rust as a generic solution for most programmers. See the Rust in the backend efforts for the perfect example.

> Proper tooling with C can prevent memory issues

So far this year, the Linux kernel has 849 overflow and memory corruption CVEs: https://www.cvedetails.com/product/47/Linux-Linux-Kernel.htm...

What is your advice to the Linux kernel developers who, by your logic, are not using proper tooling?

I’m not willing to tell them they’re doing it wrong because they’re experts in a very technically challenging field I don’t know much about. What is it you think they’re doing wrong, given your experience?

My favorite illustration of this is Xe’s blog: https://xeiaso.net/shitposts/no-way-to-prevent-this/CVE-2024...
For sure. That was adapted from https://theonion.com/no-way-to-prevent-this-says-only-nation... but still rings true. If only there were someone to avoid these huge arenas of error-prone, demonstrably impossible to write correctly conditions!

Rust, Zig, D, OCaml, Erlang, Haskell, programmers look at each other disappointedly

It is even worse than that, paraphrasing you,

ESPOL, NEWP, PL/I, PL/S, PL.8, Mesa, Modula-2, Ada, Object Pascal, programmers look at each other disappointedly

Even taking into consideration the systems programming languages until the 1980's, the list isn't exhaustive.

Oh sure! C/C++ are the only current major languages that spring to mind that make it so easy to shoot your foot clean off. Assembler, I guess, except it’s not nearly so widely used and everyone knows what they’re getting into with it.
> the Linux kernel has 849 overflow and memory corruption CVEs

4 years ago, they had 40 memory-corruption-related CVEs. Their actual vulnerability count hasn't increased by 2122% over 4 years – rather, they've become a CVE numbering authority and have started self-assigning CVEs even when there is no evidence of a vulnerability:

> Due to the layer at which the Linux kernel is in a system, almost any bug might be exploitable to compromise the security of the kernel, but the possibility of exploitation is often not evident when the bug is fixed. Because of this, the CVE assignment team is overly cautious and assigns CVE numbers to any bugfix that they identify. This explains the seemingly large number of CVEs that are issued by the Linux kernel team.

<https://lwn.net/ml/linux-kernel/2024021430-blanching-spotter...>

Having a huge number of self-issued CVEs is a policy decision on the part of the Kernel maintainers.

Your analogy would be right if talking english would make people statistically vert likely to randomly open doors or windows and shout at everyone in the street the code of their safe and crash their cars into the building while the klingon speakers would not exhibit that behavior.
Not the main argument, The only argument for Rust. The only single one argument for Rust.
There are actually quite a few nice features that go beyond memory safety:

- Type-safe generics (sorry C, `void*` doesn't count)

- Compiler errors that try to give you the root reason why your code isn't compiling and a solution to the problem

- Procedural macros to automatically generate boilerplate code for arbitrary structs/enums

- Solid package management tooling built into the workflow that doesn't rely on what Linux distro you're running

- consistent and easy-to-use macros for writing code meant to be used across OSs or architectures. I've written system-level packages compatible with Windows/Linux/MacOS/Every BSD flavor/Solaris with relative ease.

That their tooling (and community, starting from teaching examples) seems to rely on Github so much is worse than dealing with a diversity of distributions.
> If Rust was actually better, it would flow into the ground organically.

This is plainly untrue. People don’t like change even if change is for the better. I’m not saying Rust is better than C, only that you’re completely incorrect about better things winning.

> Proper tooling with C can prevent memory issues

Statements like these make me not take people seriously.

The state of the art in tooling for C to catch memory issues is miles ahead of what it used to be. But they won't catch everything, not by a long shot, especially when you often have to teach the tools how memory is managed in whatever library you're using that isn't just simple malloc()/free().

If the compiler for my language of choice can just do all this for me, and fail to compile my code when these issues might exist in my code, that's amazing. I don't want a language that lets me do broken things, and requires me to run an extra linter or tool or whatever to help me figure out what I've done wrong later.

Apparently 60 years isn't enough to learn how to use proper tooling.
- A Hundred more years ought to do the trick. Once we train AI to do it.

- What data will you feed it?

- Current C codebases and best practices.

- OOOF.

When AI gets advanced enough, compilers will be gone, there won't be a need for C anyway.

Just like Assembly got reduced to a very niche use case, after optimizing compilers got better than much humans at handling Assembly code manually.

Technically those will be AI/ML Compilers, but I digress.

It is anyway a win regarding C and its superset derivatives.

Maybe in the far, far future. The AI hallucinations, would turn optimizing compilers from "computer magic" to the "computer ultra nightmares" or maybe "My computer is having nightmares and I must scream".
That would give a new meaning to UB. :)
> If Rust was actually better, it would flow into the ground organically.

Yeah, and languages like Ada are widely used everywhere. And no one ever writes code in Cobol or JavaScript, since it was that bad.

I mean, when seatbelts were introduced, everyone[1] flocked to show how cool and safe they were. It's not like government had to mandate it from above, right?

[1] (it's a subtle joke only 2% bought Ford's cars with seatbelts) https://web.archive.org/web/20201130151600/https://www.histo...

> If Rust was actually better, it would flow into the ground organically.

Rust is flowing into the ground organically just fine, but in the kernel dev area people are actively stopping it. So the "organic" part is interfered with. Part of the time for good reasons, absolutely! But the rest of the cases do give off the "get off my lawn" vibe to me.

A better analogy than yours would be "the new people came with a proposal for a new way to list and enforce the rules in the D&D group" IMO. Something that will improve the session but might still give you a culture shock of the new way of doing things. Seems like a more accurate analogy.

> Being technically more memory safe doesn’t make people like it more.

Microsoft and Google have stated that from 60% to 75% of all CVEs are memory-safety-based. Is that not good enough of a reason to fight against memory un-safety?

Nobody is requiring you to "like" it -- more like "accept evidence backed with data by entities that work a lot with both C/C++ and Rust".

> In the end we don’t need any more of a reason to stop rust other than “we don’t like rust”.

That's not a merit-based or technical argument and such is not interesting. Why mention it?

> Proper tooling with C can prevent memory issues, taking away the main argument for rust.

Banning driving under the influence of alcohol can prevent numerous tragic car crashes. Worked beautifully, right?

>Proper tooling with C can prevent memory issues, taking away the main argument for rust.

The only way is to compile everything with -fsanitize=undefined,address,float-divide-by-zero,unsigned-integer-overflow -fsanitize-minimal-runtime. Anyone who uses less than this is reckless.

This is a very silly straw man.

RAII style is a syntax feature that doesn’t imply a specific semantics for how big the “resource” is, or how it’s allocated or freed. The only difference in Rust versus Zig is that Rust calls the de-allocator automatically and invisibly, and Zig requires allocation use an allocator parameter and an explicit call to free. Neither forces you to allocate batches at a time, nor do they prevent batching or use of arenas. To me it seems just as easy to allocate MyStruct[100] in either language.

Exactly - people seem to assume that RAII means having to heap allocate each object individually and then having to deallocate each object individually when done. Clearly that is not true and the allocation patterns are orthogonal to whether RAII is used.

RAII is also useful for non-memory related tasks like unlocking mutexes once the lock is no longer needed, which is just plainly useful in any language that allows functions to have multiple exit points.

I would prefer explicit linear types for locks, I find following Rust code using locks is very confusing - like an compiler-enforced call to `defer drop(lock)` to make it clear how it works. But I'm a noob at Rust.
A lot of concepts don’t directly mean you have to write bad code but they push you in that direction. Fact is, arena allocation or any other kind of custom allocation is pretty much non-existent in rust. Hopefully it will be implemented soon but I’m not sure if it is feasible because of lifetimes
How so? I've been using arena allocation with the bumpalo crate [1] with great success. I also wrote my own arena allocator for a WASM project, and also wrapped Nginx's memory pool via safe APIs.

I'm not going to claim that writing one correctly is easy... much like writing any low-level allocators, you have to deal with layout, drop order, unsafe cells, etc.

But using them? The lifetimes work great with them. And the allocator trait [2], once released, will make it so std collection types can be used as well.

[1] https://docs.rs/bumpalo

[2] https://doc.rust-lang.org/std/alloc/trait.Allocator.html

Ya this article is terrible.

You can use RAII with arena allocation, and you can also defer RAII cleanup to avoid stalls.

Manually calling cleanup functions also has the exact same potential for stalls.

Also perf wise, the only time huge destructor chains are generally triggered is during shutdown. In my app I run millions of destructors on shutdown and it takes about 1 millisecond, it is hardly a noticeable stall. Most of the time during shutdown is actually spent writing & compressing save/cache files, which is irrespective of RAII.

The confusing part of this article is the argument that the kernel maintainers are strongly against RAII. While I guess it’s possible Asahi Lina is unaware of this or is stubbornly including it anyway because she believes it’s such a large benefit, it seems more likely that it isn’t actually the dealbreaker.

In fairness, I’m hardly an expert on C++, Rust, or kernel development, so it’s possible that I’m missing something.

> it seems more likely that it isn’t actually the dealbreaker.

What is the dealbreaker then?

I’m not sure. One point I agree with the author is that it’s a complicated situation likely involving both social and technical factors.
I think C++ drags a lot of bad stuff in with the good, and it is reasonable to be skeptical of it.

For example, I think it was bordering on malpractice for lambdas to ship without anything resembling a borrow checker in place. With lambdas that borrow from the stack, introducing memory safety bugs is shockingly trivial.

Even if your codebase bans lambdas in C++, it's quite reasonable to not fully trust the teams which made the decision to ship them.

This is a "Structure of Scientific Revolutions" scenario: https://press.uchicago.edu/ucp/books/book/chicago/S/bo131797...

In any field, be it engineering or the sciences, accomplished and intelligent practitioners find themselves resisting new technology for reasons that are ultimately inscrutable and personal --- for example, fear of the unknown or fear of new technology devaluing their in the old. Because these practitioners are experienced and intelligent, they're able to construct elaborate plausible-sounding technical arguments against the new technologies. But since they're starting with the opposition and working backwards to a rationale, what these practitioners do when they argue against the new technology isn't so much science as apologetics.

Apologies can be frustrating because, superficially, they resemble earnest argumentation --- but because, in apologetics, the authors starts with a conclusion and works backwards to an argument, an apology contains insidious logical traps that are difficult to detect and disarm, especially when the new technology (as all new technologies do) contain genuine gaps and flaws.

It's because of this dynamic that many fields advance "one retirement at a time". We should all make a conscious effort, when evaluating new technology, to distinguish earnest technical criticisms from justifications for feeling averse to change --- and consciously suppress the latter.

Are you saying that RAII is "new technology"??
If you're a lifelong ANSI C programmer, RAII is new and scary.
RAII is all about making the compiler do what the "disciplined" C code is already doing manually. As mentioned elsewhere in this thread, a common C coding style and idiom is to goto to the end of your scope, where all the free calls go. The resulting behavior is pretty close to RAII.

The argument in favor of RAII is that it's a lot less error prone to let the compiler write that part for you.

A fine goal of course, but I feel like this post is overstating it or perhaps conflating it with something else (eg. The aside about trying to convince the GPU API maintainer that the issues they hit are also affecting drivers written in C may have little or nothing to do with rust or RAII)

The problem is that since cleanup with RAII is so 'simple', you tend to forget about it, and then you're suddenly in a situation where destroying a single object results in a cascade of thousands of other tiny destructions.

So now you actually need to think about a proper memory management strategy (even though you have automatic memory management) and will eventually arrive at arenas as the solution, and once you're there, suddenly RAII doesn't look so essential anymore (if you only have a handful of arena lifetimes to track instead of thousands of individual object lifetimes, manual memory management becomes trivial too).

E.g. just as with garbage collection, RAII is a solution to a problem that shouldn't exist in the first place (having to track the individual lifetimes of thousands of tiny individual objects with complex interdependencies).

When I switched from 'mainly C++' back to 'mainly C' about 8 years ago, I *did* think that I would miss RAII the most, turns out it was a non-issue (the main thing is to stop thinking about programs as individual objects interacting with each other).

I'm sorry, but I'm not really seeing it. How is that different from calling <struct_name>_free(<struct_name> *p)? That can trigger a cascade of thousands of other tiny destructions. There is no difference. The only argument I see against RAII is that a kernel would be better off with explicit drop() calls, but that is about it.

>E.g. just as with garbage collection, RAII is a solution to a problem that shouldn't exist in the first place (having to track the individual lifetimes of thousands of tiny individual objects with complex interdependencies).

Okay, but I sure hope you are aware that this is a fringe opinion shared by very few people. The people who need to track the individual lifetimes of thousands of tiny individual objects with complex interdependencies do in fact need that "power". E.g. the average webapp or webserver and C is a non-starter for anything that is internet connected. If I were to switch to C, the thing I would miss the most is the absence of antagonistic compilers.

> How is that different from calling <struct_name>_free(<struct_name> *p)? That can trigger a cascade of thousands of other tiny destructions.

You don't call 'struct_free(obj)' (there isn't even such a function), instead you call 'arena_reset(arena)' once. Stop thinking about individual object lifetimes, start thinking about arenas as 'lifetime buckets' :)

> I sure hope you are aware that this is a fringe opinion shared by very few people

It's really not a fringe opinion in areas of the industry where performance matters (like video game development).

> You don't call 'struct_free(obj)' (there isn't even such a function), instead you call 'arena_reset(arena)' once. Stop thinking about individual object lifetimes, start thinking about arenas as 'lifetime buckets' :)

Yes, so you can use RAII for the arena if needed. The problem is not RAII itself, is where it is applied.

And having a manual "arena_reset" function instead of a RAII-based "Arena" abstraction is objectively inferior as anyone could forget to call "arena_reset" while RAII would enforce the clean-up.

That depends heavily on what you're working with. If it's POD types then ok. But a lot of things fit into the destructor paradigm. Note it's not just memory. The R in RAII is for resource, not memory.
You can detect this case with profiling or other tools and then adjust it when appropriate.

Having a less error-prone solution for the common case and then doing something else when necessary is a great approach to software design so long as you don't run into the "something else" too often that you end up with two common idioms rather than one common idiom and some clear exceptions.

> RAII is a solution to a problem that shouldn't exist in the first place (having to track the individual lifetimes of thousands of tiny individual objects with complex interdependencies)

That's not what RAII is at all. You're projecting an extreme use case on the whole feature.

You can use RAII to clean-up the arena at once without forgetting to do so. To manage the lifetime of a socket or mutex. No one is forcing you to go full-OOP individual-allocation design when RAII is available, that's just terrible programming in general.

This isn't an inherent problem with RAII; it's a design/architecture problem. Not every object needs a destructor; if free() is a bottleneck, you have other serious problems that you need to address.

Also, arena-type approaches aren't free; you can run into serious safety problems with a large object graph, because the pointers _will_ escape containment and become stale and you won't realize it.

> personally hope that the Linux kernel never adopts any RAII, as I already have to waste way too much time for other slow software to load.

The only irrelevant example given is that of a poorly written app (Visual Studio) when discussing GPU driver. So is the GPU driver written with RAII slow? Is there a faster non-RAII version?

Theoretical situations are the best way to kill innovative ideas prematurely.
I'm more disappointed by someone who thinks that Rust macros are better than Zig comptime.

Rust macros are, IMO, one of the weakest parts of the language. Sure, you can do anything. To do so, you have to atomize down to lexical atoms, rearrange everything, and feed a new set of lexical atoms back into the stream. And to do so, you have to pull in a bunch of crates related to syntax analysis that should really be a fundamental part of the compiler and are welded to the specific Rust version. Bleargh!

Zig comptime has been a real breath of fresh air. It's great that it runs at compile time, and, if I need to debug it, I can generally force it to be runtime code. You write the same code you were going to write anyway without creating weird meta-languages that have a magic expansion phase that is impossible to debug.

I agree about Rust macros. Hopefully they'll upstream stuff like `syn` (and hyper-optimize it) in the future.

I appreciate the code generation abilities Rust macros give me but hell, they are difficult to wrap your head around even if you worked with them multiple times.

RAII is not unavoidable though.

The whole point of being against RAII is that forcing it has downsides you can not get away from.

but Rust has mem::forget, and it is not even unsafe (it is in std only currently, but I see no reason why the kernel can not have its own version)

So this to me seems a pointless argument vs:

* calling the equivalent of drop() manually every time, except in some cases

* calling mem::forget only when you need it and have auto drop() everywhere else

...Maybe mem::forget has not taken hold as a pattern? Still, these arguments do not look very technical , and much more religious to me

Yep, all of what you said PLUS the fact that you can just use arena allocations where you know you'll have to allocate hundreds / thousands of objects and then free them almost at the same time.

The blog article's author even alludes to this, which makes the post kind of funny. He debunked his own argument and yet continues on as if he did not.

re: std::mem::forget not being used much

On one hand, its just a shortcut to moving a value into the transparent ManuallyDrop struct and forgetting the result. In most cases, ManuallyDrop is the preferred way, as it allows encoding this behaviour (disabling the built-in drop) on a type level.

And if you want arenas or similar structures, they simply don't need to use either. The arena struct itself usually implements Drop to enact whatever it needs for deallocating the data its storing. The things stored in them also don't need ManuallyDrop since they're just written to memory directly using unsafe calls, and the compiler doesn't know this data exists (depending on implementation, some directly use syscalls, others use a Rust datastructure like Vec<u8> for allocating their memory). Nowhere in this is mem::forget or ManuallyDrop required, and the resulting API is usually far nicer than dealing with ManuallyDrop.

I am not sure what does this article achieve except cite other people's opinions on using or not using RAII (and a few other mechanisms).

It even admits that using arena allocator in Rust would address most (if not all) of their reservations towards RAII. And mentioning that there are badly written programs out there that take a while to shut down is, while informative in terms of an anecdote, is at the same time not at all a firm evidence the same will happen in the kernel.

I mean, it's the kernel, and the uttermost attention should be given to any potential performance or stability concerns -- of course!

But it does not seem like the people against it are willing to gather data. Which again puts the Rust devs on the backfoot. Seems like constant impeding akin to a bureaucratic structure that knows that you'll eventually get what you came for but they'll put every obstacle possible on your path first. :(

But overall, the article ended just when I expected it to get a bit more serious and data-oriented. Or at least give little more than opinions / feelings about the contention point.

Biased and unsubstantiated article, written by "VP of Community at the Zig Software Foundation".

Lack of RAII in Zig is one of the reason why I refuse to write any software in it -- it's such a backwards design decision that throws decades of programming language safety in the bin. `defer` can be forgotten and is not enforced at compile-time, it's a terrible solution. Also, `defer` can be implemented in terms of RAII if needed.

Yes, yes... arena allocation can be faster than deallocating every object one by one, Casey is a convincing/charismatic individual with extreme and biased opinions and an extremely narrow programming domain, can RAII can be misused and cause slowdowns.

Every programmer that understands RAII knows these things. Guess what -- when provably better, you can use RAII for the whole arena and not each individual component. Also -- data oriented design is not inherently incompatible with abstractions and RAII.

I'm tired of seeing all these misconceptions, when -- TBH -- it's mostly a skill issue.

Beyond just performance, RAII (a cryptically named concept BTW) introduces a lot of implicit, indirect behavior, making code much more complicated and harder to reason about. When does your constructor & destructor fire when a dynamic array grows? Did you use the copy-and-swap idiom properly? Did you define all the "rule of five" methods? It's bonkers.

I think a "defer" or "on scope exit" statement is just about ideal, gets you the same benefits but much more simply.

> When does your constructor & destructor fire when a dynamic array grows?

What constructor? And why would a Drop implementation be invoked to move something? You've moved it, there's nothing left to drop.

> Did you use the copy-and-swap idiom properly?

I don't believe I'm familiar with this idiom? From some quick googling this seems to be related to panic safety and ensuring that you don't leave things in an invalid state if a panic happens. It doesn't seem to be RAII-specific, and would also apply to a defer mechanism.

> Did you define all the "rule of five" methods?

Rule of five? You mean that your Clone implementation must properly handle resources in accordance with your Drop implementation, right? I'm not sure where the other three come in?

I'm speaking about C++, where blind faith in RAII is widespread, so I don't know if Rust's implementation alleviates some of the accidental complexity.

But I suspect any system with constructors / destructors leads to a lot of the same implicit behaviors, unintended complications, and leaky abstractions, and I generally favor explicit Initialize() and Cleanup() functions that you call manually, even in languages that support the implicit/magic stuff.

Rust doesn't have constructors*.

* Well, it kind of does, in so far there is a Ctor enum variant in the compiler implementation to represent the name of a type in the value namespace, but it is not overridable by end users and has no logic.

You were conflating details of C++'s object model and implementation of move semantics with RAII. Rust also makes heavy use of RAII, and the details of its object model and move semantics are not the same as C++'s.

The only thing that RAII requires is the concept of ownership, and a cleanup function that the compiler knows about. The one implicit thing here is the automatic insertion of calls to the cleanup function.

Rust does not have the rule of five, and it does not really have copy/move constructors.

defer does not get you the same benefits as a destructor because you can easily forget to write a defer, while destructors are always called.

I'm even less sure what to make of this article after developments over the past two weeks and especially this past weekend: It seems to be mostly Rust FUD wrapped in a screed against RAII, implying that somehow Rust will harm the kernel, maybe. But in the past two weeks...

16 Nov, Linux 6.13 Introducing New Rust File Abstractions, https://www.phoronix.com/news/Linux-6.13-Rust-File-Abstract

26 Nov, 3K Lines Of New Rust Infrastructure Code Head Into Linux 6.13, https://www.phoronix.com/news/Linux-6.13-Rust

30 Nov, Linux 6.13 Hits A "Tipping Point" With More Rust Drivers Expected Soon, https://www.phoronix.com/news/Linux-6.13-char-misc-More-Rust

That last one is particularly interesting, with the following quote from Greg K-H: "rust misc driver bindings and other rust changes to make misc drivers actually possible. I think this is the tipping point, expect to see way more rust drivers going forward now that these bindings are present. Next merge window hopefully we will have pci and platform drivers working, which will fully enable almost all driver subsystems to start accepting (or at least getting) rust drivers. This is the end result of a lot of work from a lot of people, congrats to all of them for getting this far, you've proved many of us wrong in the best way possible, working code :)"

> It seems to be mostly Rust FUD wrapped in a screed against RAII, implying that somehow Rust will harm the kernel, maybe

Very interesting observation, I think you might be right.