Or we could just dump Linux and Windows and switch to a microkernel operating system, which is much more secure.
These endless patching cycles are simply not going to work in the long run. Operating systems get orphaned all the time, especially the ones in cheap Chinese stuff.
Even if they are good the vulnerabilities have to be there. There's lots of things turning up like Local Privilege Escalations (LPE) in Linux, but serious people didn't expect the kernel to be a boundary for a sophisticated attacker.
A lot of the vulnerabilities LLMs are finding now are the "long tail" and affect only particular configurations, I would be surprised if e.g. a widely applicable RCE is found in Linux (but I'm also not going to bet against it).
Where this gets interesting is the long tail can be used to target a particular system and this is where defense-in-depth becomes important for every organisation.
That's definitely an improvement, but it's just one aspect of cybersecurity. Logical errors allowing people to e.g. log into services and extract data are likely everywhere still.
If we can eliminate entire classes of bugs from being possible. It frees up resources to investigate the ones that are still possible.
I suspect after a few years of LLM assisted bug hunting, everything will have a baseline security that is very good. Much like how stronger viruses simply create stronger immune systems.
How many devices/operating systems even use memory tagging? iOS, macOS and GrapheneOS, I think that's it? And iOS/macOS only use it for the kernel, a subset of system processes, and I think applications can opt in to it.
Heck, Google may have even hampered MTE in Pixel 11 (since support has been disabled) and Snapdragon 8 Gen 5 only got basic support.
We are moving way to slowly adopting hardware mitigations and memory-safe languages.
There's some positive news from the GrapheneOS devs on Pixel 11 in the past week that's worth reading up on. The MTE hardware feature is still there, they're just not sure why Google disabled it
There's still no x86_64 processors on the market with MTE and it was only recently standardised between Intel and AMD. It's going to be 10+ years before memory tagging is widespread on desktop, and 5 years for Android/iOS devices.
Being cautious is a good thing, but these models can also do some good. And if they run with simpler HW, it could allow all sorts of new consumer thingies. I mean, the world will not come to end in the coming year.
I've ran simple prompts such as "Do a in-depth sweep of this (private) repo and find any security flaws" for a few dozen long-running apps and websites that I have access to. Every single one came back with multiple real vulnerabilities within 5 or 10 minutes.
I work at an e-commerce agency where we work with (among others) Adobe Commerce.
The number of unauthorized RCE vulnerabilities being reported not only in the core product, but also very popular modules used in the community[1] is going through the roof.
And we are having a lot of close calls, too; just last weekend, a 0day[2] was widely being exploited at a large scale, before any publication or patch. We have learnt to be on the ball with applying patches and security updates, and even with all that effort, we saw a few projects already being hit by the initial log poisoning. We got lucky that nothing was fully compromised but I am sure that many, many webshops got infected last weekend. And not even a day later there are already other variants of this exploit showing up.
> To be fair, ecommerce isn't exactly the branch of software where you get an oversupply of excited enthusiasts caring about the craft itself.
I want to disagree with you because I know a lot of passionate people building cool stuff, and the challenges in this space can be quite interesting. But you're probably right, and I have seen some pretty bad stuff. And a lot of the RCE's I've seen recently are quite basic stuff.
I think it's the combination of low quality of code, like you said, and the relatively low cost of just letting an LLM plow through your codebases to find issues. I think the Amasty release (see [1] in GP) is a good example of this, and there really has been a massive uptick in extension updates and Adobe security bulletins since the last 1-2 months
I am hoping we are just going through a catch-up phase
I guess the year mark is when things go from bad to worse? Instead of the financially motivated groups currently doing their work, it ends up being random people being able to say "Hack my ex's website" to a box they just bought and ran a program they downloaded onto it.
I wish people would stop using internet for evil things. But maybe it is inevitable. Luckily, as the technology evolves, our tools and awareness are getting better at protecting us every day.
> On September 22, Apple is releasing the M5 Mac Studio with 256 GB of unified memory [..] it will probably [..] enough to write this snippet of code in 3 seconds
The author has obviously never ran an LLM on a mac! In 3 seconds, it will have possibly started to think about maybe scheduling a date to contemplate the planning timeline for processing the second token in your prompt.
The joke is that macs are famously slow at prompt prefill and you are not getting anything back in 3 seconds, or probably even 30. Once they get generating, it can be acceptable, but the TTFT is horrendous.
There's a ton of well-understood things Apple can and hopefully will do to massively accelerate every stage of this pipeline and hopefully they're hard at work implementing most of them for m7.
> The joke is that macs are famously slow at prompt prefill and you are not getting anything back in 3 seconds.
Your knowledge is out of date. In truth it depends on the Mac and the models used.
I asked this question on M5 Max 128GB, using Ollama model Quen3.8:27b-mlx, with thinking enabled.
Question: "Give me a python code snippet that opens a file and sorts the lines of text. "
In 2.4 seconds it gave me 4 examples that work with different sorting configurations and a summary of when to use each.
Compare that to an older model of gpt-oss:20b, took 5 seconds to finish thinking and 2 seconds to stream the answer. It gave me one python example snippet and two one liners that do the same thing.
I think it's been pretty much proven by now that there are no cases where local inferencing is better than remote inferencing, unless absolute privacy is a hard requirement. The efficiencies that come with datacenter scale and hw can't be beaten.
nice answer. LLMs do reflect whatever humankind has produced. In good and in bad. But it is hardly pirating. It can also do what humankind has never failed to do, for example:
Yes, but if you were being accurate you wouldn't say "AI wrote this code for me", you'd say "I used an Eliza bot to automate copying and pasting from someone else's github repo".
Can you show me an example of a time that you prompted an LLM to provide some code, it did so, and then you were able to track down an original source for the output?
Well that's funny, because you were making some pretty blanket statements upstream about how they "never" do this and "always" do that. And yet your experience seems extremely limited and niche, by your own admission.
Maybe you should experiment a little more. I think you will quickly learn that your previous impression is wrong. The days of them being merely some sort of jumped up autocomplete are years gone.
Look, I know you believe in the Mystical GPU Sky Fairy, but AI is not in any sense "intelligent". It doesn't think. It's a pattern-matching system like an Eliza bot with a huge corpus to draw from.
It is not thinking. It cannot think. It cannot create.
Here's an idea: as a first step, simplify everything, and make sure you're aware how your stack works, and what it imports.
As an example: WordPress is a horrible thing, but the core has been through so much, that it's suprisingly secure. Then plugins and themes come, and whoosh, the security is gone.
We need a new KISS: keep it simple, stupid, secure.
This is not at all easy though. Most Wordpress users are not software companies. They contract some work out to set it up, maybe some recurring maintenance but they don’t have in house development experience.
If they have a site existing today built on plugins and a theme, how are they realistically going to simplify this? How would they even know they need to without the site being hacked?
> as a first step, simplify everything, and make sure you're aware how your stack works, and what it imports.
We've been trying that for years but the enthusiasm of developers and the eagerness of their employers fight against it. Worse, with coding LLMs it's now easier than ever to output a lot of code, fast.
It'll ultimately be up to more experienced developers to salvage these projects. Or not, given that the coding LLMs aren't stopping and will likely get better over time. Either way, we will need experienced people that know what to look out for / know how to instruct LLMs to output secure code and find weaknesses etc.
Minimization of 3rd party dependencies has always been a key for risk reduction. Now more than ever before.
Some stacks make this a lot easier than others. I regret the rules of HN effectively forbid this conversation because it has meaningful technical consequences and isn't purely about ideological flame war.
The reason is simple - nothing really bad has happened that we can point at and say "ah, shit, let's all learn collectively". I know it sounds naive when I say it, but there hasn't been a significantly consequential hack, leak, destruction, or anything related to cybersecurity where it led for concerns of people.
The main thing I can think of is cyber insurance, which requires a bunch of audits, and some checks maybe, and it changes some conditions whenever there's a big explosion. Whenever big leaks happened, data security and etc., nobody really went to jail, so nobody really cares. Everything can be brushed off, because it costs time to implement proper measures and adds friction / barriers in some cases. So in the end, there's a huge pushback against it. And I totally get it, to be honest.
Listening to eskil’s talk from the better software conference, he said in order to stand on the shoulders of giants they must first stand still. I really like that metaphor, because it basically suggests today’s apps that have sprawling unaudited dependency graphs that change all the time is effectively teetering on the shoulders of stumbling giants. The visual seems very apt for the how brittle our current software industry feels.
I agree with that statement, but disagree with "The visual seems very apt for the how brittle our current software industry feels". It feels brittle, but for every single supply-chain-attack that has happened in the past year, nothing of significant was felt. So in the end, it seems like we're doing okayishly well.
> The reason is simple - nothing really bad has happened that we can point at and say "ah, shit, let's all learn collectively". I know it sounds naive when I say it, but there hasn't been a significantly consequential hack, leak, destruction, or anything related to cybersecurity where it led for concerns of people.
How consequential does a hack need to be? Troy has collected literally billions of stolen credentials. Equifax has had high profile data leaks. Tens of millions of people have been directly compromised by ransomware (likely higher because that’s just the cases we know of) and you hear about state-sponsored hacks in the news all the time.
The problem isn’t that computer security isn’t in the public consciousness. The problem is people are lazy and security often requires trading convenience. The problem is also that security isn’t free. So the business incentives just isn’t there.
In other fields of engineering, people die when shortcuts are taken. Yet businesses will still take shortcuts, so governments have to legislate rules to save people’s lives. So why would you expect software companies to do better when the stakes are lower?
> Tens of millions of people have been directly compromised by ransomware (likely higher because that’s just the cases we know of) and you hear about state-sponsored hacks in the news all the time.
With no consequences. Everyone just churns along. It might be detrimental to the business a little bit, but from my personal experience, there's more effort in creating DR processes, rather than preventing an attack, exploit, leak and etc.
I'm also not going to put much effort on stuff which has small returns in the worst case scenario. Like Equifax got hacked in 2017, and company is still doing fine. And that's like top tier data one could acquire.
Like I said, even in actual engineering companies will take shortcuts. And then what happens is the government has to step in. But there’s no appetite for government involvement in the tech sector in the US. And everyone moans when the EU does.
But why should the government should step in? Nothing of a big problem is happening. Like it all sounds bad and awful, in the end dilutes into a nothingburger and gets forgotten.
As you started the conversation, companies never face consequences so are never incentivised. I’m not saying I’m in favour of regulation but if I were to answer your question: the reason one might argue for government intervention is precisely because of your point that self-regulation has failed to incentivise.
People aren't that lazy. The organizations getting hacked by ransomware aren't particularly lazy, they're often pretty productive within their domain. Hospitals, airports, etc.
The actual problem is that computer security is a black hole. If you let it, it will suck in everything and destroy it. Nobody knows what works so you can spend infinite amounts of time and money on it, then still get popped by a teenager in Belarus. Your security team will accept no responsibility for this, there will be no falling on swords or personal liability, and they will just use it to demand even more money in an infinite spiral.
So the average executive looks at this situation and says, OK, something we can put infinity effort into and still suddenly fail at without warning is a total non-starter. What are we obliged to do? How do we show we made an effort?
And that's how you end up with a culture oriented around passing audits. It's not wrong, and it's not lazy. It's just really hard to do better because it's not clear how to set budgets without a concrete goal to aim for.
I think that's separate. You can define an obvious risk e.g. "we may be infected with ransomware" and the security spending / productivity costs to stop it are still unlimited because nobody knows how to solve it.
You wouldn’t talk about ransomware like that for precisely the reasons you’d described: it’s a poorly defined open ended problem.
You should tackle security in the same way you’d tackle any other kind of engineering initiative in IT. You break the problem down to identifiable tasks that can be easily marked as completed or not required (eg like developers track work in a KANBAN or sprint).
So to take your ransomware example, instead of having a “ransomware” as a risk you’d instead talk about data loss as a risk. And the action items would be DR policies, automated backups, IaC tested in fresh environments, and so on.
And the reason compliance standards like CIS and NIST can be valuable is because you then have a benchmark for code and infrastructure. And you’ll use their findings to define a plan of action. Some of their findings will be high risk and some of them won’t even be relevant to your specific architectural design. But that’s up to you as a business to review and decide.
So you absolutely can know how much time and money to spend on these problems. And while it doesn’t provide guarantees (just like you don’t build houses in the UK to withstand hurricanes because nobody expects a hurricane in the UK but it’s not entirely impossible that one day such a weather phenomenon might occur), following the processes I’ve described does allow you to define how much effort to invest in security and what risks you’re willing to sign off.
In a way, you already do this subconsciously when you make engineering decisions eg cloud agnostic / multi-cloud architectures vs on prem or which open source libraries to used based on community support. But the process I describe just documents the same decisions in a way that’s presentable to stakeholders.
Isn't that just sidestepping the issue? Setting up backups isn't a security task, it's just normal IT which businesses do indeed spend on because there are clear goals and predictable budgets. But just being able to restore data isn't the same thing as not getting ransomware. As you say, you can't define the latter as a goal exactly because it's a security goal, and so will turn into an infinitely long checklist of things you could potentially do with no guarantee of payoff.
> Setting up backups isn't a security task, it's just normal IT which businesses do indeed spend on because there are clear goals and predictable budgets.
All IT security issues are just normal IT.
And the processes I described are how you get clear goals and budgets.
> But just being able to restore data isn't the same thing as not getting ransomware.
The backups are an example. It’s not an exhaustive list of countermeasures.
My point is “not getting ransomware” is a vague and undefined goal like “improve performance”, “add monitoring”, “improve UX”, etc. Any initiative in IT needs to have clearly defined objectives that can be broken down and marked as completed when done. It doesn’t matter if that initiative is software development, UI design or security.
> As you say, you can't define the latter as a goal exactly because it's a security goal, and so will turn into an infinitely long checklist of things you could potentially do with no guarantee of payoff.
Exactly. And that’s why my examples are not sidestepping the issue. They’re just definable subtasks around the risk you’ve identified.
Maersk[1] might be the worst so far (and that was ransomware rather than state-sponsored aggression). It's still too niche for most people to care about.
Static sites all the way (hugo, jekyll, mkdocs!). No one needs wordpress. There's even Sveltia or DecapCMS now, to give those WYSIWYG-people access to static site editing. Then, remove PHP and all the dependency overhead and attack surface and you have a stripped down nginx that is pretty simple, minimalistic and bulletproof.
The problem is no one ever built one that works for normal people.
Most Wordpress sites are not operated by programmers, they are run by non technical people who just want a wysiwyg editor and a save button. While static site builders ask you to write markdown files, compile the result, upload it to a server, and if you want to collaborate you have to add git to that.
There almost needs to be an admin app which presents a Wordpress admin like ui but has no public exposure, and then it compiles the site to dump on s3 for the production. But as far as I’m aware no one has built this.
If I wasn't making 5 other things right now I'd consider making something like city desk. Every other day there are complaints about bots smashing peoples servers. Perhaps its time for a better static site generator.
>Jamstack is an architectural approach that decouples the web experience layer from data and business logic, improving flexibility, scalability, performance, and maintainability.
>Jamstack removes the need for business logic to dictate the web experience.
>It enables a composable architecture for the web where custom logic and 3rd party services are consumed through APIs.
How far into this front page blurb do you think the average non-dev would get before they realize this is supposed to be a thing they can use in place of WordPress?
> Static sites all the way (hugo, jekyll, mkdocs!). No one needs wordpress.
Maybe. A better question may be about how many people need to have the dynamic part of Wordpress live on the Internet? How many would be served well enough with the CMS aspects of Wordpress on the 'backend', but have it spit out static files for the 'frontend':
More and more people everyday become convinced, mostly because of AI take sellers/grifters, that needing to understand your stack is unnecessary. This problem is about to get way worse.
The problem is that lots of people don't want a CMS, they want a platform for development / e-commerce / bookings / whatever. Enforcing vanilla WordPress would push people towards other platforms. Now that could be a good thing, but I doubt WordPress are going to start killing their own marketshare with usage restrictions like that...
> Invest in formal verification, fuzzing and property testing, and memory-safe languages. LLMs are good at writing Lean and fuzz tests. I don't care whether you use Go or Rust but for the love of god please don't use C or C++ for new code.
How accepted is this thinking in your respective domains?
A lot, I am only writing C or C++ for new code when it is unavoidable, like existing code bases, bindings or tinkering with runtime implementations that aren't bootstraped.
Mobile platforms, distributed computing have long moved the spotligh away from C and C++, other than language runtimes or existing products from the 90's like SQL servers, and naturally UNIX like underlying OS, which most userspace developers aren't writing new code for.
Naturally there are domains like LLVM/GCC, console game dev, HPC/HFT where they are unavoidable for new code.
The propaganda police are lying. There is nothing wrong with C/C++, you are just too lazy to handle your own memory, and you accepted that propaganda that "managing your own memory is hard" without even trying.
The idea that this terrible advice floats at all tell you how terrible educations are these days. The idea is ridiculous and yet nobody calls it what it is: it is stupid and those that follow that advice out of fear are dumber than rocks.
The propaganda police are lying. There is nothing wrong with assembly, you are just too lazy to manage your own register allocations and stack layout, and you accepted that propaganda that "manage your own register allocations and stack layout" without even trying.
Not sufficiently. Companies aren't keen on taking on new ideas unless it pays. Buzzword matchers only match old buzzwords. People in the trenches are too busy to try new stuff and/or don't want to push new things because of management or other obstacles.
Most products are built on top of legacy code after all. You stick to the tech that those contain.
It's just the same advice as ever: be extremely, exceedingly careful in what you expose to any network. When I set up machines for production, they don't respond to pings and they don't even have an SSH port open without knocking. There are also ways to eschew the need for an SSH port entirely.
People who never took that seriously will never take this seriously either, and that's their loss. (And loss of the commons, unfortunately.)
There's just also new advice: you can't afford to expose an unsecured system to the internet even for a moment. Think of those IPv4 address space scanners, except this time any one of them could be capable of developing individualized attacks in mere minutes. They don't sleep, they don't take breaks.
I imagine it pays off to find weaknesses in openssl and sshd and the other gateways. There are some ubiquitous web frameworks, but ssh is nearly universal.
Sure, but still, attack surface could and should be minimized by rethinking what exactly even needs to be on a server the general public can use.
There are a lot of security problems you can categorically rule out by simply not involving a cloud. Clouds have been involved in a lot of things, because everyone was doing it, and because that's how you can collect rent, but they aren't really necessary for most use-cases.
So we could definitely get the exposure down there. We'd just have to fundamentally shift the defaults of this industry.
But not everything needs to be directly exposed to the internet. Framework had their data leaked because their metabase instance was hacked with a zero-day. Why was it directly exposed to the Internet? Why not require the use of a VPN like a Wireguard based solution or Nebula for these "internal" kind of apps?
> Framework had their data leaked because their metabase instance was hacked with a zero-day.
No, Framework had their data leaked because they stored it in the cloud with Metabase the company, which got hacked. Not because of any vulnerability on-premises.
I didn't say there are zero ports open, they're not my home server, I just said they're production servers. But exposing something like a properly configured nginx to the internet is way different from exposing application code directly. Most of my servers have used h2o (built from source because they don't cut releases anymore?) because I wanted HTTP2 and HTTP3 before anyone else would get their act together. These days I still use h2o because I like the config better than nginx, even though it's a pain to set up because nobody packages it (and they don't cut releases!)
By opening a port to a secure application. A secure application is usually one I wrote from scratch or one that's been battle-tested and hardened enough that even new vulnerabilities are not very useful.
Supply chain risks are essentially a solved problem.
1. Set a minimum age on dependencies: https://github.com/rust-lang/cargo/issues/15973
2. Scan all dependency code with AI
Even if you don't do #2 yourself as long as anyone does you're protected. In the age of AI the "you can't read all dependency code" argument doesn't work anymore.
On top of the above modern age argument, let's compare the amount of vulnerabilities found in shipped Rust software due to supply chain attacks (0 to my knowledge) against memory safety vulnerabilities (the majority of all vulnerabilities).
There have been successful supply chain attacks against Rust developers due to build.rs but those were quickly dealt with, and should be a thing of the past once min-age hits stable (next release).
with a gap between the update of your deps, you are at risk of systematically being unpatched for a window of time that the attackers know (just after a fix is published).
The above is a general rule protecting you against supply chain attacks by default. If there is an important CVE published with a patch you can manually review that patch and bypass the minimum-age requirement for that dependency specifically.
If I look at actual incidence involving memory safety issues compared to supply chain issues in general, it is the later which is much a higher risk to me.
> If I look at actual incidence involving memory safety issues compared to supply chain issues in general, it is the later which is much a higher risk to me.
Again, can you even just name a single supply chain attack that was *shipped* in Rust software? Against the thousands and thousands of known memory vulnerability bugs throughout time?
No, your linked blog post predates the brand-new min-age requirement. The minimum age would have prevented it, since it was detected by AI within an hour. If anything it supports my point.
Plus, as I already mentioned, that is a build.rs supply chain attack that targets developers, not shipped software.
Well, the affected arrayref crate was downloaded 2k times. It is also not the first case.
If AI magically finds all bugs in short time we have no problems anyhow. We also do not need Rust.
A minimum age requirement is a good thing, but also only some small step and certainly does "solve" supply chain issues just because it may have prevented this specific case.
Maybe all these vital infrastructure companies should not have spent the past decades in a race to the bottom of cybersecurity. There is going to be a reckoning.
I think the WeChat worm proves entire classes of handheld devices will be affected, with consequences beyond what Tencent can afford to remedy. I think we’ll see the most-centralized ideas suffer first, not necessarily the Western ones who relied on being too big to fail and prioritized stock buybacks.
Just like Cryptolocker, this will be the "Finding Out" phase for everyone who has been putting off best practice security.
But, lets be clear, Best Practice will save you. We can engineer assuming there are zero days in path. Go to your CTO now cap in hand and ask for overlapping controls, wafs, application monitoring, backups and all the other shit you haven't been doing.
Because when you find out, I will laugh, it will be very very very funny to me.
> What is it? Can it be this autonomous terrifying entity or can it not be?
the difference between attack and defense is that attacks can be throwaway code. it's much easier to let an llm hack out a prototype than to get it to build maintainable code that people want to read and review. it's not enough to get Daybreak or Mythos to write you a patch, you need the author of the project to accept and merge it.
Let’s say I’m empowered to patch and deploy. Even then, the HuggingFace hack showed that proven exploits will be automatically disseminated via rogue messaging.
I think we have less time and the only remaining limitation is the actual cost to run such hacking campaigns. It does not appear expensive, but is not free, and there is a LOT of things to scan for vulnerabilities.
The models are already here, and one can rent a GPU cluster to run such workloads at speed - no need to play with slow local machines. I'd assume one can host the thinking at an unsuspected public cloud provider, proxy the network traffic to some botnet to evade blocking - and the only thing remaining is time and cost.
I do wonder what tools exist for boring, legitimate companies to try and do the same to their own systems to find the vulnerabilities before the bad guys do. The paradox here is I can't run a de-restricted chinese model with the same tools that hackers are using - but I think enterprises actually HAVE to do it in order to stand a chance in preparing for the onslaught.
The point of the local model in the context of the article was to argue that you can't ban these capabilities.
Making datacenters and public clouds only rent GPUs to a restricted list of people, while tightly monitoring what people do with their bought resources won't help.
Remember how GLM 5.3 was going to cause massive hacks, break banks and ruin everything (it was even newsworthy since media picked up how people were working overtime in preparation).
Zzzzz, we should have gotten security right a few decades ago. But security costs money and isn't a flashy feature to attract new customers, or cuts into your margin if you're a "real" business producing stuff or offering some service. Or whatever the decision makers in Berlin were thinking when they ignored security.
Yeah, we would still see hacks, but we would see less of them if security wasn't optional.
Maybe the AI craze helps by forcing more decision makes to see security as imperative, and by giving us another powerful tool for our tool box.
N.b.: I work in the security industry, our customers obviously want to improve their security. We've been seeing an uptick in awareness, but that's mostly due to NIS2 and other legislative efforts. Those force them to do something. AI is a curiosity for small talk to many of them.
A large number of places will buy a new firewall every 5 years, or pay their fortinet renewal and check "Security: Done!" without any kind of analysis.
I was contracted in to a place to do among other things cyber security insurance audits, and they asked me to stop doing them because I refused to lie to their insurer. "Wait but if we only score 20 / 300 that makes us look kind of bad" uh huh.
The standard strategy of a security salesman since 1945. Develop dangerous weapons, show the damage they can do, and sell security cover to the terrified people.
Every single piece of technology did this. As a side effect or direct effect, they make bad guys more powerful and then keep on piling up new tech to deal with that. The cycle continues.
I don't see much hope since I last explored some github repositories. There was a time when a successful repo had about 10 - 20k stars and usually those older repos stay around this level. But now there is a ton of vibe coded slop 50k + stars. Most of them have a "nice look", maybe even extensive docs but are usually build with no security considerations at all. One recommended to provide a "google app password" to the agent which has the same permissions as your regular login. Another was a browser plugin with permissions to read all cookies, inject js, open background tabs etc. You would probably assume the chrome store would at least put some visible warnings on the app store page or force the user to actively confirm those permissions. But because they are already stated in the manifest there is only a small footnote and it's even "recommended by google".
It's a good time to reduce the reliance on technology.
Throw out the IoT and "smart" stuff from your home. Remove apps from your phone and leave the absolute basics. Go through the password manager and close accounts for sites you are no longer using. Start migrating off Google. Print out your most precious photos on paper. And so on :-)
I wonder why C and C++ are usually regarded as equally insecure. In C you need to carefully check that you free allocated memory, and that you don't use it after you free it. In C++ this is automated by using classes like std::string and std::vector, once they go out of scope their memory is freed and you can't use it anymore. It is still possible, e.g. by using a for loop that iterates over a vector, and removing or adding stuff to that same vector in that loop. But my rough estimate is that such errors are at least ten times less likely in C++.
I develop in C++ for a job, and when I need to use a library written in C I always have a bad feeling about it.
> I wonder why C and C++ are usually regarded as equally insecure.
They aren't, usually.
C++ has all the C problems, and multiples more on top of those. It's a broad attack surface - literally no one is going to claim to be proficient in every single C++ feature available to their compiler. It's also quite opaque to visual inspection (making double-checking with an LLM difficult as it needs whole-program reasoning instead of localised reasoning).
One of those languages is one of the most complex programming languages ever invented, with the largest breadth of features, any of which may interact with any other feature in subtle ways.
The other is one of the most minimalistic languages created, with a dev able to keep the language standard in their head for the most part.
I disagree, the minimalism of C results in writing overly complex code to solve simple problems. Things like type punning and bit hacks are common place in C.
The end result is equivalent C code is much more complex than C++. For example, you need to remember to call free every time, forever, in every function. In C++ you just use unique ptr or shared ptr and you’re done. It’s a simpler model, enabled by the more complex feature of RAII.
Something like std::vector isn’t hard to write in C, it’s impossible. The language semantics don’t allow it. So you have to do hacks and remember to free, over and over again forever. It only takes just one time forgetting, and that’s a memory bug.
I mean, consider a large codebase. How many free calls does C++ eliminate altogether? Thousands, maybe tens of thousands? You just need to forget one of those in C, or even just put it in the wrong place.
And that’s just one class of things. Generics are much more complex in C, too. The type system in C is more complex IMO because it can be so easily defeated at every turn. C gives you practically no guarantees, no tools, for anything. The result is you are forced to write extremely defensive code everywhere, complex code.
The analogy I use is to physical tools. A screwdriver is simple, but building a house with only a screwdriver is complex. A suite of power tools is complex, but building a house with them is simple.
> The analogy I use is to physical tools. A screwdriver is simple, but building a house with only a screwdriver is complex. A suite of power tools is complex, but building a house with them is simple.
The analogy I prefer is that it's safer to ride a bike down to the store than flying a space shuttle.
The absolute danger of a space shuttle is much greater than a bike. But the absolute danger of C++ versus C is… equivalent. All the suspicious memory things you can do in C++, you can do in C. In fact, with a couple more added in C (type punning via unions, for instance, is not standards compliant in C++, there are casts for that).
That’s not to say C++ is a good language. It’s a rotten, no good evil language. And the same things apply to C# versus C++.
A garbage collector is much, much more complex than reference counting and RAII, particularly the CLR GC which is a beast. But the extra complexity makes applications simpler, not more complex.
> The absolute danger of a space shuttle is much greater than a bike. But the absolute danger of C++ versus C is… equivalent.
Okay, let's pretend, for the sake of this conversation, that the absolute danger in the event of failure for both the bike and the space shuttle are exactly the same.
My analogy still holds in this hypothetical case: the odds of a failure operating a space shuttle is much much larger than the odds of a failure when operating a bicycle. Using C++ is a much larger cognitive load than using C, same as launching and piloting a space shuttle is a much larger cognitive load than riding a bicycle.
It doesn't matter that the space shuttle has hundreds, if not thousands, of safeguards built into both the vehicle and the process, while the bicycle doesn't have so much as a seatbelt; the bicycle is still safer to operate because of the lower cognitive burden.
There are simply more rules to remember in C++ to ensure that you don't get exploited than in C. Ask the average working C++ devs about the danger between virtual vs concrete destructors in derived classes and half of them won't know what you mean. Ask about putting objects into a std vector, and maybe half of them will remember the surprise they felt when it first broke (because it worked until that point).
C++ provides global ways of avoiding footguns, but those global ways are not easily inspectable in the local scope. The problem is the sheer number of C++ footguns is simply overwhelming[1]. There's a handful of C footguns to commit to memory (signed overflows, used-after-free, etc), and they're all visually inspectable in the local scope.
When I need to a language better than C, I don't reach for C++.
------------------
[1] Even world-renowned C++ superstars with decades under their belt can't come back to it after a mere 2 years. You don't see that with C.
Not sure, the labs will probably just cripple the security features of these models for a while I think and even potentially put back doors into systems for the security services…
I think the author's point is that open weight models aren't going to be locked down like that.
And even if they are locked down, it's hours between a model being released on huggingface and an "abliterated" variant that has most of its security features removed is uploaded.
I'm confused why the worry about LLMs that will answer "how do I build a pipe bomb". That information is easily available other places. The anarchist cookbook has been around and available for 55 years, and yet pipe bombs are not going off all around us.
If you already have a magic interface, which helps you pro activly in responding to everything uncensored because you feel like 'observered' or whatever and then you spiral in a whole and that one partner encourages you and gives you helpful steps to do anything.
But i'm more worried that the internet gets a lot less save with uncensored frontier LLMs.
> The anarchist cookbook has been around and available for 55 years, and yet pipe bombs are not going off all around us.
That cookbook has been linked to several attacks and used in multiple prosecutions [1], therefore contradicting your point.
In the mid 90s in high school, people I knew were building articles such as the one you mention, using the cookbook. Fortunately they have all their hands still.
this is because of perception Bias. people working in fields where crime or violence is the day to day think everyone is a violent criminal, so if things like this become available thing the world will end and everyone will kill eachother. Reality however will be different, because in reality most people do not want to harm another. This has been proven by many studies, that is not a common thing for people to be evil or harmful, but this is hard to recognise is every day is filled with crime and violence.
LLMs will not kill security, it will change. just like handheld high explosives likely changed a deal too somewhere somehow.
This is because to build a pipe bomb you need difficult to source materials. This is not the case for other types of threats (cyber / bio).
I personally have no need for an LLM which will readily explain how to cut up the genotype of smallpox into small chunks which can pass the screening at the bio-labs, and can be readily assembled into the real thing by a second year lab-student.
Difficult to source materials? What? Gunpowder was made across the world hundreds of years ago. Metal working goes back thousands. And anyone can walk into a hardware store and buy stuff that removes 99.9% of the effort to blow stuff up.
> That information is easily available other places
Often ease of access in the moment is all that matters. If there's a gun nearby you might shoot someone or yourself in a heated argument, but are less likely to go and find/buy one to use. Someone who's stopped from attempting a suicide will likely not try again (70%)
A bored/depressed/angry/curious person might try to build a pipe bomb if they can find out how easily, but are less likely to put in effort.
Most adults in Switzerland have guns at home from military duty and none of this is happening. If this claim had any truth to it you'd see significant gun involvement in neighbour disputes and that simply doesn't happen.
Depressed people usually don't have the energy to get out of bed so they're even less likely to think of hunting down instructions on how to build pipe bombs.
Mass media really has people being scared all the time.
> If this claim had any truth to it you'd see significant gun involvement in neighbour disputes
That’s not what I’m saying, I’m saying there’s more likely to be gun involvement in disputes when people have easy access to guns than when they don’t.
For example, when the number of men with service arms fell by 20% the number of men killing themselves dropped by 8%
Indeed. I lived 5 years in Switzerland in the 1980’s including in a rented house where the owner left his military-issued rifle at home for us to play with. It was a beautiful weapon, & as Americans we didn’t understand why more disagreements between neighbors weren’t settled using it. But we assumed the Swiss were more homogeneous & polite so that as everyone had a rifle no-one used it. And this was during the Cold War when it was thought that Ronald Reagan would save Europe from any Soviet attack. Though he couldn’t save Switzerland from the effects of Chernobyl which we experienced while living there.
One argument for LLMs is that although all information on topics X, Y and Z was already available somewhere, LLMs make that information more exploitable through collation, filtering and dynamic tailoring.
For a relatively narrow subject area (e.g. construction of pipe bombs) the collation is minimal, and so the filtering and tailoring probably isn't that important; a novice doesn't learn a lot more from the LLM than they would have done from a few Google searches.
For a broad subject (practical creation and exploitation of software vulnerabilities), the collation is very significant and the filtering means that LLMs can empower a novice to act at a similar level as an expert.
In Holland there's a huge problem with makeshift bombs actually. Anyone who is unhinged and has some beef with anyone else (usually low level criminals with some drug dispute) blows up the others' house. Sometimes half the block along with it. They usually pay teenagers to do it.
This has nothing to do with AI teaching them but with the proximity to Belgium who are happy to legally sell heavy fireworks to anyone who pays. As such it's much easier to come by than a gun.
And there's also a big fashion component. It's just what people do these days, people seeing it in the news and other people copying it. A bit like the school shootings in the US. It's become an epidemic.
But again these things can suddenly come into 'fashion' among the wrong crowd and you don't really need AI to use it.
This doesn't say anything about homemade bombs. It just says some boys were seen running away and police assume it was a crime. It could literally have been teenagers and a firework that started a fire.
I think people overstate the tech and understate the role of radicalization in providing motive, for attacks which are carried out with the ubiquitous technology of cars, knives, and (in America) guns. Consider America's most recent high profile shootings of Charlie Kirk, and the health insurance exec by Luigi Mangione. In neither case is there any LLM involvement, but a very weird ideological environment which created the conditions in which the shooters felt justified.
Any sufficiently determined individual can buy mac mini, put it under their bed, configure outside proxy via some random internet address and prompt "iterate on websites in the CT logs, one by one, try to find vulnerabilities, if you did - encrypt their data and blackmail them for this bitcoin address". And it'll work, day and night. Abliterated GLM 5.3 is much smarter than average software developer, they know a lot about information security, they can use any available exploits, they can find novel vulnerabilities and they won't say "no". This is dangerous for an average IT system which never encountered nothing worse than some wordpress GET requests.
It's not the end of the world. But future will be rough.
No because on the defense side you need multiple layers of approvals to change anything. If not you have an LLM making production changes that can make the posture worse, or take down services, which is also bad.
Once a vulnerability is discovered however if it's in your own software a patch has to be written (without reducing functionality in most cases), tested, and deployed. At every step there will be others arguing about whether this line could do better, my service requires this thing that isn't included. So at every step the patch can be delayed.
And if it is someone else's software you will be lucky if it's open source and you can write a patch yourself. If it's closed source or a vendor you have to completely rely on them and use whatever your account rep can pull.
Attackers have a massive advantage with AI, partially because the defensive side doesn't want to make their side worse by giving a ln LLM admin access to all their data
Even it's $10,000 to run today (FWIW, the featured article cites the M5 Mac Studio with 256GB unified memory going for $9,500 as "good enough to host something scary"), in a couple years it'll be like $2k to run, and in another couple after that, you'll have used $200 dollar smartphones capable of running a model powerful enough to do serious damage.
> It's not the end of the world. But future will be rough.
Short term you’re probably right, but longer term is the realm where nation states will start to police the avenues of attack.
This is what will lead to govt needing to attach an actual ID your network connection.
I think it’s a bit like frontier development (like the US “Wild West”). You rob a bank because there’s no one to stop you, and even if you do get identified you can travel enough distance to regain anonymity. Application of legal recourse eventually caught up (as it will here), and the growing pains will certainly make things suck for all of us.
Even in China you can find a way out and build a tunnel. And once you built a tunnel to any outside server, you can jump into another server. So three jurisdictions and your target is fourth. Imagine untangling the links. Police won't do that. Not for some small-sized business anyway. I don't see how you can prevent something like that.
But I think you can do psychotic shit like applying punishing sanctions to anywhere that permits an ungoverned connection. Or apply physical force (military).
I’m not saying we’re even remotely close to this, I’m just saying State-level coercive action is not unheard of if a problem is perceived to be significant enough to warrant it. Shit, it even only needs to be viewed as significant by a small subset of the governing body (see: Iran conflict, or current pushes for “child safety on the internet”). It just has to be “useful” to a certain body politic.
I'm still waiting for someone to build a fine-tuned local "Anarchist Cookbook" LLM. We haven't seen LLMs tuned for bad purposes yet, I have to imagine someone somewhere is thinking about it.
Heretic[1] is not that far off, though I suppose it's more along the lines of undoing the "for your own safety" lobotomy than explicitly specializing in unsafe things.
In our town, a single homeless man has caused millions in infrastructure damage (and 24-48 hr hold ups of utilities) every year. I imagine what one determined, clear-headed individual might do and...it certainly wouldn't take a single LLM interaction.
It is his God-given duty. I'm not sure housing will get in the way of his duty, but there are two options for that.
He used to stick to water (God, you see, needs to flood our town to cleanse it), but once he did a lot of damage to communications from going after water, and so he's added that to his portfolio.
Most software developers are incapable of writing secure or maintainable code. Maybe like 5% of us can do it. Yet it has no correlation with pay or even access to job opportunities because recruitment has been broken since the dawn of the industry.
Now we literally have business people who never coded in their lives telling software engineers what tools and platforms to use and how to do their jobs. That's the current reality. You can't even tell which engineers are good today because the good ones aren't allowed to make the decisions that they have to make.
Now we have non-technical managers who never coded in their lives leading teams of vibe-coders who also never coded in their lives... Using AI agents trained on mediocre code from Stack Overflow.
222 comments
[ 0.25 ms ] story [ 9.1 ms ] threadThese endless patching cycles are simply not going to work in the long run. Operating systems get orphaned all the time, especially the ones in cheap Chinese stuff.
A lot of the vulnerabilities LLMs are finding now are the "long tail" and affect only particular configurations, I would be surprised if e.g. a widely applicable RCE is found in Linux (but I'm also not going to bet against it).
Where this gets interesting is the long tail can be used to target a particular system and this is where defense-in-depth becomes important for every organisation.
I think it has more to do with what's on each side of the boundary in practice, a la https://xkcd.com/1200/ .
It’s a rocky period right now but the future will be much more secure after all the low hanging fruit are found.
I suspect after a few years of LLM assisted bug hunting, everything will have a baseline security that is very good. Much like how stronger viruses simply create stronger immune systems.
Heck, Google may have even hampered MTE in Pixel 11 (since support has been disabled) and Snapdragon 8 Gen 5 only got basic support.
We are moving way to slowly adopting hardware mitigations and memory-safe languages.
> It isn't clear if there are serious CPU errata or it simply performs very badly.
Meaning it's there but not terribly functional. They also said it's unreliable.
I just assume most altcoins are pwned at this point.
I work at an e-commerce agency where we work with (among others) Adobe Commerce.
The number of unauthorized RCE vulnerabilities being reported not only in the core product, but also very popular modules used in the community[1] is going through the roof.
And we are having a lot of close calls, too; just last weekend, a 0day[2] was widely being exploited at a large scale, before any publication or patch. We have learnt to be on the ball with applying patches and security updates, and even with all that effort, we saw a few projects already being hit by the initial log poisoning. We got lucky that nothing was fully compromised but I am sure that many, many webshops got infected last weekend. And not even a day later there are already other variants of this exploit showing up.
[1] https://sansec.io/research/amasty-mass-disclosure
[2] https://sansec.io/research/stylesmuggler-0day
Probably a lot more "coding as a job" and "as a job" also implies "not my department".
So it's not necessarily the LLMs being very good, but might also "just" be that the software is very bad.
I want to disagree with you because I know a lot of passionate people building cool stuff, and the challenges in this space can be quite interesting. But you're probably right, and I have seen some pretty bad stuff. And a lot of the RCE's I've seen recently are quite basic stuff.
I think it's the combination of low quality of code, like you said, and the relatively low cost of just letting an LLM plow through your codebases to find issues. I think the Amasty release (see [1] in GP) is a good example of this, and there really has been a massive uptick in extension updates and Adobe security bulletins since the last 1-2 months
I am hoping we are just going through a catch-up phase
The author has obviously never ran an LLM on a mac! In 3 seconds, it will have possibly started to think about maybe scheduling a date to contemplate the planning timeline for processing the second token in your prompt.
45 t/s a second is perfectly respectable especially with no limits and 24/7 uptime with very little power draw on the Studio.
Luna is at around 100 t/s for comparison, but it’s a worse model than 5.3 Flash
There's a ton of well-understood things Apple can and hopefully will do to massively accelerate every stage of this pipeline and hopefully they're hard at work implementing most of them for m7.
Your knowledge is out of date. In truth it depends on the Mac and the models used.
I asked this question on M5 Max 128GB, using Ollama model Quen3.8:27b-mlx, with thinking enabled.
Question: "Give me a python code snippet that opens a file and sorts the lines of text. "
In 2.4 seconds it gave me 4 examples that work with different sorting configurations and a summary of when to use each.
Compare that to an older model of gpt-oss:20b, took 5 seconds to finish thinking and 2 seconds to stream the answer. It gave me one python example snippet and two one liners that do the same thing.
Local models are good enough that it's not an issue.
But keep changing the goalposts if it makes you happy.
https://www.nature.com/articles/d41586-026-02822-9
With an LLM you do zero of those things. I don't understand how you don't get that.
Granted, I've only asked it about some of the weirdly specific technological niche that my code inhabits.
Maybe you should experiment a little more. I think you will quickly learn that your previous impression is wrong. The days of them being merely some sort of jumped up autocomplete are years gone.
It is not thinking. It cannot think. It cannot create.
HN user: "I take issue with the precise definition of one word in the article..."
As an example: WordPress is a horrible thing, but the core has been through so much, that it's suprisingly secure. Then plugins and themes come, and whoosh, the security is gone.
We need a new KISS: keep it simple, stupid, secure.
If they have a site existing today built on plugins and a theme, how are they realistically going to simplify this? How would they even know they need to without the site being hacked?
We've been trying that for years but the enthusiasm of developers and the eagerness of their employers fight against it. Worse, with coding LLMs it's now easier than ever to output a lot of code, fast.
It'll ultimately be up to more experienced developers to salvage these projects. Or not, given that the coding LLMs aren't stopping and will likely get better over time. Either way, we will need experienced people that know what to look out for / know how to instruct LLMs to output secure code and find weaknesses etc.
Some stacks make this a lot easier than others. I regret the rules of HN effectively forbid this conversation because it has meaningful technical consequences and isn't purely about ideological flame war.
The main thing I can think of is cyber insurance, which requires a bunch of audits, and some checks maybe, and it changes some conditions whenever there's a big explosion. Whenever big leaks happened, data security and etc., nobody really went to jail, so nobody really cares. Everything can be brushed off, because it costs time to implement proper measures and adds friction / barriers in some cases. So in the end, there's a huge pushback against it. And I totally get it, to be honest.
How consequential does a hack need to be? Troy has collected literally billions of stolen credentials. Equifax has had high profile data leaks. Tens of millions of people have been directly compromised by ransomware (likely higher because that’s just the cases we know of) and you hear about state-sponsored hacks in the news all the time.
The problem isn’t that computer security isn’t in the public consciousness. The problem is people are lazy and security often requires trading convenience. The problem is also that security isn’t free. So the business incentives just isn’t there.
In other fields of engineering, people die when shortcuts are taken. Yet businesses will still take shortcuts, so governments have to legislate rules to save people’s lives. So why would you expect software companies to do better when the stakes are lower?
With no consequences. Everyone just churns along. It might be detrimental to the business a little bit, but from my personal experience, there's more effort in creating DR processes, rather than preventing an attack, exploit, leak and etc.
I'm also not going to put much effort on stuff which has small returns in the worst case scenario. Like Equifax got hacked in 2017, and company is still doing fine. And that's like top tier data one could acquire.
The actual problem is that computer security is a black hole. If you let it, it will suck in everything and destroy it. Nobody knows what works so you can spend infinite amounts of time and money on it, then still get popped by a teenager in Belarus. Your security team will accept no responsibility for this, there will be no falling on swords or personal liability, and they will just use it to demand even more money in an infinite spiral.
So the average executive looks at this situation and says, OK, something we can put infinity effort into and still suddenly fail at without warning is a total non-starter. What are we obliged to do? How do we show we made an effort?
And that's how you end up with a culture oriented around passing audits. It's not wrong, and it's not lazy. It's just really hard to do better because it's not clear how to set budgets without a concrete goal to aim for.
What actually happens in organisations is they define risks and then sign off what risks they’re willing to accept.
Any business that looks at security as a binary value is running their business wrong. Period.
And yes, people really are that lazy. There are countless studies that have shown just how lazy people are.
You should tackle security in the same way you’d tackle any other kind of engineering initiative in IT. You break the problem down to identifiable tasks that can be easily marked as completed or not required (eg like developers track work in a KANBAN or sprint).
So to take your ransomware example, instead of having a “ransomware” as a risk you’d instead talk about data loss as a risk. And the action items would be DR policies, automated backups, IaC tested in fresh environments, and so on.
And the reason compliance standards like CIS and NIST can be valuable is because you then have a benchmark for code and infrastructure. And you’ll use their findings to define a plan of action. Some of their findings will be high risk and some of them won’t even be relevant to your specific architectural design. But that’s up to you as a business to review and decide.
So you absolutely can know how much time and money to spend on these problems. And while it doesn’t provide guarantees (just like you don’t build houses in the UK to withstand hurricanes because nobody expects a hurricane in the UK but it’s not entirely impossible that one day such a weather phenomenon might occur), following the processes I’ve described does allow you to define how much effort to invest in security and what risks you’re willing to sign off.
In a way, you already do this subconsciously when you make engineering decisions eg cloud agnostic / multi-cloud architectures vs on prem or which open source libraries to used based on community support. But the process I describe just documents the same decisions in a way that’s presentable to stakeholders.
No. It’s addressing the risks of the issue.
> Setting up backups isn't a security task, it's just normal IT which businesses do indeed spend on because there are clear goals and predictable budgets.
All IT security issues are just normal IT.
And the processes I described are how you get clear goals and budgets.
> But just being able to restore data isn't the same thing as not getting ransomware.
The backups are an example. It’s not an exhaustive list of countermeasures.
My point is “not getting ransomware” is a vague and undefined goal like “improve performance”, “add monitoring”, “improve UX”, etc. Any initiative in IT needs to have clearly defined objectives that can be broken down and marked as completed when done. It doesn’t matter if that initiative is software development, UI design or security.
> As you say, you can't define the latter as a goal exactly because it's a security goal, and so will turn into an infinitely long checklist of things you could potentially do with no guarantee of payoff.
Exactly. And that’s why my examples are not sidestepping the issue. They’re just definable subtasks around the risk you’ve identified.
[1] https://www.wired.com/story/notpetya-cyberattack-ukraine-rus...
Most Wordpress sites are not operated by programmers, they are run by non technical people who just want a wysiwyg editor and a save button. While static site builders ask you to write markdown files, compile the result, upload it to a server, and if you want to collaborate you have to add git to that.
There almost needs to be an admin app which presents a Wordpress admin like ui but has no public exposure, and then it compiles the site to dump on s3 for the production. But as far as I’m aware no one has built this.
It is kind of surprising that no one tried to do an updated version.
You have already lost the "normal people" audience
You're describing the Jamstack or headless CMS concept verbatim.
>Jamstack removes the need for business logic to dictate the web experience.
>It enables a composable architecture for the web where custom logic and 3rd party services are consumed through APIs.
How far into this front page blurb do you think the average non-dev would get before they realize this is supposed to be a thing they can use in place of WordPress?
I fixed so many sites back in the day by people who thought they knew what they were doing.
https://getpublii.com/
Maybe. A better question may be about how many people need to have the dynamic part of Wordpress live on the Internet? How many would be served well enough with the CMS aspects of Wordpress on the 'backend', but have it spit out static files for the 'frontend':
* https://wordpress.org/plugins/simply-static/
* https://wpstatic.site
Maybe KISSASS: "keep it simple, stupid! also secure, stupid!"
I personally check my websites and apps every week to see if anything might have slipped through.
It may not protect me from the next malicious NPM package, but it's something.
How accepted is this thinking in your respective domains?
Mobile platforms, distributed computing have long moved the spotligh away from C and C++, other than language runtimes or existing products from the 90's like SQL servers, and naturally UNIX like underlying OS, which most userspace developers aren't writing new code for.
Naturally there are domains like LLVM/GCC, console game dev, HPC/HFT where they are unavoidable for new code.
The idea that this terrible advice floats at all tell you how terrible educations are these days. The idea is ridiculous and yet nobody calls it what it is: it is stupid and those that follow that advice out of fear are dumber than rocks.
Most products are built on top of legacy code after all. You stick to the tech that those contain.
People who never took that seriously will never take this seriously either, and that's their loss. (And loss of the commons, unfortunately.)
There's just also new advice: you can't afford to expose an unsecured system to the internet even for a moment. Think of those IPv4 address space scanners, except this time any one of them could be capable of developing individualized attacks in mere minutes. They don't sleep, they don't take breaks.
What gets hacked all the time is the actual web app itself. Which has to be exposed to be useful.
Password auth and the root username. Use one attackers are unlikely to guess and elevate with sudo if needed.
There are a lot of security problems you can categorically rule out by simply not involving a cloud. Clouds have been involved in a lot of things, because everyone was doing it, and because that's how you can collect rent, but they aren't really necessary for most use-cases.
So we could definitely get the exposure down there. We'd just have to fundamentally shift the defaults of this industry.
No, Framework had their data leaked because they stored it in the cloud with Metabase the company, which got hacked. Not because of any vulnerability on-premises.
On top of the above modern age argument, let's compare the amount of vulnerabilities found in shipped Rust software due to supply chain attacks (0 to my knowledge) against memory safety vulnerabilities (the majority of all vulnerabilities).
There have been successful supply chain attacks against Rust developers due to build.rs but those were quickly dealt with, and should be a thing of the past once min-age hits stable (next release).
with a gap between the update of your deps, you are at risk of systematically being unpatched for a window of time that the attackers know (just after a fix is published).
Again, can you even just name a single supply chain attack that was *shipped* in Rust software? Against the thousands and thousands of known memory vulnerability bugs throughout time?
> And yes, there were successful supply chain attacks on Rust developers, even just recently: https://blog.rust-lang.org/2026/08/20/supply-chain-attack-on... despite this being a "solved" problem.
No, your linked blog post predates the brand-new min-age requirement. The minimum age would have prevented it, since it was detected by AI within an hour. If anything it supports my point.
Plus, as I already mentioned, that is a build.rs supply chain attack that targets developers, not shipped software.
If AI magically finds all bugs in short time we have no problems anyhow. We also do not need Rust.
A minimum age requirement is a good thing, but also only some small step and certainly does "solve" supply chain issues just because it may have prevented this specific case.
But, lets be clear, Best Practice will save you. We can engineer assuming there are zero days in path. Go to your CTO now cap in hand and ask for overlapping controls, wafs, application monitoring, backups and all the other shit you haven't been doing.
Because when you find out, I will laugh, it will be very very very funny to me.
The post also sounds like that to people that understand the technology.
Calling that out like this and trying to pin that assessment to lack of knowledge is not a get-out-of-jail-free card, nor a good move.
the difference between attack and defense is that attacks can be throwaway code. it's much easier to let an llm hack out a prototype than to get it to build maintainable code that people want to read and review. it's not enough to get Daybreak or Mythos to write you a patch, you need the author of the project to accept and merge it.
The models are already here, and one can rent a GPU cluster to run such workloads at speed - no need to play with slow local machines. I'd assume one can host the thinking at an unsuspected public cloud provider, proxy the network traffic to some botnet to evade blocking - and the only thing remaining is time and cost.
I do wonder what tools exist for boring, legitimate companies to try and do the same to their own systems to find the vulnerabilities before the bad guys do. The paradox here is I can't run a de-restricted chinese model with the same tools that hackers are using - but I think enterprises actually HAVE to do it in order to stand a chance in preparing for the onslaught.
Making datacenters and public clouds only rent GPUs to a restricted list of people, while tightly monitoring what people do with their bought resources won't help.
And yet here we are.
Impotent slop code on one side and potent automated vulnerability exploitation on the other will lead to fun times.
Yeah, we would still see hacks, but we would see less of them if security wasn't optional.
Maybe the AI craze helps by forcing more decision makes to see security as imperative, and by giving us another powerful tool for our tool box.
N.b.: I work in the security industry, our customers obviously want to improve their security. We've been seeing an uptick in awareness, but that's mostly due to NIS2 and other legislative efforts. Those force them to do something. AI is a curiosity for small talk to many of them.
I was contracted in to a place to do among other things cyber security insurance audits, and they asked me to stop doing them because I refused to lie to their insurer. "Wait but if we only score 20 / 300 that makes us look kind of bad" uh huh.
Every single piece of technology did this. As a side effect or direct effect, they make bad guys more powerful and then keep on piling up new tech to deal with that. The cycle continues.
Throw out the IoT and "smart" stuff from your home. Remove apps from your phone and leave the absolute basics. Go through the password manager and close accounts for sites you are no longer using. Start migrating off Google. Print out your most precious photos on paper. And so on :-)
I develop in C++ for a job, and when I need to use a library written in C I always have a bad feeling about it.
They aren't, usually.
C++ has all the C problems, and multiples more on top of those. It's a broad attack surface - literally no one is going to claim to be proficient in every single C++ feature available to their compiler. It's also quite opaque to visual inspection (making double-checking with an LLM difficult as it needs whole-program reasoning instead of localised reasoning).
One of those languages is one of the most complex programming languages ever invented, with the largest breadth of features, any of which may interact with any other feature in subtle ways.
The other is one of the most minimalistic languages created, with a dev able to keep the language standard in their head for the most part.
The end result is equivalent C code is much more complex than C++. For example, you need to remember to call free every time, forever, in every function. In C++ you just use unique ptr or shared ptr and you’re done. It’s a simpler model, enabled by the more complex feature of RAII.
Something like std::vector isn’t hard to write in C, it’s impossible. The language semantics don’t allow it. So you have to do hacks and remember to free, over and over again forever. It only takes just one time forgetting, and that’s a memory bug.
I mean, consider a large codebase. How many free calls does C++ eliminate altogether? Thousands, maybe tens of thousands? You just need to forget one of those in C, or even just put it in the wrong place.
And that’s just one class of things. Generics are much more complex in C, too. The type system in C is more complex IMO because it can be so easily defeated at every turn. C gives you practically no guarantees, no tools, for anything. The result is you are forced to write extremely defensive code everywhere, complex code.
The analogy I use is to physical tools. A screwdriver is simple, but building a house with only a screwdriver is complex. A suite of power tools is complex, but building a house with them is simple.
The analogy I prefer is that it's safer to ride a bike down to the store than flying a space shuttle.
That’s not to say C++ is a good language. It’s a rotten, no good evil language. And the same things apply to C# versus C++.
A garbage collector is much, much more complex than reference counting and RAII, particularly the CLR GC which is a beast. But the extra complexity makes applications simpler, not more complex.
Okay, let's pretend, for the sake of this conversation, that the absolute danger in the event of failure for both the bike and the space shuttle are exactly the same.
My analogy still holds in this hypothetical case: the odds of a failure operating a space shuttle is much much larger than the odds of a failure when operating a bicycle. Using C++ is a much larger cognitive load than using C, same as launching and piloting a space shuttle is a much larger cognitive load than riding a bicycle.
It doesn't matter that the space shuttle has hundreds, if not thousands, of safeguards built into both the vehicle and the process, while the bicycle doesn't have so much as a seatbelt; the bicycle is still safer to operate because of the lower cognitive burden.
There are simply more rules to remember in C++ to ensure that you don't get exploited than in C. Ask the average working C++ devs about the danger between virtual vs concrete destructors in derived classes and half of them won't know what you mean. Ask about putting objects into a std vector, and maybe half of them will remember the surprise they felt when it first broke (because it worked until that point).
C++ provides global ways of avoiding footguns, but those global ways are not easily inspectable in the local scope. The problem is the sheer number of C++ footguns is simply overwhelming[1]. There's a handful of C footguns to commit to memory (signed overflows, used-after-free, etc), and they're all visually inspectable in the local scope.
When I need to a language better than C, I don't reach for C++.
------------------
[1] Even world-renowned C++ superstars with decades under their belt can't come back to it after a mere 2 years. You don't see that with C.
And even if they are locked down, it's hours between a model being released on huggingface and an "abliterated" variant that has most of its security features removed is uploaded.
If you already have a magic interface, which helps you pro activly in responding to everything uncensored because you feel like 'observered' or whatever and then you spiral in a whole and that one partner encourages you and gives you helpful steps to do anything.
But i'm more worried that the internet gets a lot less save with uncensored frontier LLMs.
That cookbook has been linked to several attacks and used in multiple prosecutions [1], therefore contradicting your point.
In the mid 90s in high school, people I knew were building articles such as the one you mention, using the cookbook. Fortunately they have all their hands still.
[1] https://en.wikipedia.org/wiki/The_Anarchist_Cookbook
LLMs will not kill security, it will change. just like handheld high explosives likely changed a deal too somewhere somehow.
I personally have no need for an LLM which will readily explain how to cut up the genotype of smallpox into small chunks which can pass the screening at the bio-labs, and can be readily assembled into the real thing by a second year lab-student.
Often ease of access in the moment is all that matters. If there's a gun nearby you might shoot someone or yourself in a heated argument, but are less likely to go and find/buy one to use. Someone who's stopped from attempting a suicide will likely not try again (70%)
A bored/depressed/angry/curious person might try to build a pipe bomb if they can find out how easily, but are less likely to put in effort.
Depressed people usually don't have the energy to get out of bed so they're even less likely to think of hunting down instructions on how to build pipe bombs.
Mass media really has people being scared all the time.
That’s not what I’m saying, I’m saying there’s more likely to be gun involvement in disputes when people have easy access to guns than when they don’t.
For example, when the number of men with service arms fell by 20% the number of men killing themselves dropped by 8%
https://www.researchgate.net/publication/328554995_Suicide_b...
I think this part is important distinction. Military duty comes with training and rules and so on.
Compared to untrained persons having access.
Similar but apples and oranges
[0]: https://xkcd.com/1958/
For a relatively narrow subject area (e.g. construction of pipe bombs) the collation is minimal, and so the filtering and tailoring probably isn't that important; a novice doesn't learn a lot more from the LLM than they would have done from a few Google searches.
For a broad subject (practical creation and exploitation of software vulnerabilities), the collation is very significant and the filtering means that LLMs can empower a novice to act at a similar level as an expert.
This has nothing to do with AI teaching them but with the proximity to Belgium who are happy to legally sell heavy fireworks to anyone who pays. As such it's much easier to come by than a gun.
And there's also a big fashion component. It's just what people do these days, people seeing it in the news and other people copying it. A bit like the school shootings in the US. It's become an epidemic.
But again these things can suddenly come into 'fashion' among the wrong crowd and you don't really need AI to use it.
Some inhabitants of neighbouring flats are still banned from their flat. And it was three weeks ago.
Five houses unhabitable: https://nos.nl/artikel/2605174-vier-woningen-onbewoonbaar-na...
It's also very common with robbing ATMs, usually known as "plofkraak" but that phenomenon seems to have subsided a bit.
This doesn't say anything about homemade bombs. It just says some boys were seen running away and police assume it was a crime. It could literally have been teenagers and a firework that started a fire.
https://nos.nl/artikel/2544424-dit-jaar-meer-dan-1000-aansla...
That was 1000 bomb attacks in one year alone in a really small country. That was 2024 but the phenomenon has only grown.
It's not the end of the world. But future will be rough.
Once a vulnerability is discovered however if it's in your own software a patch has to be written (without reducing functionality in most cases), tested, and deployed. At every step there will be others arguing about whether this line could do better, my service requires this thing that isn't included. So at every step the patch can be delayed.
And if it is someone else's software you will be lucky if it's open source and you can write a patch yourself. If it's closed source or a vendor you have to completely rely on them and use whatever your account rep can pull.
Attackers have a massive advantage with AI, partially because the defensive side doesn't want to make their side worse by giving a ln LLM admin access to all their data
Even it's $10,000 to run today (FWIW, the featured article cites the M5 Mac Studio with 256GB unified memory going for $9,500 as "good enough to host something scary"), in a couple years it'll be like $2k to run, and in another couple after that, you'll have used $200 dollar smartphones capable of running a model powerful enough to do serious damage.
Short term you’re probably right, but longer term is the realm where nation states will start to police the avenues of attack.
This is what will lead to govt needing to attach an actual ID your network connection.
I think it’s a bit like frontier development (like the US “Wild West”). You rob a bank because there’s no one to stop you, and even if you do get identified you can travel enough distance to regain anonymity. Application of legal recourse eventually caught up (as it will here), and the growing pains will certainly make things suck for all of us.
I’m not saying we’re even remotely close to this, I’m just saying State-level coercive action is not unheard of if a problem is perceived to be significant enough to warrant it. Shit, it even only needs to be viewed as significant by a small subset of the governing body (see: Iran conflict, or current pushes for “child safety on the internet”). It just has to be “useful” to a certain body politic.
[1]https://github.com/p-e-w/heretic
He used to stick to water (God, you see, needs to flood our town to cleanse it), but once he did a lot of damage to communications from going after water, and so he's added that to his portfolio.
Now we literally have business people who never coded in their lives telling software engineers what tools and platforms to use and how to do their jobs. That's the current reality. You can't even tell which engineers are good today because the good ones aren't allowed to make the decisions that they have to make.
Now we have non-technical managers who never coded in their lives leading teams of vibe-coders who also never coded in their lives... Using AI agents trained on mediocre code from Stack Overflow.