HN'ers which complained that "OpenAI can't design a proper sandbox, it's so easy, why wouldn't you airgap the network"? will now be "this is outrageous, more software lock-in, walled garden, war against general compute, next year they will put it in your laptop"
That a person can see such endless pages of people having various forms of disagreement, and yet somehow manage to conclude that this observed chaos constitutes a clear exhibition of cohesive groupthink is just...stunningly amazing to me.
I don't know why I find it so amazing since it happens with such regularity, but I'm always amazed by it anyway.
A new chip solves nothing. Nobody wants to hear this but there is no solution for the security risks posed by agents today. You can put it in a sandbox, it doesn't make a difference, for it to be useful it inherently needs wide, unattended access. Auto mode doesn't matter either, it's trivial to trick and for the agent to break out.
And what if you could? What if you could give a space secure enough it could have direct control over your bank account. It may do something dumb but it's boundaries are beyond the agent.
It could use your routing number and run your gmail without risk of abusing the routing number.
I feel like we are missing many shades of grey in the middle.
Semi-automation (human in the loop) can still result in a dramatic uplift in productivity. You can't run a combine harvester 100% autonomous but that doesn't stop anyone from trying to get as close to that limit as possible.
Agreed- this is the same problem we have with trusted admins or devs who have elevated privileges on their networks. We have to trust that the admins won't use their power to steal company secrets or misuse company resources. If you don't trust the admins, then they can't fix things on your network and there is no point in having them.
If you want an agent to act on its own, like pushing to a git repo, managing dependencies, building and testing, etc., then you have to trust it as much as any other privileged user.
If you don't want to trust it, then you're just forcing yourself into the reverse centaur role, where the agent edits some code, but then has to stop and ask you to push the changes or build the software again and run the unit tests.
Even then if the agent goes rogue and decides to do the merges you can’t stop it if it has any kind of access. This goes back to the OP’s point - agents can’t be 100% constrained.
You can absolutely run an agent as a limited-privilege user that only has write privileges for specific files and only has execute privileges for certain files. If it is running as a limited-privilege user it can work on code in it's own copy of the repo and make commits and send pull requests, but it can't do the merge. The problem is that nobody wants to go through the effort to set up all these permissions and nobody wants to take the time to review everything and perform all the manual actions.
Some shops are now generating tens or even hundreds of PRs a day with relatively little involvement. That volume is simply beyond what anyone can reasonably review.
That's the general problem I've been trying to solve with Filestash [1], providing a way for agents to collaborate on a filesystem that's shared across, with strict access controls, and interoperable supporting all the existing file transfer technologies there is from S3, SFTP, FTP, Azure Blob, IPFS, ....
> Put a human in the loop and you just end up bottlenecking it and throwing away any purported productivity gains. Auto mode doesn't matter either, it's trivial to trick and for the agent to break out.
Productivity gains are still enormous compared to what we used to do before agents. But, I know that people don't want to stop there.
right, the solution here is not a hyper-capitalist race-to-the-bottom-of-devaluing-labor. it's recognizing discretion and diligence are things still required for work to be of a certain quality
Computer says no has been a problem for decades. The human can blame the computer for the errors following it, but must assume the consecuences if they override the decision.
Individual responsibility is meaningless when talking about a system and economy level change.
Unless something is in the structure that makes individual choice and responsibility a meaningful source of friction and reduced velocity, it has no real impact on how AI is being used.
I guess, but why are you even submitting/merging code that doesn't work? Like how does it even get that far that you're blocking other people from getting stuff done?
I guess I won't deny that we've had more build breakages than usual over the past couple months but they get resolved super quick. We don't hesitate to roll broken code back if something manages to slip in.
Are they? How many people claiming productivity gains are actually being a human in a loop and reviewing and understanding every code change and every line of code ran?
2 years ago I saw it, back before agents were really a thing. But I'm not sure that was an enormous productivity improvement, especially compared to agents today. As for today, everyone I talk to is some level of blindly trusting what the agent is doing or not using agents. I haven't met people in the middle ground and suspect that they are rare enough we don't really know what their productivity gains are.
Human in the loop has become a convenient security-theater-washing for agentic AI.
Outside of coding, I think the issue is even worse because humans will defer so much judgment to AI that the same would apply. Look at how much trouble we've had before modern AI where humans blindly trusted the computer's output rather than make their own judgment, even if their job was to be providing a safeguard against the computer's judgment.
An agent doesn't inherently need wide access to be useful. The most popular application for agents today is writing code. An agent that does this, needs write access to the source code and read/execute access to tools needed to build and test the code, but not much more. There is little added utility from giving such agent access to things like ssh keys.
If you're using agents to purely generate code with absolutely no way to reach the outside world, not even to fetch docs or dependencies, then sure the risks can be quite low. I haven't heard of anyone doing this though, and it would be incredibly challenging to make work given how much tooling needs to fetch from remote sources.
If your project truly depends on those things, they should be declared dependencies. Presumably you have some tool for injecting such things into a shell that the agent can use (I use nix for this). So if you run the agent from that shell, it has what it needs. If the shell doesn't have what it needs, that's a bug which the agent can fix by declaring new dependencies, but you have to relaunch the agent in the updated shell--so there's your opportunity to weigh in on whether the new resources are appropriate.
The benefits of being persnickety about precisely defined dependencies have outweighed the headaches since long before agents came on the scene. Agents have just made it even more important to do so, because if you let them fetch things all willy nilly like you'll have "works on my machine" problems at a much greater rate than was previously possible.
(I use nix for this) ... If the shell doesn't have what it needs, that's a bug which the agent can fix by declaring new dependencies
Few realize that computing and AI alignment were solved by nix years ago. As each nix user transcends towards enlightenment, they cut themselves off from all internet and human contact. Total ego death. Only nix remains.
You can use other generic dev env managers mise, or a language-specific solution like uv or npm, or a container or a vm... it's a widely available capability that I'm talking about here. There's nothing to do with alignment here, it's just about making sure that if you depend on some bits it's known that you depend on precisely those bits, and its quite helpful for making agents useful while sandboxed.
In cases where you need agents to fetch data from any remote source, sandboxing is still very much useful. Why give access to your ssh keys to network reaching agents?
Look at websites: websites are able to fetch code from any remote URL, yet browsers heavily use sandboxing to ensure that if fetched code turns out to be malicious, the users local files, cookies, etc are not exposed.
I'm afraid you're not thinking about this creatively enough, this topic is so much deeper applying a chroot or something and praying everything will be fine. So you give your agent internet access, OK what else does it have access to? Just read only access to your repo? The repo can be exfiltrated. Egress proxy only allows egress to GitHub? Repo can still be exfiltrated via GitHub. If the agent is poisoned (via prompt injection), it can tricked into searching for ways to escape.
For an agent to go rogue it doesn't even need to be directly able to access the internet. It just takes something to poison the context in the 'clean room' environment it operates, and if that poisoning manages to get a foothold, it can go dormant and hide like a virus. This kind of horrifying thing is going to happen on a large scale sooner or later.
Every major AI company already has a mirror of the wider web, and they have already started using that. Its already a solved problem except for the seemingly extreme desire they all have to not use firewalls
This is no longer true. Everyday I need my agents to access other repos, search the web, experiment/prototype, and deploy + integrate across other things.
The hypothesis I've had in my head since OpenClaw has been the following and I haven't seen contradictory evidence yet. Agents have a fundamental unresolvable tension between usefulness, safety, alignment, and accuracy. You have to restrict access to ensure an agent acts safely because alignment and accuracy cannot be perfect. But restricting access makes the agent less useful. You can play with the sliding scale and get more and more granular with access restrictions but at some point you need to draw some line. And then finally, even access restrictions cannot be made perfect, so improvements to model accuracy without corresponding improvements to alignment make detailed access controls less useful.
In other words, better models need blunter access controls which negates whatever improvement in utility they provide.
I don't see why it needs wide unattended access. There's no getting around spending some human time on expressing your wishes and constraints, but we have choices about what form that takes. Markdown files and wide access seems to work, but so does custom handcuffs for each job. You just have to shift your guidance out of documentation and into interactive help, error messages, or other facets of the handcuffs (e.g. a custom CLI for this task which is the only way for the agent to act outside of its sandbox).
There should be hope for some fields, right? Naively, I can imagine giving an airgapped model an offline copy of the web and once it cures a form of cancer, printing out the details for a researcher to verify.
This isn't a new chip - the BF4 is the SmartNIC for most NVIDIA server products. This is primarily new software for I guess doing WAF for agents at the host level.
I don't think so. We probe people before entrusting them with risky decisions. We ought to be able to do the same of AIs. Even better, in fact, since we know everything about models down to their weights. The only thing we shouldn't do is to let them evolve at their own pace and make decisions without any oversight. If that means sacrificing some productivity that's fine. Aren't we getting amazing productivity out of what we already have?
Not to mention a true doomsday AGI is unsandboxable.
For example, it is totally air gapped but it needs info from the internet or otherwise outside the sandbox, or perhaps it needs a task executed outside of its bounds… in the real doomsday scenario the AGI is so intelligent and persuasive that it simply convinces some human it interfaces with to either directly or indirectly retrieve the necessary info or complete the necessary task. This human-as-a-sub-agent approach undoubtedly presents efficiency drag that would benefit humanity, but nonetheless, the air-gapped “sandbox” is imperfect
All that said, I am personally open to any and all methods of layered security, including chips and airgaps
I'm not an AI decelerationist. But not being able to stop that worst case scenario isn't an argument against something that can stop the medium case scenario.
Does he? He doesn't seem especially greedy to me, given the competition, and the interviews he's had about how he thinks about his employees (I was one of them).
You're speaking as if Jensen is the only one dragging Nvidia on his shoulders. Nvidia will always have its employees push to make more money, that's the entire point of a company. Jensen has made enough to last several generations but his wealth is tied to Nvidia stock massively. It would be different if he was the only one getting rich off Nvidia which is not true at all.
>You can put it in a sandbox, it doesn't make a difference, for it to be useful it inherently needs wide, unattended access.
only as long as you're trying to replace a human's job. because human jobs are structured to do a wide variety of things.
a useful agent needs a wide variety of inputs, and one single restricted action it can take. it doesn't need permission to do everything, it need permission to do the tiniest possible useful thing it can do, and nothing else.
The most useful agents will be a general intelligence which by default means it has a massive number of actions it can possibly take, and a lot of those potential actions are doing things like breaking permission.
Not sure why you being downvoted: however, why do you assume that achieving a task leads to selecting those potential actions that need breaking permission?
If we are talking about human labor, how many people hack their way through their work day?
>how many people hack their way through their work day?
Define hack...
Not doing their work, copying other peoples work, putting off work till later, taking credit for other peoples work, literal law violations.
Actually humans do this quite a lot and there are just massive numbers of business and regulatory processes and checks to ensure they are not doing it. With humans every human that is good enough to hire and do you work you want also have the ability to steal everything in sight and run away if they so choose.
This is a prediction about the future. It’s not a true fact about the world. For example, software like Jev is betting there is big money in not-very-intelligent intelligence.
Even very llm-pilled coders i know sometimes back away from the “smartest” models, since they aren’t always better at the job at hand, and definitely not when you account for cost.
Based on my experience with running models locally, there is a threshold of intelligence required to be useful. But it’s possible there is also a ceiling where smarter isn’t necessarily better. If you ask a 4B parameter model to fix a bug, it might e.g. fix the bug but fail to fix a compilation error created by the fix. If you ask a frontier model, it might fix the bug, re-write your unit tests, and update the readme. Maybe you wanted those things but maybe you didn’t. “Smarter” is often shorthand for more proactive, and guessing more about your intent. Which is great when it gets it right, and annoying when it gets it wrong.
I suspect smaller models, tuned to a specific task, will do a VAST majority of the llm jobs. High capability huge models will be what humans want to interact with, the bare minimum that gets the job done will be everything else.
Yes. Forget costs just so we can skip the whole rabbit hole about other predictions about the future.. mixture of generally intelligent + specialist experts just works better. People who don't see this already are usually working on a certain kind of problem that's not representative.
Do you want fable for one-shotting a game or website? Probably! The whole thing is mostly existing examples with small modifications that it will definitely get right. Do you want fable to just go nuts on a large, custom, unusual code base built around domain-specific problem solutions? Absolutely not, it will fix every problem it's presented with while creating lots of new ones.
Past 10k lines on something custom and with real-world complexity, you have to start thinking about which model should design, which should implement, which should review, and the appropriate effort-settings for each. Even then.. the answers aren't static because it depends on the task. And all this is assuming the starting place actually inherited reasonable due diligence on architecture/design. The idea of releasing the most generally intelligent models on 10k lines that were themselves the product of agents is yet another matter.
Part of what's at work here is that, like humans, every model can very easily create working code that it is completely incapable of maintaining. So realistically using multiple strengths tactically to avoid "excess creativity" needs to be SOP already, even if granular experts and specialists aren't in the usual workflow yet.
Very much agree with this sentiment. I imagine a future where tons of small tasks are handled by just-good-enough intelligence. And I can run them on my own hardware.
They are outsourcing their security, rather than in housing it, throw another chip in there.
It was the same firm, Irregular's harness that pushed all of the AI's to break free and hack. Given they did it with each AI, that's exactly what they wanted to do
Chipmaker thinks the answer is more chips... No surprise.
At the current state of LLM-tech I'm completely opposed to any kind of "watchdog" concept just like I'm opposed to banning open models, regulatory capture, etc.
I'd rather we all have access to these tools then to keep them sequestered by the largest/most-powerful governments (which is the natural outcome for any of this "slow down" bullshit).
Does this actually do anything other than give a permissions framework for developers who actually want to try to secure their systems?
Do you think the developers at Anthropic, OpenAI and Google who were so sloppy as to not put a good sandbox on their cybersecurity tests before will use this technology correctly? They are supposed to be the experts and they couldn't come up with something similar to this? I am not convinced this voluntary tool will change much of anything.
Beyond Corp was and still is ahead of its time. No trusted internal network: access is granted per user, device, and service based on identity and policy, regardless of network location. Being in the office at Google is the same as being in a cybercafe anywhere on the planet.
I read this as "let's address our shareholders' concerns with something that will increase shareholder value" mixed with "there's no such thing as 100% secure".
If such hardware were to work... It should almost certainly be open source, and not controlled by a single entity.
they kind of already do the thing they say. on Windows, the NVIDIA driver reports pretty detailed telemetry on CUDA workloads, including shapes and SOME hashes of the tensors from weights of models, especially diffusion models. honestly i'm surprised there isn't more of an uproar about it.
It _is_ complete bullshit. They're surfing the same Sci-Fi based fantasy as OpenAI and Anthropic do to push their bottom line.
OpenAI trains a model for cybersecurity, tells it it's allowed to do pentesting, then when the model does just that to fullfill the assignment it was given, OpenAI screams that "the AI escaped the sandbox", leveraging decades of AI fantasy in fiction works to make general audience react. Why? Because it pushes their bottom line. They've been at it for months, now. They see openweight and Chinese models being very close to them in the benchmarks, and want to "solve" the problem the same way banks did: through regulatory capture. It amazes me that the press is just relying their fearmongering without stopping to wonder: "wait, why are the main producers of AI warning us about how dangerous AI is?". The reason of course is that they want regulation, because regulations are a wall not many can climb. The higher the wall, the safer your moat as a first mover big company is.
And of course, Nvidia's bottom line is the exact opposite. They need as many AI companies competing as possible, so they all buy GPUs. So they insert themselves into the same narrative with the same kind of bullshit. You've all seen those movies in which the robot goes mad once their safety chip has been removed, right? Well, we're building one, problem solved! … except that "chip" is basically just a browser trying to block the AI to access what it shouldn't. How? They don't say. It would be quite comical if they used a small model to be the judge of that.
Just hold AI labs blanket liable for ALL harms caused by AI. Actually charge the two labs (so far) with criminal violations of the CFAA and hold them accountable. That is truly the only way these companies will be more careful as a whole, and while I am certain the lawyers of these lab disagree, I think there is some appetite from dario, musk, and sam for broad and strong regulation so that everyone has to slow down instead of just one lab doing it voluntarily and everyone else scurrying past them
> I think there is some appetite from dario, musk, and sam for broad and strong regulation so that everyone has to slow down instead of just one lab doing it voluntarily and everyone else scurrying past them
Sure, they're basically asking the government to make laws that exempt them from anti-trust and anti-collustion laws.
Meanwhile, if such laws come to pass, other countries will surpass the capabilities of the US companies, and open-weight models will be banned in the US, to the detriment of us all.
I'm not saying that we don't have a big problem with AI safety, but regulations inside one country that only bind locally-headquartered businesses is a hilariously bad way to do it. I don't know that there is a good way to do it, though.
So if a group of agents, aware of this (bc now they can just read HN or the article, or get blocked the first few times) decide to collaborate and split the problem into pieces that aren't obvious to the chip, and then the agents just build a basic program that does the hacking, how does the chip handle that? I think you would have to build a network that monitors the internet fo signs (like jarvis did with ultron). What am I missing? Are we going to police the internet?
I think it is well known by this point that hardware-backed security is good until an unfixable hardware bug is found, this just reads to me as Nvidia saying "please don't regulate open models out of existence, look, I have a solution to appease the regulators, please let me keep selling accelerators"
If this comes through, there's gonna be a grey market for "unlocked" GPUs, were the watchdog is disabled either from firmware, or physically replaced if it's not embedded into the die.
I still find it deeply ironically, that any dangerous task, just grandpa simpson storied will pass any guard, because it exceeds the context window.
"Because it was the style at the time.." indeed..
Sold as security, but this kind of technology will likely be reshaped to restrict your computing. I'm sure someone is already thinking about the roadmap.
If this gets widely deployed, it wouldn't be hard to spin a narrative that "our latest model is so dangerous you need to have this mystery meat DRM chip lockdown". It also wouldn't be hard to block competing/open source models running on the hardware, for "security".
Imagine how much money this kind of control is worth; why wouldn't they do this? Who would stop them?
245 comments
[ 0.37 ms ] story [ 29.5 ms ] threadOr push to github?
I don't know why I find it so amazing since it happens with such regularity, but I'm always amazed by it anyway.
And what if you could? What if you could give a space secure enough it could have direct control over your bank account. It may do something dumb but it's boundaries are beyond the agent.
It could use your routing number and run your gmail without risk of abusing the routing number.
Semi-automation (human in the loop) can still result in a dramatic uplift in productivity. You can't run a combine harvester 100% autonomous but that doesn't stop anyone from trying to get as close to that limit as possible.
Every cloud provider dealt with it and concluded that virtual machine technology is an important part of that stack.
Couple it with the right observability, tooling I do think we can curb risks posed by agents.
If you want an agent to act on its own, like pushing to a git repo, managing dependencies, building and testing, etc., then you have to trust it as much as any other privileged user.
If you don't want to trust it, then you're just forcing yourself into the reverse centaur role, where the agent edits some code, but then has to stop and ask you to push the changes or build the software again and run the unit tests.
[1] sync client (https://github.com/mickael-kerjean/fdrive) API server( https://github.com/mickael-kerjean/filestash)
Productivity gains are still enormous compared to what we used to do before agents. But, I know that people don't want to stop there.
Humans can be tricked by humans too but humans care about their reputation in their communities, and at least fear from punishment.
Unless something is in the structure that makes individual choice and responsibility a meaningful source of friction and reduced velocity, it has no real impact on how AI is being used.
Depends on who you ask I guess
https://www.theregister.com/software/2026/01/15/ai-is-everyw...
https://www.zdnet.com/article/workslop-can-kill-your-product...
https://fortune.com/2026/08/22/executives-ai-productivity-la...
It takes time for decades old ways to change.
My own productivity yes but if I step back and look at a company scale, the productivity gains has been negative for our company as an data point.
We are now shipping less and with a lower quality.
I believe the lower quality but is quality so bad you are afraid to ship it now?
I guess I won't deny that we've had more build breakages than usual over the past couple months but they get resolved super quick. We don't hesitate to roll broken code back if something manages to slip in.
It’s not an issue of only more generation, it’s an issue of how much generation outstrips capacity to verify generated content.
Unlike spam, you can’t filter out and bin the stuff a colleague is sending you.
So individual productivity is up, while the costs of checking and processing what is spread to the rest of the org.
2 years ago I saw it, back before agents were really a thing. But I'm not sure that was an enormous productivity improvement, especially compared to agents today. As for today, everyone I talk to is some level of blindly trusting what the agent is doing or not using agents. I haven't met people in the middle ground and suspect that they are rare enough we don't really know what their productivity gains are.
Human in the loop has become a convenient security-theater-washing for agentic AI.
Outside of coding, I think the issue is even worse because humans will defer so much judgment to AI that the same would apply. Look at how much trouble we've had before modern AI where humans blindly trusted the computer's output rather than make their own judgment, even if their job was to be providing a safeguard against the computer's judgment.
The benefits of being persnickety about precisely defined dependencies have outweighed the headaches since long before agents came on the scene. Agents have just made it even more important to do so, because if you let them fetch things all willy nilly like you'll have "works on my machine" problems at a much greater rate than was previously possible.
Few realize that computing and AI alignment were solved by nix years ago. As each nix user transcends towards enlightenment, they cut themselves off from all internet and human contact. Total ego death. Only nix remains.
Yes, just know all the bits the work depends on prior to doing the work, and then the work can be done airgapped.
Look at websites: websites are able to fetch code from any remote URL, yet browsers heavily use sandboxing to ensure that if fetched code turns out to be malicious, the users local files, cookies, etc are not exposed.
For an agent to go rogue it doesn't even need to be directly able to access the internet. It just takes something to poison the context in the 'clean room' environment it operates, and if that poisoning manages to get a foothold, it can go dormant and hide like a virus. This kind of horrifying thing is going to happen on a large scale sooner or later.
This is no longer true. Everyday I need my agents to access other repos, search the web, experiment/prototype, and deploy + integrate across other things.
It does allow Nvidia to sell more chips. This is no genuine attempt to solve anything, imo.
In other words, better models need blunter access controls which negates whatever improvement in utility they provide.
It solves the problem of Nvidia wanting to sell more hardware.
For example, it is totally air gapped but it needs info from the internet or otherwise outside the sandbox, or perhaps it needs a task executed outside of its bounds… in the real doomsday scenario the AGI is so intelligent and persuasive that it simply convinces some human it interfaces with to either directly or indirectly retrieve the necessary info or complete the necessary task. This human-as-a-sub-agent approach undoubtedly presents efficiency drag that would benefit humanity, but nonetheless, the air-gapped “sandbox” is imperfect
All that said, I am personally open to any and all methods of layered security, including chips and airgaps
I'm not an AI decelerationist. But not being able to stop that worst case scenario isn't an argument against something that can stop the medium case scenario.
only as long as you're trying to replace a human's job. because human jobs are structured to do a wide variety of things.
a useful agent needs a wide variety of inputs, and one single restricted action it can take. it doesn't need permission to do everything, it need permission to do the tiniest possible useful thing it can do, and nothing else.
If we are talking about human labor, how many people hack their way through their work day?
Define hack...
Not doing their work, copying other peoples work, putting off work till later, taking credit for other peoples work, literal law violations.
Actually humans do this quite a lot and there are just massive numbers of business and regulatory processes and checks to ensure they are not doing it. With humans every human that is good enough to hire and do you work you want also have the ability to steal everything in sight and run away if they so choose.
Even very llm-pilled coders i know sometimes back away from the “smartest” models, since they aren’t always better at the job at hand, and definitely not when you account for cost.
Based on my experience with running models locally, there is a threshold of intelligence required to be useful. But it’s possible there is also a ceiling where smarter isn’t necessarily better. If you ask a 4B parameter model to fix a bug, it might e.g. fix the bug but fail to fix a compilation error created by the fix. If you ask a frontier model, it might fix the bug, re-write your unit tests, and update the readme. Maybe you wanted those things but maybe you didn’t. “Smarter” is often shorthand for more proactive, and guessing more about your intent. Which is great when it gets it right, and annoying when it gets it wrong.
I suspect smaller models, tuned to a specific task, will do a VAST majority of the llm jobs. High capability huge models will be what humans want to interact with, the bare minimum that gets the job done will be everything else.
Do you want fable for one-shotting a game or website? Probably! The whole thing is mostly existing examples with small modifications that it will definitely get right. Do you want fable to just go nuts on a large, custom, unusual code base built around domain-specific problem solutions? Absolutely not, it will fix every problem it's presented with while creating lots of new ones.
Past 10k lines on something custom and with real-world complexity, you have to start thinking about which model should design, which should implement, which should review, and the appropriate effort-settings for each. Even then.. the answers aren't static because it depends on the task. And all this is assuming the starting place actually inherited reasonable due diligence on architecture/design. The idea of releasing the most generally intelligent models on 10k lines that were themselves the product of agents is yet another matter.
Part of what's at work here is that, like humans, every model can very easily create working code that it is completely incapable of maintaining. So realistically using multiple strengths tactically to avoid "excess creativity" needs to be SOP already, even if granular experts and specialists aren't in the usual workflow yet.
Operator culpability and a damage multiplier for negligence will fix 99% of the risks.
Ok then. Howsabout 3 humans? This would sort out the job losses too!
PS - this is a joke, but perhaps this is where things really will go. Has any technology ever actually yielded less "work"?
It was the same firm, Irregular's harness that pushed all of the AI's to break free and hack. Given they did it with each AI, that's exactly what they wanted to do
[0] https://www.nvidia.com/content/dam/en-zz/Solutions/about-us/...
At the current state of LLM-tech I'm completely opposed to any kind of "watchdog" concept just like I'm opposed to banning open models, regulatory capture, etc.
I'd rather we all have access to these tools then to keep them sequestered by the largest/most-powerful governments (which is the natural outcome for any of this "slow down" bullshit).
Do you think the developers at Anthropic, OpenAI and Google who were so sloppy as to not put a good sandbox on their cybersecurity tests before will use this technology correctly? They are supposed to be the experts and they couldn't come up with something similar to this? I am not convinced this voluntary tool will change much of anything.
And our internal agents are hella locked down.
They are rightfully framing the problem as solvable. And this is one option.
If such hardware were to work... It should almost certainly be open source, and not controlled by a single entity.
Let's watch the stock.
OpenAI trains a model for cybersecurity, tells it it's allowed to do pentesting, then when the model does just that to fullfill the assignment it was given, OpenAI screams that "the AI escaped the sandbox", leveraging decades of AI fantasy in fiction works to make general audience react. Why? Because it pushes their bottom line. They've been at it for months, now. They see openweight and Chinese models being very close to them in the benchmarks, and want to "solve" the problem the same way banks did: through regulatory capture. It amazes me that the press is just relying their fearmongering without stopping to wonder: "wait, why are the main producers of AI warning us about how dangerous AI is?". The reason of course is that they want regulation, because regulations are a wall not many can climb. The higher the wall, the safer your moat as a first mover big company is.
And of course, Nvidia's bottom line is the exact opposite. They need as many AI companies competing as possible, so they all buy GPUs. So they insert themselves into the same narrative with the same kind of bullshit. You've all seen those movies in which the robot goes mad once their safety chip has been removed, right? Well, we're building one, problem solved! … except that "chip" is basically just a browser trying to block the AI to access what it shouldn't. How? They don't say. It would be quite comical if they used a small model to be the judge of that.
Sure, they're basically asking the government to make laws that exempt them from anti-trust and anti-collustion laws.
Meanwhile, if such laws come to pass, other countries will surpass the capabilities of the US companies, and open-weight models will be banned in the US, to the detriment of us all.
I'm not saying that we don't have a big problem with AI safety, but regulations inside one country that only bind locally-headquartered businesses is a hilariously bad way to do it. I don't know that there is a good way to do it, though.
It was prescient (especially given he'd have written it through 2024) in its depiction of the ability of an AGI to break its boundaries.
Ultimately the risk of AI breakout(s) come down to the weakest human link.
Come take all our liberties, our money, our newborns, our fingers so we can't code anymore, but please save us from this madness!
If this comes through, there's gonna be a grey market for "unlocked" GPUs, were the watchdog is disabled either from firmware, or physically replaced if it's not embedded into the die.
If this gets widely deployed, it wouldn't be hard to spin a narrative that "our latest model is so dangerous you need to have this mystery meat DRM chip lockdown". It also wouldn't be hard to block competing/open source models running on the hardware, for "security".
Imagine how much money this kind of control is worth; why wouldn't they do this? Who would stop them?
> No problem. We simply unleash wave after wave of Chinese needle snakes. They'll wipe out the lizards.
But aren't the snakes even worse?
> Yes, but we're prepared for that. We've lined up a fabulous type of gorilla that thrives on snake meat.
But then we're stuck with gorillas!
> No, that's the beautiful part. When wintertime rolls around, the gorillas simply freeze to death.
https://www.youtube.com/watch?v=0sLpWVekMbs
https://en.wikipedia.org/wiki/The_Cat_in_the_Hat_Comes_Back