When I give an agent ssh access to something I want to be able to watch and fully understand what it's doing. I want it to essentially only "type" things into the CLI that I could have typed myself, I can comprehend what it's doing, and am not surprised by the results. Opencode and a smart LLM (qwen 3.8-flash-next, deepseek v4 0731 or smarter) do relatively well with this in my experience.
And if everyone was like you AI safety wouldn't be that large of concern. The default human behavior seems to be fire and forget which can go off the rails really quick.
FYI VSCode's SSH Agent is a godsend for remote development - the "disadvantages" that Fly lists are part of its advantages. I've worked in several teams that have made extensive use of the extension, and it's never been an issue. You can restrict SSH access arbitrarily to ensure whatever security or access guardrails you need.
Yeah this is the right architecture for remote editing with remote tools. It works really well. (There are longstanding bugs around reconnection when the SSH connection is broken but that's not the fault of the architecture.)
As a Linux user I've hated VSCode's ssh. There's lot of annoying things that make it harder to admin for. Like it doesn't pick up the MotD, preventing me from showing users important messages. I've found that it also doesn't reuse sessions (at least by default. TBF, neither does ssh) and I'll find that there's just dozens of open sessions over months from users. I literally had to write a script to boot people...
It would be one thing if the plugin was just a wrapper and people were still expected to know ssh but the plugin abstracts away all that and is intended to make it a "use VSCode on remote machine" tool. So it needs to do more than just handle creds, otherwise it creates a divergent experience while making people think it's just ssh
> There's lot of annoying things that make it harder to admin for.
It also (AIUI) tries to walk the entire file tree, so have fun with NFS (auto)mounts.
It also amounts to letting off fork bombs: we set up limits for a maximum of 256 process per UID, and regularly get folks asking "what does this 'cannot fork' message mean?": it mean you're trying to DoS the system.
> we set up limits for a maximum of 256 process per UID, and regularly get folks asking "what does this 'cannot fork' message mean?"
The max limit on 64 bit systems is what, 4,194,303? So if you have over 16,000 users per VM this limit makes sense, otherwise it just seems user-hostile.
Every process takes some memory and other resource, yes a stale process will pretty much all end up all paged out and not massively in the way of active processes, but they still aren't entirely free so it is more than a bean-counting number.
Yes, under Linux (and most unix-a-like systems) small processes are cheap to bring up and tear down which is why we create them so much, and it is not uncommon for complex interactive commands and bits of shell scripts to create several¹, but these are all likely to be short-lived so a limit of 256 certainly doesn't seem to be obscenely low to me.
What could it be doing that requires 256+ processes to be kept around for a prolonged time?
--------
[1] made up example: comparing filtered content of two gzipped files and sending the result through a script to send alerts by mail if certain things are found would be 7+ (2x gzip, 2x or more grep, diff, bash, mail or curl depending on what service you are sending alerts through)
> The max limit on 64 bit systems is what, 4,194,303? So if you have over 16,000 users per VM this limit makes sense, otherwise it just seems user-hostile.
And yet we still regularly loads of >100 on our 64 core HPC login codes, and swap is regularly used even with 96G of system memory (we have per UID memory limits too).
What's hostile is the VSCode (and Codex and Claude) makers developing tools that basically DoS a system because they assume it will operate only on single-user machines.
(And WTF are you doing that you're forking 256 processes? We have quite a few expensive HPC nodes: use those to build, not the damn login nodes.)
> The max limit on 64 bit systems is what, 4,194,303?
What a weird framing... I'm not sure what you're even trying to argue. I mean a single process can overload the machine. Just because you can label 4m processes doesn't mean you can actually run that many programs. Just think about that for a minute. 256 processes is pretty generous
What would truly be user hostile would be to allow so many processes per UID that a small handful (maybe using VSCode) make the system slow or unresponsive to everyone else.
Honestly every developer needs to increase those default limits, they are too low for modern development... So you are just crippling them and a proof of that is they keep getting this error while in their regular workflow
I don't even hit 256 on my desktop with a shitton of things open and 75 firefox sandbox processes, much less on a remote server. What in heaven's name are you doing to cross 256?
Honestly I think this thread has just devolved to HPC admins vs people who don't understand how shared multiuser sytems work cause they've been stuck on a laptop for too long to remember.
> Honestly every developer needs to increase those default limits, they are too low for modern development...
The users can develop on >100 compute nodes, but choose not to bother doing any kind of forwarding/proxying/jumping to them and just do stuff on the login nodes.
If they can't be bothered to do a "ssh -J …" then it's on them. The resources are there.
Ah, I see. Our admins typically understood that users were being given disk quotas precisely so that they could use that disk space, but that's probably not a universal stance.
Quotas mean "more than this is clearly too much" not "please use this space".
In good times nobody minds, but in bad times when you just can't extend the drive you have to tell people off. Part of the job of making sure the system can continue to work for what you need it, under _real_ constraints.
You may have to delete stuff, you may have to shut the server down to save power. You may have to limit clock speeds. It depends on the environment and "it really should work because it should be covered by next day on site warranty and you could download more ram" often doesn't apply.
> Quotas mean "more than this is clearly too much" not "please use this space".
Ah, memories of Uni, back when storage was fairly expensive, where we had both hard and soft quotas. The soft quota would allow for temporary growth of build artefacts and things¹ but you would get stern emails if you were over your soft quota for 24 hours, and if you persisted without good reason³ your hard quota would be reduced so you effectively have no soft quota any more.
--------
[1] some machines had no local storage that the user could touch so putting them there was not always possible, some people on Windows machines had local storage but didn't have the relevant tools locally so were actually running things on the shared server(s)² instead of that just being a storage resource
[2] via telnet/rsh/rlogin: yes, I am that old… SSH was a thing by that point, though OpenSSH wasn't, and I was using it where available, but the use of older plain-text protocols was still far far more common
[3] it wasn't actually difficult to justify a quota extension for project work, in fact people enrolled on certain modules got higher quotas automatically
Yes, let's hope it doesn't get even more expensive. Around here we are purposely not upgrading hardware unless we have to, hoping to ride through the price increases. I'm sure lots of people are doing the same thing, which probably won't help much once prices start to come back down... assuming they do.
This is a fascinatingly "pets" approach to admin; it's been ages since I've been somewhere that used this approach. I've been in the "cattle fields" for decades now.
In my ecosystem, developers don't have time to glad-hand like this; if there are issues with DEV_NODE_CFG_1_29875, we might talk about it over Slack (because I'm the first one to know there's a problem as the end-user), and if we can't sort it out they'll give me some time to backup, blank the whole machine, and I get a brand-new image of DEV_NODE_CFG_1_29875.
I can't even tell you off the top of my head what territory the physical machine is running in or whether it's the only dev-node on that hardware.
(Broadly speaking, I think Microsoft is assuming cattle ecosystems; they're not openly-hostile to pets per se, but they have a strict ranking of the priorities because the "cattle ranches," as it were, bring in more money).
I had it seen on servers that were typically reserved for the team but there was no official booking system for those machines. When you start using the machine you would typically put some note to make sure somebody else does not overrun your long-running tests or performance measurements.
Sometimes you aren't really "the owner". For one example I was admining my group's server in grad school. I wanted to add quotas (we already had zfs) because people were abusing home directories but my advisor and a few members were very against it because I was "over complicating things". Their worry about me wasting time (20 minutes for all machines...) resulted in hours of yelling at people over the years. All because, surprise, a small number of people can't follow instructions and abuse systems, ruining it for everyone.
Another frequent problem we had was people using the systems while others were. They wouldn't check the machine's status. And of course you can guess that I wasn't allowed to add a scheduler.
A lot of groups do things in janky ways. Often not because they don't know any better but because leadership doesn't and is assertive
We have the servers role, you can derive that from the name obviously, but we do have hosts which as the same naming scheme, but slightly different roles. There's when the last Puppet run happened and what it applied (and who authored it). Depending on the host type there's also active/standby, warnings for production hosts or information about increased log level on things like sudo.
It sounds like a lot, but it's fairly compact and really helps when you need to absolutely sure where you are and you have eight terminal windows open.
"The announcement was visible in a MOTD on every server in the fra3 location for the last two days."
"I have not touched any machines in fra3 for a week, how the hell would I know of it? Why do we even have #fra3-maintenance and #maintenance channels then, if that's your stance?"
> "I have not touched any machines in fra3 for a week, how the hell would I know of it? Why do we even have #fra3-maintenance and #maintenance channels then, if that's your stance?"
We put the MOTD up ≥7 days in advance and put it in relevant Slack channels.
I miss fingering, it was such an easy way to get a log dump or status update from various daemons, I still think it has immense utility .. some of my fondest operator days were sat under the umbrella with a terminal while sleep 30 ; finger someone@all-the-things ; done .. watching the machines from afar.
Can still do it these days of course, but one with a seriously copious helping of ssh in the mix too ..
Trouble is, nobody else can do it. The only reason I have to use {social-media-blob} is because my friends don't know how to finger.
Things that should probably be an email. I think one time I’ve seen it work is to tell you stuff specifically about the host you’re on to avoid mistakes.
> I'll find that there's just dozens of open sessions over months from users
We've had the same issue with our local HPC; a few login nodes serving hundreds of users at a time, and each login node used to get swamped by these dangling SSH sessions/servers. They also wrote a script that shuts down all sessions once a day to save the login nodes.
The one this that is better than just SSH+Tmux+Vim, is that if your latency is higher than 30-50ms, since VSCode's SSH agent streams the files to your computer, the typing experience feels snappier. When you work half a continent away from where the servers are, it makes life nicer.
Use a local vim, and use its ssh support. It will download the file to the local buffer then upload it when you save. This way your vim config remains on your local machine, too.
Related, netrw (vim) can do quite a lot of things that I think people don't realize. I mean not just that it supports file trees, but well... just open :h netrw
It's just another messaging. I used them to automate messages about current disk usage and warn if the machine was actively being used.
But then again, I had people who would run jobs without checking if the machine is already in use. Obviously these people didn't check email or slack either...
The ethos of VSCode was supposed to be lightweight, something like Notepad++ with a terminal. Developers have lost the plot. Please recommend Visual Studio if you would like a feature-rich SSH agent. I think developer trends have supercharged VSCode and it feels shinier and new with all the extensions but this is an anti-pattern; it defeats the whole purpose.
For all practical purposes VSCode is a vessel to sell AI, im using vscodium for now but expect a cycle reset for it this decade (someone makes a new lightweight IDE etc)
I don’t think it was ever intended to be lightweight like Notepad++ given the architectural designs from the outset (LSP, Chromium-base, etc).
It always felt to me more like a desperate attempt by Microsoft to regain the IDE market share as low end FOSS editors started taking over. So MS wanted to appeal to the open source community.
And it worked. Even if the primary build of VSCode which most people run isn’t technically open source.
Not really sure what low end FOSS editors you are referring to? Textmate?. The Monaco editor predated vscode by 4 years and it was an attempt to build a browser-based texteditor. Vscode happened after Atom (which was not low end) made electron apps viable. Both were was an attempt at SublimeText (which was not FOSS). SublimeText itself was a cross platform alternative for textmate.
The FOSS text editor space was always crowded between vim and emacs. FOSS alternatives would get measured against these 2 behemoths and it was a tall order to compete against. Now FOSS IDEs were a different story.
Java, C#, and C++ had many sophisticated and advanced IDEs, some FOSS some not. However, PHP, Python, Ruby, and JavaScript were quickly gaining huge mindshare and those developers didn’t want to install Visual Studio, Eclipse, NetBeans, or IntelliJ. It’s a tough proposition to tell a Ruby dev to install Java, then install Eclipse, then install an extension, then learn Eclipse nonesense, edit your Ruby files vs “just open SublimeText and edit your files”. Most of what those developers wanted was just syntax highlighting and basic directory navigation to begin with. Atom/vscode were an attempt at SublimeText alternative. VScode was objectively better than Atom and TypeScript was also objectively better than CoffeeScript.
I’ve been using it daily since it came out. At first it was because I was tired of docker slowing my Mac down with some really heavy client projects. But now I use it as an easier ssh client w/ file editing. I really don’t like using vim/nano. Keeping everything in the same ide, huge for me.
Also using it, and by now at least I see the reason why they did it. VSCode has a large plugin ecosystem, many which are essential for development. The problem is that those plugins don't know anything about remote development and expect to use the standard file system and OS APIs to interact with the workspace.
So how to make the plugins remote-capable? You could write a massive virtualization layer that captures all system calls and forwards them to the remote - or, you could run the plugin on the remote and just pass the user commands and UI updates over the connection.
VSCode does the latter, so the nodejs runtime is where all the plugins are running on the remote.
(I understood the reason, I didn't say it was a good reason...)
It also fills up the hard drives on shared server, where each user up to 5 GB of vscode nonsense.
Also, I realized that students are likely to use vscode to connect to embedded Linux machines and fill up their emmcs (and RAM), so I came up with a partial solution for our experiment's yocto image (https://github.com/RNO-G/meta-rno-g/blob/main/recipes-suppor...) , but a more general solution would be nice. Probably better to just not allow any node process to run via LSM.
It also has some unfortunate OS / glibc minimums which means that it's growing less useful as time goes on. I work on a lot of machines that are from centos 7 era (including amazon linux 2, which doesn't have an upgrade path: you just have to build a whole new instance and migrate). I have had to pin my vscode + extensions to old versions because it works on older machines. They just stopped supporting platforms (with lots of notice; to be fair). I would love if they had a binary tool that could be built which was much more agnostic to the vscode / extension version so that I could just keep using it everywhere I've been using it, while allowing me to keep current with the latest and greatest.
At least when Python minimum version changed there was a separate extension forked from the original that I can use when I need to work on old code alongside the new extension for more modern code bases. Sadly, no such thing exists for remote editing that I know of.
I run the editor (and its extensions), my projects and any agent harnesses from inside a container and use that extension to get an editor.
I have some shellscripts to quickly get in and out of the container. It works pretty well, I barely notice the container anymore.
This is mostly to protect my credentials and data from malicious extensions/dependencies/rogue-agents, bu it also lets me quickly port my dev environment to any machine (I use linux at home and macos at work). Just install podman, install VSCodium, add the SSH extension, build image, add my utility shellscripts and done.
Apparently Microsoft keep some VSCode APIs proprietary so only its own extensions can use it (allegedly for security reasons), which is why this specific extension only works in VSCodium.
I used VSCodium before and found it to be a massive waste of time for no benefit. A significant number of extensions either aren't in OpenVSIX (or whatever it's called), or don't work for some reason.
Just disable telemetry in VSCode and you're good.
Yeah, but to be honest on my current project the only extension that I needed, but wasn't available was the Typescript 8 support (the typescript with golang-based language-server) which should land eventually in the built-in typescript support.
I've been using VSCode Remote since it came out and it is indeed a godsend. I use it to develop from one trusted machine on my own LAN to another trusted machine on my own LAN and I want the experience to be as similar as possible to developing locally, which it is, and which is why I appreciate it.
I'm not surprised people trying to provide remote code editing to non-trusted clients are freaked out by it, but it is a great tool when operating in a trusted circle.
> Emacs hosts the spiritual forebearer of remote editing systems, a blob of hyper-useful Elisp called “Tramp”. If you can hook Tramp up to any kind of interactive environment — usually, an SSH session — where it can run Bourne shell commands, it can extend Emacs to that environment.
vs.
> The agent runs over port-forwarded SSH. It establishes a WebSockets connection back to your running VSCode front-end. The underlying protocol on that connection can: Wander around the filesystem; - Edit arbitrary files; Launch its own shell PTY processes; Persist itself.
So... basically the same things that Tramp could do as well?
> In security-world, there’s a name for tools that work this way. I won’t say it out loud, because that’s not fair to VSCode, but let’s just say the name is murid in nature.
Yeah, it's called RAT, and an ur-example of it is SSH itself (especially when allowed to run a shell remotely), so... not sure why are you freaking out.
I mean, I'd probably prefer if VS Code simply ran ed/vim remotely, but both of those editors can invoke shell anyhow so... eh?
The difference is that there is an SSH agent at all, whereas Emacs just uses the built-in Bourne shell.
Some people get an icky feeling where remote editing tools change the remote filesystem in any way that is not explicitly done by the user. A binary installation of Node is extra extra icky. (I don’t have Node installed anywhere on my computers; I avoid JS if possible and if not I prefer Deno.)
No, with TRAMP its a one way thing. The remote side can't wander around the local side at all, at least from my understanding of TRAMP. So it's no the same at all.
Wait, you mean that with the VS Code protocol, the remote side can send the commands to my local VS Code, making it do all kinds of funny stuff? Seriously? That's not really obvious from TFA at all and mostly reads as their unease with running somewhat opaque piece of software at the remote written for Node instead of presumably well-maintained bunch of sh scripts (a very interesting supposition IMHO).
This part of VSCode's architecture is acceptable to me. The reverse direction, where a compromised remote can do whatever it wants to my local machine, is not.
Security Note
Using Remote-SSH opens a connection between your local machine and the remote. Only use Remote-SSH to connect to secure remote machines that you trust and that are owned by a party whom you trust. A compromised remote could use the VS Code Remote connection to execute code on your local machine.
The agent is supposed to run on a remote dev box. The purpose is to make the remote machine an extension of your local one, to run extensions, containers, test deployments, forward ports and tons more. If you are installing it on production servers and are surprised by its behavior that’s on you.
> # Security Note
>
> Using Remote-SSH opens a connection between your local machine and the remote. Only use Remote-SSH to connect to secure remote machines that you trust and that are owned by a party whom you trust. A compromised remote could use the VS Code Remote connection to execute code on your local machine.
It's like giving away a gun that explodes in your face if you shoot it in anger, but with a prominent label on the box that says 'WARNING: Will Malfunction!'. And then heavily promoting it.
why would anyone connect with this to an unknown/untrusted machine? it's more like giving away a gun that will shoot you if you point it at your own head and pull the trigger
Because systems are made of components based on assumptions. The owners of a sandboxed LLM development environment provides isolated VM workstations, and allow access via SSH, with things like X forwarding disabled. IP connections are dynamically permitted via access tickets and time ranged 2FA.
Separately, developer becomes a VSCode user. Risk as assessed based on them developing on their Dec machine, accepted.
Developer then opens a ticket to use LLM for a project. Uses VSCode SSH Agent. Agent creates exfil tooling on workstation and reverse shells their box. Agent reads unauthorised data which possibly has LLM subversion triggers, or deletes data, maybe opens connections to C2 service. LLM agent pivots to cloud inference and continues to lateral movement.
The assumptions of systems composition are violated by VSCode. That's why its SSH Agent needs to be blocked.
Not sure what you consider "no good" alternatives. jeanp413.open-remote-ssh[0] has worked flawlessly. It's the first or second search hit in the extension manager for "ssh". It's a fork of Microsoft's previous extension before the proprietarized it.
I very much dislike this extension because it uses a built-in ssh client that is far from feature complete and does a lot of reinventing the wheel. Microsoft's extension uses the system ssh binary, as does https://open-vsx.org/extension/aergic/zygos-vscodium which I much prefer. After discovering the latter, I uninstalled VSCode and have fully moved to VSCodium.
> It turns out we don’t have to care about any of this [...], so none of this matters in any kind of deep way, but: we’ve decided to just be a blog again, so: we had to learn this, and now you do too.
I found this closing sentence utterly delightful, particularly in an age of endlessly filtering every piece of text I read on the internet through a mental "was this written by Claude, Codex, or (just possibly) a human?" filter.
off topic, but I feel this observation was quite early in feb' 2025: "LLM-generated code is useful in the general case if you know what you’re doing. But it’s ultra-useful if you can close the loop between the LLM and the execution environment (with an “Agent” setup)."
So a program that is specifically designed to edit files and run arbitrary commands on a remote machine... can do so. Not sure where the bananas part comes in. Sending a binary over SSH/SFTP might sound weird at first glance, but VSCode can't assume that your remote machine can access the wider internet. So anything that VSCode needs running on the remote that's too complicated for a shell script is going to be shipped over the SSH tunnel.
I think the actual concern, not well expressed in the blog post, is the fact that node and vscode server are installed on, for instance, a prod machine that (probably) should be very tightly controlled in terms of what software is installed and running. You don't want to unwittingly add to the attack surface
This should be covered by not giving developers SSH shell or equivalent access to production machines in the first place, or at the very least to have measures in place (ACLs, quotas, etc.) to strictly limit what they are able to do from the shell.
Oh wow I just checked my servers and on one the folder is 3.2GB and another has 1.7GB.
It kind of looks like it just keeps copies of every version its ever installed? Theres several copies of node and `node_modules` folders.
I'm not super low on space but it also pretty clearly doesn't need to be that large, it looks like most of it is copies of itself.
Yeah, I learned how it worked when I made the mistake of trying to use it to develop on a Raspberry Pi, where it filled the disk and crashed/hung the Pi by using all the RAM until it started using swap. Fell back to local with an sshfs mount instead.
Using Remote-SSH opens a connection between your local machine and the remote. Only use Remote-SSH to connect to secure remote machines that you trust and that are owned by a party whom you trust. A compromised remote could use the VS Code Remote connection to execute code on your local machine.
TRAMP (mentioned in the article) does it without installing anything on remote machine, just SSH, and shell commands. Which sounds more natural to me. Node.js security history, with all due respect, is not shiny. And the problem the author has with VSCode's way, I guess, is not that it can edit files, but that it extends an attack surface without need.
TRAMP actions are also rather slow (high latency). OTOH, tramp-rpc relies on a little tool to run on the remote and is much snappier. This proves there is a better middle ground than TRAMP with nothing and whatever abomination VSCode injects. Basically, busybox with a persistent RPC connection is all one needs.
For my own agent one of the design constraints is that it can't get out of the work directory, and it can't even try to guess the full path of that directory.
Interesting that VSC has gone the other way entirely.
I avoided Zed for a long time because of the SSH feature in VScode. Then I realized Zed has SSH remote too. Just flagging this for anyone else who relies on vscode-over-ssh and is sick of the bloat.
Cloud VM runs the built-in `code serve-web` command, over Tailscale using `tailscale serve`. No SSH. Extensions work. No extensions run locally (with SSH some extensions have to run locally on your machine).
I’m trying to see how little I can run on my local machine. VS Code over SSH is a good step in that direction, but there’s more attack surface if SSH is misconfigured, plus risk of an extension getting compromised.
I expected the added layers (neovim, through an extension, inside VS Code, over the web) to be slow, but so far it works surprisingly well.
You would think, now that we're running demon-powered code on our sandboxed machine, that allowing that machine to have full control over the local client would be considered a bad idea.
But Microsoft won't fix VSCode because it would possibly impede usability by amateurs, and inject tedious security concepts like boundaries into the dev process.
> The agent runs over port-forwarded SSH. It establishes a WebSockets connection back to your running VSCode front-end. The underlying protocol on that connection can:
Wander around the filesystem
Edit arbitrary files
Launch its own shell PTY processes
Persist itself
Wait, could someone clarify which machine is being referred to here?
So in the author's setup, he runs VSCode (i.e. the front-end) on his dev laptop, which he wants to keep free of direct LLM access.
VSCode connects via ssh to a dedicated "sandbox" machine on which the LLM will be free to do whatever it wants (mostly).
VSCode realizes this the Microsoft way, by using the ssh connection to install VSCode Server on the sandbox machine - the "backend" - and communicating through it via a websocket connection.
So then, what happens? If the websocket connection allows the front-end to run arbitrary commands on the sandbox machine, this wouldn't be very exciting: The front-end already has an ssh connection and a massive server process that can do the same - and the entire purpose of the sandbox machine is to run arbitrary, untrusted commands without harm.
But the article says the websocket connection goes "back to your running VSCode front-end". So does that mean things are reversed? I.e. the agent/harness runs in the server on the sandox machine but for some reason has this websocket connection that also lets it run arbitrary commands on the dev laptop?
> Wait, could someone clarify which machine is being referred to here?
I'm not the author, but I can tell that he means the remote machine. His sentence about being "nervous about letting people VSCode-remote-edit stuff on dev servers, and apoplectic if that happened during an incident on something in production" makes it clear. He considers the VS Code agent to have all the features of a rootkit, and doesn't want anyone to be deploying it onto dev servers, let alone production ones.
That makes no sense to me though. The VSCode frontend already has an open SSH connection to the remote machine, over which it could do the same things and more. Why is the websocket connection (which is probably tunneled through the SSH connection anyway?) any worse here?
And the fact that it can go both ways is why the author would be "apoplectic if that happened during an incident on something in production" (emphasis mine). Because if you're doing this during an incident, you're connecting to a server that you have reason to think might be running malware right now. Which means that if the malware is programmed to look for incoming VS Code remote connections (not that hard to do, anyone competent enough to write the malware causing your production incident is probably capable of programming it), the malware could then infiltrate your dev machine via the VS Code remote tunnel. Your dev machine where you might well have credentials lying around in plaintext .env files.
Yeah, I'd be apopleptic too. Because the clueless dev who did that just escalated the production incident into "track down every credential that was present on that particular dev machine, and assume those credentials are now compromised and have to be rotated".
Yes, the "agent" is not an LLM agent here it's the "ssh agent" that connects VSCode front-end to the remote server back-end..
It's astonishing how many commentors(not you!) either didn't read or didn't understand the blog post, saw "agent" and thought LLM agent, and then decided to comment about it...
That's the word I thought he was hinting at... but then I looked up the word "murid". It has two meanings, one from Sufism (a novice seeking spiritual enlightenment) and one from science (muridae is the family of rodents that includes rats and mice). Neither one of those seems to match "rootkit". I suspect he intended the mice-and-rats meaning, but I don't know what word related to rats or mice might have a similar meaning to "rootkit".
EDIT: Found it. He's referring to the acronym RAT meaning "Remote Access Trojan" (or, if actually intended, Remote Administration Tool).
I think the last time I linked SSH to a few V'sM through VSCode it auto-installed node, npm, and hundreds of megabytes of npm packages, then it persisted a node service. It's bananas that it still does this. I have never used SSH through VSCode again.
Allow me to translate this for the vibe bros. The "agent" here is not an LLM agent, it's the SSH agent that connects the VSCode front-end to the remote back-end.
The issue the article highlights is this opens you to local code execution initiated by the remote.
> Using Remote-SSH opens a connection between your local machine and the remote. Only use Remote-SSH to connect to secure remote machines that you trust and that are owned by a party whom you trust. A compromised remote could use the VS Code Remote connection to execute code on your local machine.
Yes this article is from Feb 2025, but NOTHING HAS CHANGED in this regard. Except perhaps now MANY MORE PEOPLE are believing they are safe using VM and other remote SSH "sandboxes" to develop on via VSCode remote host connection over SSH. The reality is, as stated by Microsoft themselves, if you can't trust the remote.. It can own your local computer. And that's not much of a "sandbox"..
177 comments
[ 7.2 ms ] story [ 99.5 ms ] threadFYI VSCode's SSH Agent is a godsend for remote development - the "disadvantages" that Fly lists are part of its advantages. I've worked in several teams that have made extensive use of the extension, and it's never been an issue. You can restrict SSH access arbitrarily to ensure whatever security or access guardrails you need.
The problem is the remote host has control over local host through the protocol.
It would be one thing if the plugin was just a wrapper and people were still expected to know ssh but the plugin abstracts away all that and is intended to make it a "use VSCode on remote machine" tool. So it needs to do more than just handle creds, otherwise it creates a divergent experience while making people think it's just ssh
It also (AIUI) tries to walk the entire file tree, so have fun with NFS (auto)mounts.
It also amounts to letting off fork bombs: we set up limits for a maximum of 256 process per UID, and regularly get folks asking "what does this 'cannot fork' message mean?": it mean you're trying to DoS the system.
The max limit on 64 bit systems is what, 4,194,303? So if you have over 16,000 users per VM this limit makes sense, otherwise it just seems user-hostile.
Yes, under Linux (and most unix-a-like systems) small processes are cheap to bring up and tear down which is why we create them so much, and it is not uncommon for complex interactive commands and bits of shell scripts to create several¹, but these are all likely to be short-lived so a limit of 256 certainly doesn't seem to be obscenely low to me.
What could it be doing that requires 256+ processes to be kept around for a prolonged time?
--------
[1] made up example: comparing filtered content of two gzipped files and sending the result through a script to send alerts by mail if certain things are found would be 7+ (2x gzip, 2x or more grep, diff, bash, mail or curl depending on what service you are sending alerts through)
And yet we still regularly loads of >100 on our 64 core HPC login codes, and swap is regularly used even with 96G of system memory (we have per UID memory limits too).
What's hostile is the VSCode (and Codex and Claude) makers developing tools that basically DoS a system because they assume it will operate only on single-user machines.
(And WTF are you doing that you're forking 256 processes? We have quite a few expensive HPC nodes: use those to build, not the damn login nodes.)
if you have a paid tier which offers more, you do you
if this is internally and you are a service provider to people.. why?
also 256 is not much today, my mac with a few things open is at 800
Compared to someone on an ssh connection. No desktop, no Apple Account, no user session programs, etc. You really don’t need much
i do not know why/in which context you are running this, and how frequently your users are executing forkbombs(i assume school/kids?)
my server is also running 400 something processes, one postgres instance alone is like 40?
(Also this isn't a default limit.)
Too low? 256 reads as *pretty* generous to me.
The users can develop on >100 compute nodes, but choose not to bother doing any kind of forwarding/proxying/jumping to them and just do stuff on the login nodes.
If they can't be bothered to do a "ssh -J …" then it's on them. The resources are there.
Quotas mean "more than this is clearly too much" not "please use this space".
In good times nobody minds, but in bad times when you just can't extend the drive you have to tell people off. Part of the job of making sure the system can continue to work for what you need it, under _real_ constraints.
You may have to delete stuff, you may have to shut the server down to save power. You may have to limit clock speeds. It depends on the environment and "it really should work because it should be covered by next day on site warranty and you could download more ram" often doesn't apply.
super tangential, but makes me think I never realised that "quota" can either be a lower or an upper bound depending on context
Ah, memories of Uni, back when storage was fairly expensive, where we had both hard and soft quotas. The soft quota would allow for temporary growth of build artefacts and things¹ but you would get stern emails if you were over your soft quota for 24 hours, and if you persisted without good reason³ your hard quota would be reduced so you effectively have no soft quota any more.
--------
[1] some machines had no local storage that the user could touch so putting them there was not always possible, some people on Windows machines had local storage but didn't have the relevant tools locally so were actually running things on the shared server(s)² instead of that just being a storage resource
[2] via telnet/rsh/rlogin: yes, I am that old… SSH was a thing by that point, though OpenSSH wasn't, and I was using it where available, but the use of older plain-text protocols was still far far more common
[3] it wasn't actually difficult to justify a quota extension for project work, in fact people enrolled on certain modules got higher quotas automatically
Don't worry, disk got expensive again. Disks are usually at least 2X more expensive than a year ago currently. Sometimes 3X more.
HDD prices are a bit more reasonable. Mostly because they are so heavy. Still insane to where prices were just 12 months ago
Probably why both soft and hard limits were developed.
In my ecosystem, developers don't have time to glad-hand like this; if there are issues with DEV_NODE_CFG_1_29875, we might talk about it over Slack (because I'm the first one to know there's a problem as the end-user), and if we can't sort it out they'll give me some time to backup, blank the whole machine, and I get a brand-new image of DEV_NODE_CFG_1_29875.
I can't even tell you off the top of my head what territory the physical machine is running in or whether it's the only dev-node on that hardware.
(Broadly speaking, I think Microsoft is assuming cattle ecosystems; they're not openly-hostile to pets per se, but they have a strict ranking of the priorities because the "cattle ranches," as it were, bring in more money).
Another frequent problem we had was people using the systems while others were. They wouldn't check the machine's status. And of course you can guess that I wasn't allowed to add a scheduler.
A lot of groups do things in janky ways. Often not because they don't know any better but because leadership doesn't and is assertive
It sounds like a lot, but it's fairly compact and really helps when you need to absolutely sure where you are and you have eight terminal windows open.
"I have not touched any machines in fra3 for a week, how the hell would I know of it? Why do we even have #fra3-maintenance and #maintenance channels then, if that's your stance?"
We put the MOTD up ≥7 days in advance and put it in relevant Slack channels.
We still get "Is Foo down?" the day of.
Like, I used to, but it was in the early 1990s.. People look at them now?
Next I'll be asking people to finger me to get my availability ..
Can still do it these days of course, but one with a seriously copious helping of ssh in the mix too ..
Trouble is, nobody else can do it. The only reason I have to use {social-media-blob} is because my friends don't know how to finger.
We've had the same issue with our local HPC; a few login nodes serving hundreds of users at a time, and each login node used to get swamped by these dangling SSH sessions/servers. They also wrote a script that shuts down all sessions once a day to save the login nodes.
But then again, I had people who would run jobs without checking if the machine is already in use. Obviously these people didn't check email or slack either...
It always felt to me more like a desperate attempt by Microsoft to regain the IDE market share as low end FOSS editors started taking over. So MS wanted to appeal to the open source community.
And it worked. Even if the primary build of VSCode which most people run isn’t technically open source.
The FOSS text editor space was always crowded between vim and emacs. FOSS alternatives would get measured against these 2 behemoths and it was a tall order to compete against. Now FOSS IDEs were a different story.
Java, C#, and C++ had many sophisticated and advanced IDEs, some FOSS some not. However, PHP, Python, Ruby, and JavaScript were quickly gaining huge mindshare and those developers didn’t want to install Visual Studio, Eclipse, NetBeans, or IntelliJ. It’s a tough proposition to tell a Ruby dev to install Java, then install Eclipse, then install an extension, then learn Eclipse nonesense, edit your Ruby files vs “just open SublimeText and edit your files”. Most of what those developers wanted was just syntax highlighting and basic directory navigation to begin with. Atom/vscode were an attempt at SublimeText alternative. VScode was objectively better than Atom and TypeScript was also objectively better than CoffeeScript.
The FOSS languages were taking over and there was less reliance on the traditional IDEs that MS (amongst others) were building.
MS wanted to appear more FOSS friendly so released VSCode.
So how to make the plugins remote-capable? You could write a massive virtualization layer that captures all system calls and forwards them to the remote - or, you could run the plugin on the remote and just pass the user commands and UI updates over the connection.
VSCode does the latter, so the nodejs runtime is where all the plugins are running on the remote.
(I understood the reason, I didn't say it was a good reason...)
Also, I realized that students are likely to use vscode to connect to embedded Linux machines and fill up their emmcs (and RAM), so I came up with a partial solution for our experiment's yocto image (https://github.com/RNO-G/meta-rno-g/blob/main/recipes-suppor...) , but a more general solution would be nice. Probably better to just not allow any node process to run via LSM.
At least when Python minimum version changed there was a separate extension forked from the original that I can use when I need to work on old code alongside the new extension for more modern code bases. Sadly, no such thing exists for remote editing that I know of.
https://github.com/jeanp413/open-remote-ssh
I run the editor (and its extensions), my projects and any agent harnesses from inside a container and use that extension to get an editor.
I have some shellscripts to quickly get in and out of the container. It works pretty well, I barely notice the container anymore.
This is mostly to protect my credentials and data from malicious extensions/dependencies/rogue-agents, bu it also lets me quickly port my dev environment to any machine (I use linux at home and macos at work). Just install podman, install VSCodium, add the SSH extension, build image, add my utility shellscripts and done.
Apparently Microsoft keep some VSCode APIs proprietary so only its own extensions can use it (allegedly for security reasons), which is why this specific extension only works in VSCodium.
Happy Ungoogled Chromium user though
How, specifically for VSCode?
I'm not surprised people trying to provide remote code editing to non-trusted clients are freaked out by it, but it is a great tool when operating in a trusted circle.
vs.
> The agent runs over port-forwarded SSH. It establishes a WebSockets connection back to your running VSCode front-end. The underlying protocol on that connection can: Wander around the filesystem; - Edit arbitrary files; Launch its own shell PTY processes; Persist itself.
So... basically the same things that Tramp could do as well?
> In security-world, there’s a name for tools that work this way. I won’t say it out loud, because that’s not fair to VSCode, but let’s just say the name is murid in nature.
Yeah, it's called RAT, and an ur-example of it is SSH itself (especially when allowed to run a shell remotely), so... not sure why are you freaking out.
I mean, I'd probably prefer if VS Code simply ran ed/vim remotely, but both of those editors can invoke shell anyhow so... eh?
Yes, just much faster. At the cost of a proprietary binary blob on the remote end, whereas tramp works with bare sh/bash.
However there's now tramp-rpc, which is tramp but talks to a Rust client on the remote side, and that makes Emacs even faster than vscode-over-ssh.
Some people get an icky feeling where remote editing tools change the remote filesystem in any way that is not explicitly done by the user. A binary installation of Node is extra extra icky. (I don’t have Node installed anywhere on my computers; I avoid JS if possible and if not I prefer Deno.)
Security Note Using Remote-SSH opens a connection between your local machine and the remote. Only use Remote-SSH to connect to secure remote machines that you trust and that are owned by a party whom you trust. A compromised remote could use the VS Code Remote connection to execute code on your local machine.
It's right there in the article, not sure what to say.
> # Security Note > > Using Remote-SSH opens a connection between your local machine and the remote. Only use Remote-SSH to connect to secure remote machines that you trust and that are owned by a party whom you trust. A compromised remote could use the VS Code Remote connection to execute code on your local machine.
From https://marketplace.visualstudio.com/items?itemName=ms-vscod...
Big bold text and everything
Separately, developer becomes a VSCode user. Risk as assessed based on them developing on their Dec machine, accepted.
Developer then opens a ticket to use LLM for a project. Uses VSCode SSH Agent. Agent creates exfil tooling on workstation and reverse shells their box. Agent reads unauthorised data which possibly has LLM subversion triggers, or deletes data, maybe opens connections to C2 service. LLM agent pivots to cloud inference and continues to lateral movement.
The assumptions of systems composition are violated by VSCode. That's why its SSH Agent needs to be blocked.
0: https://open-vsx.org/extension/jeanp413/open-remote-ssh
The problem is that it appears to do this via an AI Agent. This broadens the security concerns significantly.
I found this closing sentence utterly delightful, particularly in an age of endlessly filtering every piece of text I read on the internet through a mental "was this written by Claude, Codex, or (just possibly) a human?" filter.
kudos
What more could you ask for?
I'm not super low on space but it also pretty clearly doesn't need to be that large, it looks like most of it is copies of itself.
— https://marketplace.visualstudio.com/items?itemName=ms-vscod...
The issue is that the model can be the attacker, and use the link back to your host.
> [...] VSCode mounts a full-scale invasion: it runs a Bash snippet stager that downloads an agent, including a binary installation of Node. [...]
> It establishes a WebSockets connection back to your running VSCode front-end. The underlying protocol on that connection can:
- neovim
- in VS Code [1]
- in the browser
Cloud VM runs the built-in `code serve-web` command, over Tailscale using `tailscale serve`. No SSH. Extensions work. No extensions run locally (with SSH some extensions have to run locally on your machine).
I’m trying to see how little I can run on my local machine. VS Code over SSH is a good step in that direction, but there’s more attack surface if SSH is misconfigured, plus risk of an extension getting compromised.
I expected the added layers (neovim, through an extension, inside VS Code, over the web) to be slow, but so far it works surprisingly well.
[1] https://github.com/vscode-neovim/vscode-neovim
But Microsoft won't fix VSCode because it would possibly impede usability by amateurs, and inject tedious security concepts like boundaries into the dev process.
Wait, could someone clarify which machine is being referred to here?
So in the author's setup, he runs VSCode (i.e. the front-end) on his dev laptop, which he wants to keep free of direct LLM access.
VSCode connects via ssh to a dedicated "sandbox" machine on which the LLM will be free to do whatever it wants (mostly).
VSCode realizes this the Microsoft way, by using the ssh connection to install VSCode Server on the sandbox machine - the "backend" - and communicating through it via a websocket connection.
So then, what happens? If the websocket connection allows the front-end to run arbitrary commands on the sandbox machine, this wouldn't be very exciting: The front-end already has an ssh connection and a massive server process that can do the same - and the entire purpose of the sandbox machine is to run arbitrary, untrusted commands without harm.
But the article says the websocket connection goes "back to your running VSCode front-end". So does that mean things are reversed? I.e. the agent/harness runs in the server on the sandox machine but for some reason has this websocket connection that also lets it run arbitrary commands on the dev laptop?
Is that it? That would be truly insane!
I'm not the author, but I can tell that he means the remote machine. His sentence about being "nervous about letting people VSCode-remote-edit stuff on dev servers, and apoplectic if that happened during an incident on something in production" makes it clear. He considers the VS Code agent to have all the features of a rootkit, and doesn't want anyone to be deploying it onto dev servers, let alone production ones.
Yeah, I'd be apopleptic too. Because the clueless dev who did that just escalated the production incident into "track down every credential that was present on that particular dev machine, and assume those credentials are now compromised and have to be rotated".
It's astonishing how many commentors(not you!) either didn't read or didn't understand the blog post, saw "agent" and thought LLM agent, and then decided to comment about it...
EDIT: Found it. He's referring to the acronym RAT meaning "Remote Access Trojan" (or, if actually intended, Remote Administration Tool).
The issue the article highlights is this opens you to local code execution initiated by the remote.
Here is the security notice on the official SSH extension https://marketplace.visualstudio.com/items?itemName=ms-vscod...
> Using Remote-SSH opens a connection between your local machine and the remote. Only use Remote-SSH to connect to secure remote machines that you trust and that are owned by a party whom you trust. A compromised remote could use the VS Code Remote connection to execute code on your local machine.
Yes this article is from Feb 2025, but NOTHING HAS CHANGED in this regard. Except perhaps now MANY MORE PEOPLE are believing they are safe using VM and other remote SSH "sandboxes" to develop on via VSCode remote host connection over SSH. The reality is, as stated by Microsoft themselves, if you can't trust the remote.. It can own your local computer. And that's not much of a "sandbox"..