Ask HN: Best practices for editing remote code locally?

105 points by E_Evan ↗ HN
Working on a remote codebase from a local computer accessible via ssh, what’s the best/easiest way to write, edit and test code?

181 comments

[ 6.4 ms ] story [ 238 ms ] thread
What is wrong with using ssh -X and the do the same workflow you would do if you were on premise?
sometimes it is just too slow or laggy. some software even refuses to run at all that way.
Because we don't have gigabit symmetrical lines everywhere.
I've done X forwarding over slow wifi and it worked great. OP isn't editing video.
I'm doing X forwarding for 15 years and unless you have ~20mbps symmetrical lines, you can't do anything productive.

However, if you absolutely must, use X2go.

I certainly haven't used it that much, but I never felt like the connection quality was a huge concern for a usable DE. It's something like 30 years old now isn't it?
The thing with X11Forwarding is, it's never designed to be ran over WAN or, slower connections to be precise. It carries X11 commands, not the video itself and is synchronous and doesn't compress anything. As a result it both needs low latency and saturates connections quickly.

When the updates are occasional, fine, but when you call make or run something which updates fast/creates a lot of output, things get hairy, quick. Even on local networks with a couple of connections. Been there, tried that, experienced it all.

X2Go transmits the image, compresses it, and can work with much slower and higher latency network connections. It can also resume sessions.

It depends a lot on the application. Old apps that use old X fonts etc. work okay-ish, but by default a lot of modern apps will be pushing too much over the connection because they'll be doing the font rendering into a buffer.
local emacs + TRAMP works really nice
TRAMP has been too slow for me, and I've never had a reliable LSP session over it. Tried both eglot and lsp-mode. eglot is the better of the two for this.

I ended up creating a "runtime" script that first rsync all the files over to my remote machine, then run whatever arguments, `runtime <host> 'make'`. there are some flags too like `-s` to have it not rsync, etc. This works very well. But then this is maybe cheating since my code is still local. The reason for this is my M1 mac can't build our gigantic C++ code base for a whole host of various insane reasons.

I've had good luck with Tramp+Aquamacs myself. There might be a short delay at save time, but I can live with that.
An important tool to find or write is one which will automatically sync your changes made locally (from the comfort of your IDE of choice) to the remote machine every time you save it.

You want to be able to iterate quickly and not have to manually run a command every time you want to sync changes. It's also a huge help to tie this into a way to run the unit tests for files which have been changed so you can get feedback and quickly as possible.

Ideally, everything should be made so automatic that you barely even notice the code isn't running on your machine.

For this i combine entr and rsync
local code editing + rsync command that you run manually or automatically(e.g. using fswatch)
SSH in and use a terminal editor. Or, create an SMB/CIFS share and mount it locally.
I am used to work with "remote - SSH" vs-code extension with a remote server and it work like a charm, you can't tell that you are not at home. I do all my coding this way BTW
Can you elaborate on the situation? Personally I’d just keep a local copy of the code and then use git or rsync to sync the remote system to my local version whenever I’m ready to.

If it’s just small things like a quick config change or something, I’d personally just ssh in and modify with vim.

For testing, you probably shouldn’t be doing that on a production system, perhaps you could stick the code in a docker container and run the tests locally? If you’re familiar with docker, that shouldn’t take more than a couple hours to set up, and likely much less time than that if the project isn’t super complex.

Feel free to explain your use case further and I bet people could give more targeted advice!

(comment deleted)
Since you seem to know what you’re talking about, I’m looking for some advice: I work for a large and very well known engineering company and currently our incredibly terrible workflow looks something like this: Develop locally using VStudio -> transfer repo to network drive -> login to tester via three layers of Remote Desktop Connections -> run and debug code live on tester RDC The tester setup is virtually all GUI based. It’s also located on the other side of Asia which is the reason of logging in via RDC. Is there possibly a better way of making this workflow more contained in a terminal/single ide environment, without having to jump through the RDC loops?
I'm no networking wizard so this isn't really my area of expertise but I have a few thoughts and questions. Why do you need three layers of RDC? Is this due to networking constraints, where the ultimate server (I'll call it the prod server) you are trying to reach is unreachable from the internet directly? You might be able to tunnel into the server using ssh tunnels (see this guide [1]) through an intermediate server which your computer and the prod server both have access to, and then you'd be able to directly scp or rsync files to the server, as well as run commands over ssh, but this would be increasing your attack exposure from a security perspective, which folks may not be happy about since you're at a large engineering company.

A few questions that would help me give more actionable advice: What is the "tester" you're using? Is it essential to interact with it via a GUI or does it have a command line interface you would prefer to use? Is the OS Linux or Windows? What are the the three layers of RDC?

Feel free to email me at my hn username @gmail.com to discuss further since this is getting pretty specific to your setup! Sounds like a fun problem to try and solve :)

I personally prefer Eclipse Remote Development Tools, which can either work via a dedicated daemon, or SSH.
git?

If it's impossible to clone the environment for some reason, VS Code and JetBrains IDEs have remote support. Even vim can edit remote files.

It's not clear from your question what exactly is your problem, would you be willing to elaborate?

Since I like my local development environment, I mostly use `sshfs`.

It mounts a remote directory onto a local directory and lets you access remote files as if they're local, with slight latency. Then you can use whatever tools you use locally to edit/compile/run the software.

I also use this setup. Works well for editing and reading the files. But be aware that some file-intensive operations don't work as well. It is better to grep or git in the remore server instead of in the local mount. Be careful if your shell prompt is configured to show git status, because that won't be instantaneous.
sshfs works for only 10 files or less project.
How so?
Never do find in files or run package managers?

JetBrains never works on sshfs as it parses files for dependencies.

Actually, you're right. I just tested recursively grepping a remote directory with a thousand 16KiB files. The speed is completely catastrophic:

    $ time grep -r foo test
    ...
    real    7m36,154s
    user    0m0,110s
    sys     0m0,249s
The average download speed from remote host is ~2MiB/s, so it isn't a network issue - a thousand 16KiB files sum up to 16MiB so it shouldn't take more than ten seconds. Seven minutes is terrible.

I don't understand why is this happening. Is this an inherent limitation of remote filesystems or is sshfs implementation just sucky?

EDIT: parallelization seems to help - running 100 concurrent reads at a time uses up to ~1MiB bandwidth. I suppose ordinary unix tools don't play well with remote filesystems because they do stuff sequentially.

That being said, if JetBrains is struggling with SSHFS, then they really need to fix their file browsing code.

I think it's because it's not simply a file download but instead keeps on asking for file content or meta data making a roundtrip on each files to cause excessive latencies against many files.

All the tools are basically designed to work on local files and they tend to work bad interacting directly on mounted file systems.

Visual Studio Code Remote SSH? https://code.visualstudio.com/docs/remote/ssh it will however install VS Code Server on target machine. So, for permanent development it is good. Not great if you only want to change one config file in PROD environment.

> The Visual Studio Code Remote - SSH extension allows you to open a remote folder on any remote machine, virtual machine, or container with a running SSH server and take full advantage of VS Code's feature set. Once connected to a server, you can interact with files and folders anywhere on the remote filesystem.

Once connected, even terminal runs on remote host. Moreover you get to forward ports from remote host to local using VSCode UI.

It also do a lot of other stuff, for instance, if you have a python linter that need the python binary to check code, then you can specify the python path of the remote machine and vs-code will use it for linting, work also with other langages like C.

Also you can use embeded terminal and type "code myfile.txt" and it will open the file.

If used with windows, it also automaticaly remap localhost to the machine if you are launching a local server with your terminal (can be annoying tho)

Basicaly it try to do everything to make you feel that you aren't on a remote ssh connexion and it work really well

I have recently started using this for a project, I’m on a Mac and the legacy codebase only supports CentOS7… I have a VM setup on my machine and use VS Code to “remotely” develop on the VM. Works really well!
I'm experimenting with using this VSCode feature to edit code on a Spark master, with the code storage on an EFS volume. So far, this seems to be allowing me to have a local-feeling environment, with a well-factored codebase that I can reference from a Jupyter notebook (inside VSCode), that has high-bandwidth and low-latency access to our data repositories.

I can also choose to temporarily vertically scale the remote host if I want to run single-node operations (ie Pandas).

- My preference is to run my IDE or editor on the remote, and only run a thin client (RDP > X11, mosh > ssh) on the local. Because I’m happy in nvim, I would just install nvim and my dot files on the remote and access it via mosh/tmux. But this is not for everyone; if you’re doing Java an IDE is a must. So, try RDP or X11 forwarding for those.

If you can’t install software on the remote (or you can’t install a GUI stack), read on:

- Most IDEs like Intellij can be configured to do all their execution on a remote over SSH. I worked this way using PyCharm for my student job at UC Berkeley in 2012 and it worked great.

- SSHFS is okay but not great because of slowness. It’s fine for editing but sucks for running interpreted code locally. So, try to do all execution on the remote.

- VS Code has handy support of editing on a remote without needing SSHFS

- if permitted, a two-way file sync solution like Unison or Mutagen gets you native file system performance on the local. We used Unison for local editing with remote execution for Airbnb’s Ruby codebase for several years (2015-2018+). But it’s more fiddly to work with than SSHFS.

SSHFS caches reads on my machine. Here's a sequential timing of sha256sum operation on a ~10MiB file:

    $ time sha256sum file
    c2359f8b63dfef9f50e62b2e7b4ef2613c290c3a1ffa135cfa255b14eafd3ffb  file

    real    0m5,205s
    user    0m0,127s
    sys     0m0,011s
    $ time sha256sum file
    c2359f8b63dfef9f50e62b2e7b4ef2613c290c3a1ffa135cfa255b14eafd3ffb  file

    real    0m0,221s
    user    0m0,093s
    sys     0m0,004s
That should be enough to prevent slowness for interpreted code, since textual files are small anyway. SSHFS only needs to send edited files to the remote machine.
No one wants to open up project files over sshfs...

Do a search on all files and you'll need a break.

Code parsing definitely can take forever using JetBrains.

Depends on the project size.

My average project size ~12MiB - 19MiB when using vendor/ directory with Golang. It's less than a minute wait on a 1MiB connection to load the whole directory and cache it.

For compiled languages, binary size would be a problem so it's probably smart to write the compilation output somewhere outside sshfs-mounted directory and run it locally. For interpreted languages generally it shouldn't be a problem since everything is in memory. Python does write stuff in __pycache__/, but in my experience it's rarely more than a few hundred kilobytes per cache directory; it's also possible to turn off writing __pycache__/ with PYTHONDONTWRITEBYTECODE=1.

i use emacs+TRAMP for vcs, mosh for remote terminal presence, and vscode+ssh or jetbrains gateway for dev.
I have recently built a dev server for myself using https://github.com/coder/code-server (VS Code in the browser, hosted on the remote server). With very little effort, you get an always-on dev environment with built in terminal so you don't even have to ssh in if you don't feel like it (if you're happy working within the editor's terminal window pane).

As a bonus, I also setup openvnc, cloudflared (DNS over HTTPS), and pihole. And tmux and mosh of course.

With this setup I can even do my work from an iPad (once you get the VPN setup correctly).

It's nice to be able to reboot my laptop without losing my place in my work.

Ideally, just use a text editor on the remote machine such as Vim, Nano, or Emacs.

If you want a full IDE, or just your set of extensions & tools, you could install it there and use X11 forwarding like so:

    ssh -X me@box /usr/bin/code
I believe VSCode also has a remote mode, but I haven't used it.
X11 forwarding was not a great experience last I tried it.

Every now and then clipboard would stop working or resized windows would retain their original dimensions etc. Not entirely sure if it had to do with my setup or network.

I switched to tmux+neovim a year ago and it has been a much better experience.

Many editors support the rmate protocol, enabling you to open a remote file from a ssh session in your local editor.
I've seen some of my colleagues running 'Visual Studio Code remote' [1] which allowed them to edit source code on a remote machine with a native text editor; run the code and tests remotely with a local interactive debugger, and so on.

I haven't tried it myself, but they seemed to like it.

[1] https://code.visualstudio.com/docs/remote/ssh

VS Code + The Remote SSH extension is a remarkable solution to developing code (not just editing!) on a remote machine.

When Covid hit and my main work machine was still a desktop, I worked from my home using VS Code and the remote plugin. This lasted for 18 months, and the only time I ever had an issue was due to a power outage in the office. Data, code (everything, really) lived on my work desktop that was sat in an empty office while I was at home.

There's no latency, the terminal opens as if you were on the remote machine, and ports are automatically forwarded. It runs code, tests, etc. all on the remote machine.

If there's a better solution out there it's almost certainly down to IDE preference.

Seconded. It is so seemless that it is literally possible to not notice which machine you are on if you are just firing up something new for quick exploration.
I use VS code with the same plugin for working remotely. I also use jupyter lab with port tunneling and there's no difference to what it was like working locally. My company issued computer is just a VM off in some cloud. Working like this has a oddly retro-futuristic feel to it. Like we've gone back to the days of mainframe development.
Just a suggestion - VS Code (now) has a super nice support for jupyter. You don't have to start the server externally, vim mode and all your extensions like github copilot work fine, proper autocompleting works, etc. Debugging is absolutely amazing. I've recently switched myself and am never going back to browser-based jupyter.
Debugging jupyter notebooks, as in you can set a breakpoint then shift-enter to run a cell and hit it? I'll have to give that a try!
E.g., you can set up a breakpoint in your own local module that your notebook is importing and get the full vscode debugging experience by executing a cell.
Relatedly, iTerm's tmux integration[1] pairs very well with this setup.

[1]: https://iterm2.com/documentation-tmux-integration.html

This is the best solution I have found so far being a long time vim user. ssh + tmux + vim is all you need. Least amount of potential problems in this setup. vim runs locally on the server where "remote development" is being done, so there is very little friction in terms of getting it to do what you'd like it to do given all the plugins you can add to it for beefing it up.

I have heard stories from coworkers about work being lost when editing is done remotely instead of locally, so never ventured that way.

I'm definitely team emacs -nw, but I use the same tmux setup. Tmux's ability to reconnect to a session after disconnect is definitely it's killer feature.
Exactly what I was thinking, it really is crazy nice for development over SSH or within a container. The Live additions are neat for sharing as well though I've only dabbled in it.
And you can run the debugger!
I started with this and still prefer it from a functionality stand point but the one use case where it falls apart for me is when you have an internet connection with occasional lag spikes.

This could be a variety of scenarios like working remote on bad ISPs or cellular or a crappy VPN but all that matters is when it acts up it will cause lag, or dropped connections depending on severity. Too many interruptions while trying to focus.

For this reason I keep everything local and use an rsync wrapper over my build system that streams output back. The added delay for each command disrupts workflow less than it happening during editing.

How does it handle disconnects? With two kids and lots of interruptions my laptop is going to sleep a lot if I don’t have it plugged in between sessions, does it reconnect everything automatically so you wouldn’t notice it much?
depends on length of the disconnect. it can be seamless and if it fails, it asks you to reload itself, conveniently placing a button to do just that in the same dialog window.
Make sure to add TCPKeepAlive yes into your ssh config so the ssh doesn’t disconnect. Vscode will disconnect but then it’s just a matter of clicking retry
I recently registered for a free VPS from Oracle and could get a box in my country - then those setups start working well.

With my VPS from DigitalOcean the ping will be 170ms and then its no bueno.

Since you seem to know what you’re talking about, I’m looking for some advice: I work for a large and very well known engineering company and currently our incredibly terrible workflow looks something like this: Develop locally using VStudio -> transfer repo to network drive -> login to tester via three layers of Remote Desktop Connections -> run and debug code live on tester RDC The tester setup is virtually all GUI based. It’s also located on the other side of Asia which is the reason of logging in via RDC. Is there possibly a better way of making this workflow more contained in a terminal/single ide environment, without having to jump through the RDC loops?
I had similar experiences but some layers are Citrix and others RDC. Seeing your situation brought back my anxiety on those connections. Sorry for this. Is there any terminal servers to be able to ssh port forwarding? Thus to reduce layers, while as the test setup is gui based at least one layer of RDC is required. Reducing nested Remote Desktop would for sure help with the experience. Sorry again for you situation.
Is there a way to do your testing without a gui? I have no idea what your testing setup requires, but is there some technical reason why tests are not done automatically in your ci/cd pipeline?

If it must be done manually, can you just ssh/rdp directly into the testing box? The network where the testing box exists could use some tunneling protocol (stunnel, socks, wireguard, vpc peering...) to make it more accessible from your dev machine or your company VPN.

Could this be used to do firmware development on Pi? One thing I’ve always hated was having to either SSH and use vim, or login directly to the Pi and use an IDE directly on it.

I would love to use my main mac + VS Code to edit code and dispatch commands like compile / run

I've done this exact setup (though not for firmware development). VS Code C/C++ extension also supports arm64 as well (https://devblogs.microsoft.com/cppblog/visual-studio-code-c-...). This works well with remote ssh.
Thank you! I'm going to give it a try, sounds like it could be perfect for me to finally get deeper into firmware development. Just can't stand working directly on the Pi.
VSCode install processes to support remote editing. I think PI is supported, but I know it had issues with certain architectures last year. Its very easy to try though.
It is, I have used it and it works great on a pi3.
I used this setup a lot myself and it worked great except that name completions, intellisense didn't work as expected.
You might need to install all the language servers and extensions on the remote host too. Typically VS code realizes they're missing and prompts to install them. But if there's an issue you might need to dig around and see why it can't install the LSP services there (perhaps you don't have permission or access to get them).
I like vscode + remote ssh a lot, but one thing to be aware of is that the node server that it installs on the remote can be a bit memory hungry if you're on a small machine (i.e., low-end droplet or vps). One of the rsync or sftp remote adapters is much nicer to such environments.
switched to vscode from neovim in tmux a couple years ago, which used to be my weapon of choice for more than a decade (vim before neovim for the purists). practically haven't looked back. it really is amazing.
Nova (https://nova.app) has a remote file browser + terminal that I've found work surprisingly well, it has the advantage of being Mac native as well!
Thank you for sharing this. I will give the 30-day trial a shot. I find VSCode just fits my needs on my boxes (Mac, Win, Linux) but I am always open to try stuff.
Nova is nice, but its selection of plugins and integrations (things like linters, code syntax highlighting for specific languages etc) is just vastly behind VSCode. Mac native is nice to have, but the feature set in VSCode is so hard for a small competitor like Panic to beat now, especially when Nova is relatively exspensive.

If you are purely doing web design work, I could maybe recommend Nova for its nice CSS/HTML tooling, but again you can recreate much of this with VSCode anyway. When you factor in Nova is 99 dollars this conversation becomes a lot more difficult when VSCode is such a strong free competitor.

Nova has been through several iterations/product resets, I owned it back when it was called "Coda" too. VSCode by comparison has been relatively stable and its ecosystem a lot healthier.

I'd actually recommend VSCode + another Panic Software app for remote file access on a Mac: Transmit. Transmit is mac native standalone remote filesystem access tool, its basically the tool they built into Nova as a standalone, and 45 bucks:

https://panic.com/transmit/

VSCode + Transmit is IMO the best in class tools on MacOS for this kind of remote access work, although you can easily substitute Transmit for a free tool like CyberDuck or similar too and keep software cost at zero.

Vscode being free, I suppose the only real contenders are Sublime Text or JetBrains. Everything else either free or paid seem like a non option.
If you are on a Mac, using Transmit (https://panic.com/transmit/) you can mount a SSH/SFTP connection as a local disk and edit there in your editor of choice.
Do people who work over sshfs never search for a string over all files? Or your editor wants to parse codes from all files to find dependencies etc? Save on upload of a local copy is far superior.
Couple of solutions (some already mentioned):

- SSH remote editing (e.g. ssh + remote Emacs/Vim)

- VS Code + Remote SSH

- Emacs TRAMP

- JetBrains Gateway

- mutagen? (I haven't played with it, I'm not sure if it's suitable for your case)

Old unix tools like vi or emacs and "make" were built when nearly all computers were remote, and work really well for this. Especially when combined with something like "screen" to preserve multiple sessions..
I use atom and remote ftp plugin. With this setup, I can connect to my dev server with my local machines.

1 machine is Macbook air at home, another is windows in the office

Best practice is to write, test, build etc. locally using a version control system and deploy a tested bundle to the remote computer when ready.

Editing code directly on a remote server is something I used to do in the 90s before git and other tools made good practices much easier.

Oh, I love your response. Thank you.

My workflow was as follows

- Remote into server and develop from there

- Git to different location, could even be same server

- Occasionally pull on local machine for redundant backup

This allowed me to

- Segregate builds for my testing and the business's UAT

- Allowed me to always have a stable build locally that I could look at from home

Before, I did develop on my local machine and push to a server. Some part of me still thinks that's the proper way to do it.

But I am always seeing what works best for the current situation and developing directly on my server served me well once upon a time.

Best practice changes rapidly, local dev environments have proven to be an endless pain in the ass. Apple M1 silicon is the final nail.

The next 10 years of “best practices” will return to dumb clients and dynamically provisioned dev environments. It’s simply so much easier.

As someone who has responsibility over how shitty the local development experience is for my org, I recently saw GitHub Workspaces and am curious if that could be a viable option.
I've been using GH Codespaces for a side project when on the go for a while now. When I'm travelling or recently when moving house I've been able to use my terrible but light weight laptop with vscode + gh codespaces. It's been a complete dream! When running the same project locally on the laptop it's slow and makes development a real pain but codespaces has given the laptop a new lease of life.

I'm not sure if the price lines up well enough for a daily-drive for a company, but if I were managing dev environments for an org these days I'd probably jump to something similar. Perhaps something like a decent size EC2 on AWS where devs can tear it down and spin it up again to start from scratch within a couple minutes.

Throw away dev environments feels like a game changer having used it for the past 6 months on-the-go.

It sounds like the exact same thing but with added latency. I hate latency.

It baffles me why anyone would choose an incompatible architecture only to run dumb clients on it. You might as well just save your money and run the dumb clients on 20 year old hardware.

Or you could spend less money on a laptop that is actually compatible with whatever platform you are deploying to and enjoy low-latency local development.

I’d try VSCode with Remote SSH. Maybe it’s because I’m 10ms from my datacenter but I experience no lag and things have always felt just as speedy while on random public wifi.