Hey HN, my name is Erik. This is my second Show HN for Booste – looking forward to your thoughts.
Booste is a command-line interface (CLI) that takes code written on your local editor/IDE and executes it in EC2. It runs remotely, in a pre-configured environment, but works with your local tools thanks to a background filesync process.
I want to fix the “it works on my machine” issue by bypassing personal machines entirely. As if Docker and Heroku had a lovechild.
How I came to building Booste:
Being a mechanical engineer turned software dev, there were countless times where my laptop (glorified tablet) couldn’t handle the workload needed, so I made Booste v1: a tool to host full desktop apps on AWS and stream it over RDP (remote desktop protocol) to clients. Some may recall the Show HN of Booste v1 six months ago.
Booste v1 sucked. Unusable latency, crap unit economics, and an infrequent pain point.
Through it all, the most common customer request was for a hosted VS Code. Prospective users wanted the perks of a cloud IDE (namely no-setup environments) but wanted to use their own tools. Diving deeper, it seems that most large companies build out their own (usually buggy) version of remote “devboxes”. The need seemed consistent and clear.
After the first Show HN and a failed W20 interview, I pivoted. I gutted the graphical desktop component, zoomed in on developers, and rebuilt as a CLI: Booste v2.
Running any shell command proceeded by the world ‘booste’ will trigger the CLI. The CLI tunnels that command through SSH to EC2, executes it there, and streams the stdout back down to your command line. Filesync makes sure EC2 is ready with your code.
Where “npm start” would run an app on localhost, “booste npm start” runs the same app in the cloud, visible to your team by URL. You can edit the code locally (VS Code, Sublime, Vim) and the save is synced and hot-swapped on the live URL within a second.
How the environment sharing works:
- The creator of a “Codebox” (EC2 instance) gets a barebones Ubuntu VM, on which they can set environment variables, install interpreters and packages, or set up databases. (pre-built env templates coming soon).
- Team members join the codebox by ID and password with a single “booste join” command.
- Codebox members get repos placed next to each other on the directory tree, so code doesn’t clash.
- There is “booste clone” feature in the works, where you can clone someone’s config without actually entering and messing with their codebox.
Under the hood:
-The CLI is built in Python using the Click library
-Codebox infrastructure is automated with the AWS SDK.
---Currently, they are EC2 instances, for the sake of getting the beta live. I plan to migrate to a more flexible containerized setup in the near future. Containers make much more sense.
-I built a Dropbox-like filesync, inspired by the rsync.
---It monitors a chosen directory on your local machine and syncs it with the codebox.
---When files in that directory are added, modified, or removed, the codebox reflects this within a second. Only differentials are sent.
---All communication is encrypted with frequently cycled keys, unlike rsync.
---Bulky files (ML data, dependency packages), which would otherwise slow rsync to a crawl are ignored by a customizable .boosteignore file.
-Since files sync asynchronously (on edit), the only latency is from ssh. There’s a 0.5 - 1 second delay on booste commands.
I’ve moving from closed to open beta. Fueled by covid and the now mandatory remote development teams, I made a free tier for you.
Give it a try! Let me know what you like. Let me know what you hate.
A few things come to mind on how this could be really useful:
- not having to install things like Docker and Postgres on my own machine, save space, not having to get a computer with as much space in the first place
- not having the heaving lifting on my machine means that I won't get the whirring fan and heat from my laptop
- not having to deal with managing different versions of node on my machine for different projects
- developer onboarding could be faster. There were a lot of little details to get up and running. Getting the npmrc token on my machine so I can install the private packages etc. etc. These could just live in the cloned cloud box that every new developer gets.
- lower the barrier of entry for new developers to just start doing something fun and useful. First day of bootcamp was this install-fest where instructor had to babysit everyone to make sure they got everything configured right. It's probably good for people to learn that stuff eventually but it would help to be able to just jump right in.
One downside: if the wifi is down you won't get able to get anything done at all.
My go-to cloud IDE so far has been Stackblitz. I use it to do little proof of concept stuff, share the link with people. But I prefer working in my own text editor. And you can't do anything more than a simple throwaway project in stackblitz.
I have yet to see what the usage will be, between having one living "shared" environment, or multiple "cloned" environments.
Wifi is an unfortunate issue. Probably a mid-term feature, but one of my early beta users suggested an "offline" version of booste where we clone images downward to the laptop, for use during internet downtimes. Could be a move.
Hmm. I like the idea of making development box setup easier with tooling but this seems less flexible than just doing a vagrant up with an Ubuntu config, or using docker desktop, and costs money. Do people have dev machines that don’t have like 10 gb free to run a VM? Maybe I’m missing something.
You're right on this. I had figured that this solution was solved too, until I heard about how frequently Booste is built internally at growing companies. The tracking and distribution of images seems to be the unsolved problem, and centralizing those on a cloud platform could be the move.
> ---Bulky files (ML data, dependency packages), which would otherwise slow rsync to a crawl are ignored by a customizable .boosteignore file.
Did you know rsync supports providing exclusion filters in files? See `--exclude-from` and `--cvs-exclude`. The latter uses .cvsignore files just like git uses .gitignore (and, I assume, you use .boosteignore).
I'm not trying to be snarky, but I genuinely feel like this could be implemented in less than 50 lines of code, maybe even 0 lines of code. You could just use bare ssh and rsync. Or you could use a remote volume mount. Or you could configure your docker client to connect to a remote docker server. There's a million different ways to do this and none of them are difficult, especially since this is targeted at engineers. I feel like I'm missing the point.
This sort of thing happens often enough that I would love to see some sort of list of various SaaS products and how to build their equivalent DIY solutions with standard open source programs.
The classic build or buy. It seems we'd be one of the first SaaS products in the space, but I've heard of a few open-source projects as well.
Is there any site that ties open-source programs to their equivalent SaaS product? That would be very interesting to look at, to see what went into building these various SaaS products.
No snark taken :) your comment is definitely in good nature and is appreciated.
Sounds easy to implement, and many companies have attempted to do so. Generally, the solutions you've suggested are what they turn to, with mixed results. Rsync bogs down, bare ssh doesn't let you use gui editors, and the users I've talked to so far consider docker to be opaque.
Folks here on HN are strong engineers. They're happy working with current tools, because they've already gone through the ringer to learn them.
My assertion is that this skillset shouldn't need to be learned. When I write code, I don't need to choose which CPU core is used, and I'm grateful for that abstraction. I'd like Booste to abstract cloud ops in the same way.
Unless you've managed to write a better sync tool than rsync, your product will suffer more sync issues.
The point about GUI editors not supporting remote development is basically false. VSCode, GVim, IntelliJ, Eclipse, Visual Studio, Sublime Text all support remote dev.
As far as Docker's opacity, I guess that's a fair point, although in my opinion it's kind of unacceptable these days to not understand Docker or, if not Docker, then containerization in general, as a software engineer.
The Booste implementation of rsync is definitely not indestructible, especially during this beta. Been a pain in my butt, but making progress.
Main improvements over rsync:
- .boosteignore. Just like a .gitignore, a .boosteignore helps filesync filter past bulky files that may not need to be monitored and uploaded every time.
- end-to-end encryption
Admitted flaws of our current algorithm:
It's built pretty ground-up in python. Haven't speed tested against rsync, but using it so far, I've never managed to edit code and run it through booste before the sync made it through to remote. IE the ssh tunnel doesn't beat rsync, mostly because files sync asynchronously. That said, I plan to do port it to a compiled language to cut time.
booste looks nice, though i'm definitely not the target audience and usually take issues with developers not wanting to figure out why something they've build only works on their pc. it's usually because of subtle bugs and non-resilient software design, which is a bad idea to push to production... but this product thankfully does much more than enable that kind of behavior :)
but thats besides the point, i only wanted to nitpick a little:
> Main improvements over rsync: - .boosteignore. Just like a .gitignore, a .boosteignore helps filesync filter past bulky files that may not need to be monitored and uploaded every time. - end-to-end encryption
thats a verly little improvement over rsync, considering it already has a file-based blacklist feature as well and is usually used over ssh, which is also encrypted ;)
though it (to my knowledge) doesnt behave like a gitignore, so whitelisting files inside of a blacklist is afaik not possible.
alias rsy(){
if [ -f ~/.rsync_ignore ]; then
rsync --exclude-from="~/.rsync_ignore" $*
fi
rsync $*
}
Good to know on Rsync - my understanding was that it was unencrypted, which is true, but almost always is used in conjunction with SSH. You're right :)
I'll definitely play around with this alias and see how it performs relative to .boosteignore
Not saying that this is what's happening here, but it's just so amusing to me how much this sounds like the response to the original dropbox post on HN.
Still experimenting with how to pitch it. Dropbox for devs makes sense, knowing the details, but would be super vague for anyone who doesn't already know anything about booste.
Other pitches that fall into that "short but vague" pitch.
For years the ssh type lag made a remote editor too painful. Now we have things like mosh (with local echo) or syncthing or even native vim scp url support. What makes this better? I can’t tell from
the website.
I guess one issue you’ll run into then is that VS Code supports something very similar. Instead of editing locally and syncing, with VS Code, you can run VS Code “locally”, but it can connect to a remote server over SSH where you edit files directly on the remote server and it gives you terminal access to run your code there too.
I’m not saying your idea isn’t viable, but there are multiple players moving in this direction.
To me, the main thing you’ll have to convince me of is that this is better than editing code locally, but running the code in a local Docker container. For me, this solves the issue of “it runs on my machine”.
From my perspective, your main value add is trying to make executing and testing code on a remote server frictionless (it’s not quite there, but pretty good so far). And that you can edit on a low-resource machine, but effectively “develop” on a more powerful remote system. But these might be a tough sell if you’re targeting the HN crowd (which I’m not sure you are).
Thanks for the tip. I've heard mentions of the VS Code tool, but never had it clearly explained until now. Definitely something to watch out for. The "pulling" aspect seems more elegant than the way Booste stores locally and pushes, so I can definitely take that as inspiration.
The docker point is fair too. Docker containers solve the machine problem, and are incredibly helpful teck. There are perks of cloud hosting (visibility by team, potential to monitor prod and have the codebox reflect it, performant hardware) that we can layer on top of docker container tech. Just hypotheses, to be clear. But to me, we're doing to Docker what Google Docs did to MS Word.
Are you aware that you can set DOCKER_HOST to something like `ssh://root@example.com` and then run git/editor/docker locally, and the docker client will invoke `ssh` to connect to the remote host and access the socket/daemon? The builds all happen remotely, on big/fast boxes.
Coupled along with something like docker-machine for instantiating those remote instances, what does this setup not offer that yours does?
Hi Erik. I don't have the "it works on my machine" problem often, because I mostly work on my own or with very few other people, so I may not be your target audience.
I do have the "my laptop's hardware is not enough" problem, and my current approach is simply spinning up an EC2 instance with the desired resources and ssh into it and then cloning the git repo I'm working on.
Normally, I set up the instance once (I have to add the ssh keys to my github account), and (if it's an expensive instance) start/stop it so I don't leave it running over night. Not a big pain point as of how I currently work.
I could see myself using Booste if it allowed me to, with a single command, spin up and sync bigger instances that would run for a short period of time (eg, for ML model training) and then intelligently stopping/terminating it. Also, if you integrate more than one cloud provider, I could switch to the cloud provider best fit for my needs (cost, spin up time, latency, cpu/gpu, etc).
A couple thoughts on the current implementation (you might be already addressing):
- I generally have more than one ongoing project at a time. Having everything in ~/Booste sync'ed with every box doesn't work for me (aside from the fact I'd use a different location)
- I sometimes do need to work with big data files, and even if I don't want them syncing all the time, I do want a way to retrieve them easily.
- I also often work with jupyter notebooks, so I'd need a way to tunnel (like I do with ssh)
Super helpful, thanks for touching on what does and doesn't have an immediate need for you.
re: ~/Booste directory, just pushed and update yesterday that allows you to move your filesync directory around to wherever you want. It's not quite perfect yet, as I haven't built a way to clean old project files out of the codebox. But your new repo will overwrite anything that'd get int the way, so functionally, you're good to go there.
There's a "booste upload bulk" command, which takes a .boosteignored "bulk/" directory, for hefting code up to the codebox without bogging down filesync. Haven't get built a "booste download bulk" command, since customer pull recently has been more web-dev oriented, but I'll get on that once more ML folks play with Booste.
Codeboxes have auto-spindown. Default four hours live, but could make a command to monitor processes and spindown once done. Like a flag "booste --stop-when-done python3 train.py"
Nothing for Jupiter notebooks yet, and it's frankly going to be a tough technical nut to crack. Interested in exploring though.
(Edit: added below)
As for cloud flexibility, I'm sure more enterprise scale companies would absolutely require to host Booste on their own infrastructure. Like a bring-you-own-server model. Bring-you-own-server and multi-cloud not in the works in the near future, but perhaps mid-long term.
31 comments
[ 3.6 ms ] story [ 66.2 ms ] threadBooste is a command-line interface (CLI) that takes code written on your local editor/IDE and executes it in EC2. It runs remotely, in a pre-configured environment, but works with your local tools thanks to a background filesync process.
I want to fix the “it works on my machine” issue by bypassing personal machines entirely. As if Docker and Heroku had a lovechild.
How I came to building Booste:
Being a mechanical engineer turned software dev, there were countless times where my laptop (glorified tablet) couldn’t handle the workload needed, so I made Booste v1: a tool to host full desktop apps on AWS and stream it over RDP (remote desktop protocol) to clients. Some may recall the Show HN of Booste v1 six months ago.
Booste v1 sucked. Unusable latency, crap unit economics, and an infrequent pain point.
Through it all, the most common customer request was for a hosted VS Code. Prospective users wanted the perks of a cloud IDE (namely no-setup environments) but wanted to use their own tools. Diving deeper, it seems that most large companies build out their own (usually buggy) version of remote “devboxes”. The need seemed consistent and clear.
After the first Show HN and a failed W20 interview, I pivoted. I gutted the graphical desktop component, zoomed in on developers, and rebuilt as a CLI: Booste v2.
Running any shell command proceeded by the world ‘booste’ will trigger the CLI. The CLI tunnels that command through SSH to EC2, executes it there, and streams the stdout back down to your command line. Filesync makes sure EC2 is ready with your code.
Where “npm start” would run an app on localhost, “booste npm start” runs the same app in the cloud, visible to your team by URL. You can edit the code locally (VS Code, Sublime, Vim) and the save is synced and hot-swapped on the live URL within a second.
How the environment sharing works:
- The creator of a “Codebox” (EC2 instance) gets a barebones Ubuntu VM, on which they can set environment variables, install interpreters and packages, or set up databases. (pre-built env templates coming soon).
- Team members join the codebox by ID and password with a single “booste join” command.
- Codebox members get repos placed next to each other on the directory tree, so code doesn’t clash.
- There is “booste clone” feature in the works, where you can clone someone’s config without actually entering and messing with their codebox.
Under the hood:
-The CLI is built in Python using the Click library
-Codebox infrastructure is automated with the AWS SDK.
---Currently, they are EC2 instances, for the sake of getting the beta live. I plan to migrate to a more flexible containerized setup in the near future. Containers make much more sense.
-I built a Dropbox-like filesync, inspired by the rsync.
---It monitors a chosen directory on your local machine and syncs it with the codebox.
---When files in that directory are added, modified, or removed, the codebox reflects this within a second. Only differentials are sent.
---All communication is encrypted with frequently cycled keys, unlike rsync.
---Bulky files (ML data, dependency packages), which would otherwise slow rsync to a crawl are ignored by a customizable .boosteignore file.
-Since files sync asynchronously (on edit), the only latency is from ssh. There’s a 0.5 - 1 second delay on booste commands.
I’ve moving from closed to open beta. Fueled by covid and the now mandatory remote development teams, I made a free tier for you. Give it a try! Let me know what you like. Let me know what you hate.
- not having to install things like Docker and Postgres on my own machine, save space, not having to get a computer with as much space in the first place
- not having the heaving lifting on my machine means that I won't get the whirring fan and heat from my laptop
- not having to deal with managing different versions of node on my machine for different projects
- developer onboarding could be faster. There were a lot of little details to get up and running. Getting the npmrc token on my machine so I can install the private packages etc. etc. These could just live in the cloned cloud box that every new developer gets.
- lower the barrier of entry for new developers to just start doing something fun and useful. First day of bootcamp was this install-fest where instructor had to babysit everyone to make sure they got everything configured right. It's probably good for people to learn that stuff eventually but it would help to be able to just jump right in.
One downside: if the wifi is down you won't get able to get anything done at all.
My go-to cloud IDE so far has been Stackblitz. I use it to do little proof of concept stuff, share the link with people. But I prefer working in my own text editor. And you can't do anything more than a simple throwaway project in stackblitz.
I have yet to see what the usage will be, between having one living "shared" environment, or multiple "cloned" environments.
Wifi is an unfortunate issue. Probably a mid-term feature, but one of my early beta users suggested an "offline" version of booste where we clone images downward to the laptop, for use during internet downtimes. Could be a move.
Did you know rsync supports providing exclusion filters in files? See `--exclude-from` and `--cvs-exclude`. The latter uses .cvsignore files just like git uses .gitignore (and, I assume, you use .boosteignore).
Is there any site that ties open-source programs to their equivalent SaaS product? That would be very interesting to look at, to see what went into building these various SaaS products.
Sounds easy to implement, and many companies have attempted to do so. Generally, the solutions you've suggested are what they turn to, with mixed results. Rsync bogs down, bare ssh doesn't let you use gui editors, and the users I've talked to so far consider docker to be opaque.
Folks here on HN are strong engineers. They're happy working with current tools, because they've already gone through the ringer to learn them.
My assertion is that this skillset shouldn't need to be learned. When I write code, I don't need to choose which CPU core is used, and I'm grateful for that abstraction. I'd like Booste to abstract cloud ops in the same way.
The point about GUI editors not supporting remote development is basically false. VSCode, GVim, IntelliJ, Eclipse, Visual Studio, Sublime Text all support remote dev.
As far as Docker's opacity, I guess that's a fair point, although in my opinion it's kind of unacceptable these days to not understand Docker or, if not Docker, then containerization in general, as a software engineer.Edit: list formatting
The Booste implementation of rsync is definitely not indestructible, especially during this beta. Been a pain in my butt, but making progress.
Main improvements over rsync: - .boosteignore. Just like a .gitignore, a .boosteignore helps filesync filter past bulky files that may not need to be monitored and uploaded every time. - end-to-end encryption
Admitted flaws of our current algorithm: It's built pretty ground-up in python. Haven't speed tested against rsync, but using it so far, I've never managed to edit code and run it through booste before the sync made it through to remote. IE the ssh tunnel doesn't beat rsync, mostly because files sync asynchronously. That said, I plan to do port it to a compiled language to cut time.
This is a tall claim, given rsync uses ssh for encryption of connection.
Diving deeper, it seems rsync doesn't encrypt itself, but SSH obviously does. You got me on that one :)
but thats besides the point, i only wanted to nitpick a little:
> Main improvements over rsync: - .boosteignore. Just like a .gitignore, a .boosteignore helps filesync filter past bulky files that may not need to be monitored and uploaded every time. - end-to-end encryption
thats a verly little improvement over rsync, considering it already has a file-based blacklist feature as well and is usually used over ssh, which is also encrypted ;)
though it (to my knowledge) doesnt behave like a gitignore, so whitelisting files inside of a blacklist is afaik not possible.
I'll definitely play around with this alias and see how it performs relative to .boosteignore
[1] https://news.ycombinator.com/item?id=8863
Still experimenting with how to pitch it. Dropbox for devs makes sense, knowing the details, but would be super vague for anyone who doesn't already know anything about booste.
Other pitches that fall into that "short but vague" pitch.
Filesync + SSH
Docker + Heroku
*Cloud IDE without the IDE
(edit: list formatting)
For years the ssh type lag made a remote editor too painful. Now we have things like mosh (with local echo) or syncthing or even native vim scp url support. What makes this better? I can’t tell from the website.
Booste allows GUI code editors like VS Code and Sublime.
I’m not saying your idea isn’t viable, but there are multiple players moving in this direction.
To me, the main thing you’ll have to convince me of is that this is better than editing code locally, but running the code in a local Docker container. For me, this solves the issue of “it runs on my machine”.
From my perspective, your main value add is trying to make executing and testing code on a remote server frictionless (it’s not quite there, but pretty good so far). And that you can edit on a low-resource machine, but effectively “develop” on a more powerful remote system. But these might be a tough sell if you’re targeting the HN crowd (which I’m not sure you are).
The docker point is fair too. Docker containers solve the machine problem, and are incredibly helpful teck. There are perks of cloud hosting (visibility by team, potential to monitor prod and have the codebox reflect it, performant hardware) that we can layer on top of docker container tech. Just hypotheses, to be clear. But to me, we're doing to Docker what Google Docs did to MS Word.
Coupled along with something like docker-machine for instantiating those remote instances, what does this setup not offer that yours does?
I do have the "my laptop's hardware is not enough" problem, and my current approach is simply spinning up an EC2 instance with the desired resources and ssh into it and then cloning the git repo I'm working on.
Normally, I set up the instance once (I have to add the ssh keys to my github account), and (if it's an expensive instance) start/stop it so I don't leave it running over night. Not a big pain point as of how I currently work.
I could see myself using Booste if it allowed me to, with a single command, spin up and sync bigger instances that would run for a short period of time (eg, for ML model training) and then intelligently stopping/terminating it. Also, if you integrate more than one cloud provider, I could switch to the cloud provider best fit for my needs (cost, spin up time, latency, cpu/gpu, etc).
A couple thoughts on the current implementation (you might be already addressing): - I generally have more than one ongoing project at a time. Having everything in ~/Booste sync'ed with every box doesn't work for me (aside from the fact I'd use a different location) - I sometimes do need to work with big data files, and even if I don't want them syncing all the time, I do want a way to retrieve them easily. - I also often work with jupyter notebooks, so I'd need a way to tunnel (like I do with ssh)
Hope this helps and good luck!
re: ~/Booste directory, just pushed and update yesterday that allows you to move your filesync directory around to wherever you want. It's not quite perfect yet, as I haven't built a way to clean old project files out of the codebox. But your new repo will overwrite anything that'd get int the way, so functionally, you're good to go there.
There's a "booste upload bulk" command, which takes a .boosteignored "bulk/" directory, for hefting code up to the codebox without bogging down filesync. Haven't get built a "booste download bulk" command, since customer pull recently has been more web-dev oriented, but I'll get on that once more ML folks play with Booste.
Codeboxes have auto-spindown. Default four hours live, but could make a command to monitor processes and spindown once done. Like a flag "booste --stop-when-done python3 train.py"
Nothing for Jupiter notebooks yet, and it's frankly going to be a tough technical nut to crack. Interested in exploring though.
(Edit: added below)
As for cloud flexibility, I'm sure more enterprise scale companies would absolutely require to host Booste on their own infrastructure. Like a bring-you-own-server model. Bring-you-own-server and multi-cloud not in the works in the near future, but perhaps mid-long term.