That's pretty amazing how quickly I can install and use an app. It's instant as far as I can tell. This makes me wonder - how difficult would it be to take a sandstorm app and migrate it to the internet? And how difficult is it to migrate an app to your platform?
Sandstorm uses a native-code sandbox on Linux which can run standard Linux binaries, so generally porting an app to Sandstorm is pretty easy. Most of the work is actually in removing the app's existing login system and relying on Sandstorm's authentication instead. David touched on this in his post about porting ShareLaTeX last week:
Most of our existing apps are actually ports of existing open source Linux-based servers. Most were ported by Jason Paryani over the last month, which tells you how easy it is. This includes apps written in PHP, Python, Node, etc.
Note that the dev tools (and all the platform source code) are available to anyone. There are some WIP docs on the wiki:
It's probably actually harder to port _away_ from Sandstorm simply because you have to add a login system and other such features. Adding is a lot harder than removing. :)
Oh, maybe I misunderstood. I thought your question was about porting code, not data.
It's actually incredibly easy to move your data between Sandstorm hosts. Unlike, say, Google apps, Sandstorm apps are not tied to the hosting, so you can change hosts while still using exactly the same app. You can also run Sandstorm on your own machine if you want (it's open source).
Porting data from a Sandstorm version of an app to a non-Sandstorm version is also relatively easy. If you click the "download a backup" button, it's just a zip file that contains the app's database in whatever format it uses. But, how exactly to set up the app outside of Sandstorm depends entirely on the app.
Regarding web2py: I don't know much about it, but I assume it runs on Linux so it should work fine for writing a Sandstorm app...
Web2Py is a python-based website, like django, but it's a bit different. Since you say you are passing an authentication header, I don't see how it could work with Sandstorm, since it wouldn't really be used as a one-user app anyway.
So the apps on the platform are essentially forked versions of open source apps. If you're editing the authentication system, which I imagine can introduce some heavy dependencies, how do you plan on handling updates from the original developers? Presumably there will be manual labor involved with merging the origin master branch into the Sandstorm app. Do you plan to do this yourself?
> If you're editing the authentication system, which I imagine can introduce some heavy dependencies...
Not really true. Sandstorm acts as a proxy in front of the apps and adds headers which identify the user. It takes very little code on the app side to read these headers and use them rather than use the app's own authentication.
That said, we do have some changes which we'll have to rebase from time to time. Eventually we hope that app developers will be interested in maintaining these changes themselves and explicitly targeting Sandstorm, but that is probably a ways away still. In the meantime, maintaining the edits doesn't seem to painful.
Just to tie a few other things together, kentonv was the primary author of Google's Protocol Buffers, and current author of open-source Cap'n Proto [1].
While at Google, he initiated the Jeff Dean Facts prank [2], and in his spare time part-designed his own LAN-party-optimized house. [3]
So, you know, if he thinks he can provide a differentiating hosting platform, I'd have a glance.
Just need to correct one thing: I didn't invent Protobufs (Jeff Dean and his colleague Sanjay Ghemawat did), but I did write most of version 2 and was responsible for open sourcing it.
Thanks for the correction! In a way, it's kind of demoralizing that it's actually yet another product of the Jeff & Sanjay Brain Trust. It highlights how much they're in a league of their own, unattainable.
That said, my googler friends say they much prefer ProtoBufs v2 ;)
Part of the reason Jeff and Sanjay invented half the tech at Google is that they're incredibly smart. The other part is that they were there very early on, and basically have had free reign to spend their time however they want. Of course they spend it all inventing things, which means they output a lot of awesome stuff. Certainly not just any hacker can produce the kind of quality work they do, but I think there are a lot of other very smart people at Google and elsewhere who could be doing similar things if given the same deal. But, most of those other people didn't bet on Google as early as Jeff and Sanjay did, so...
And, of course, since it's in relatively small letters on that page, it is probably valuable to remind folks: Sandstorm is also being crowdfunded! [1] I've gone ahead and supported it, and I hope that you will too: Sandstorm is an application that I really want to run on my own machines, and giving Kenton et al the resources to develop it seems like a big step forward in building a truly /open/ cloud environment.
Great stuff. Could you share some data on the server and the traffic? Ram/cores/uplink? This looks like the kind of stuff I've been thinking about for a long time - will have look at the code and test it on a server. Is everything in place to run a clone of your demo server easily?
Over the course of yesterday we had 747 demo users launch 952 app instances.
Currently we're on GCE for price reasons, though we're looking for good non-Google, non-Amazon options for our eventual release.
Sandstorm tends to be memory-bottlenecked since we're launching separate instances of each app for each user, although we make up a lot of it by shutting down instances when they aren't in active use.
I was super-paranoid about the demo totally breaking under load (having not tried this before), so I upped the server to GCE's largest-memory instance at 104GB of RAM.
That turned out to be completely unnecessary. It never used more than 6GB (and CPU was mostly negligible). Doh. Oh well, better to have spent too much than too little.
We also have a separate front-end server which does our SSL, so that our private keys don't reside on the same server as apps, just as an extra precaution. That machine is just a regular n1-standard-1 instance. CPU usage was in the single digit percentages there as well.
Thanks for the detailed reply. I had some trouble building the whoke thing under Debian stable (with a 3.14 kernel from backports, trying to force everything to work under my user id in $HOME/opt. Not too surprising. Meteor serms to be part of the problem (got it working with its self-contained binary install) - as well as the apparent need for a newer libfuse and some c++ headers I couldn't quite figure out. Although I was building with clang-3.5). I'll have a go at getting the vagrant box up, and then trying again in a Debian testing vm/chroot. If that fails, I'll have to go with an ubuntu vm/chroot :-)
At any rate, sounds like my 4 core gbuplink hetzner box with 16gb ram should have no problem working with this (obviously overkill for a personal playpen, but its good to know I have lots of headroom...).
Are you planning to stay with meteor going forward? The other components seem much less complicated to get up and running, and more sys.adm. friendly. An example being the need (?) for node/npm and meteor's bundled node/npm...
It sounds like you were installing from source. There are indeed a lot of moving parts to deal with there. We also have a binary installer (and auto-updater) which "just works" with essentially no dependencies. I understand some people aren't comfortable with binary installers, but you have to understand that building from source will always be more work. :) That said, feel free to file issues for any obstacles you encounter that aren't already covered in the docs.
We do plan to stick with Meteor. It's a pretty awesome framework. I find Meteor is incredibly easy to get running, but this is again due to its binary installer script which maybe you don't like. :)
I believe that Meteorite (the "mrt" tool which you had to install separately using npm) is slated to go away before Meteor 1.0 as that functionality will be rolled into Meteor itself. So, that part will get better, at least. For now, you actually can (and arguably should) install it using Meteor's bundled npm rather than the system npm.
Actually, the mrt tool wasn't a problem (I had node installed from source, in my home-dir under opt from before -- working quite well with xstow. Meteor not installing from source is partly due to me running Debian stable -- I'm not surprised when bleeding edge frameworks won't compile, just a little bit disappointed ;).
I was indeed installing from source, as that is a requisite for contributing to the project (if I can't build it how could I possibly submit patches).
I have no problem with you citing any one distribution/version as supported however -- it would be madness to demand support for "everything" from an alpha source release! And I think aiming for cross-platform-distribution comparability would be wasted effort at this point.
With Vagrant from wheezy-backports, everything came up fine -- just had to(?) increase the vms ram to 512 mb for ipython notebook to work.
Now my only problem is figuring out how to get the hacker cms to publish anything (to users without login). I've understood that the cms should be stuffing files under /var/www (in my vagrant/virtualbox vm) -- so I'll have to check if "publish" pushes anything there -- and if there appear to be an nginx-instance set up to serve files from /var/www...).
As for meteor -- I have a few reservation of their architecture in general -- but my main problem is the complexity of the setup -- especially how hart it seems to be to build everything needed from source. But then again, they're not yet at 1.0, are they?
(Still seeing as how easy nodejs is to build from source, it is a shame meteor manage to somehow mangle that process...)
The recommended way to "publish" from Hacker CMS is to follow the in-app "DNS setup" instructions, rather than point nginx directly at the storage. The latter was the old way of doing it -- is there a doc you're seeing somewhere that still recommends that?
No, (not other than the api docs for devs, that explains that one way to publish from (any) app is to push files to /var/www). I guessed the dns-setup was one way -- wasn't sure if that would work without a routable address though (eg: 10.0.0.0/x on a lan). Or in my case, with the test setup in a vagrant box.
[edit: That is, I couldn't find any docs on how to publish from hacker cms, but I did find this for devs:
A cool project in this "own your data" space is https://camlistore.org/ which IIUC could be the storage system of a personal cloud.
OT: I always take a closer look when a computer screen appears in a video - in this case I couldn't recognize what you're running there. My guess would be a a tiling window manager? But the other screen shows Windows so I'm curious.
The code on the right window looks like Haskell due to the one liners.
That's me running Qt Creator (my personal favorite IDE) on Linux. You may be disappointed to learn that the code is C++. :) Sandstorm's lower layers are all C++ since we interface closely with Linux syscalls, though I think in that shot I'm actually editing Cap'n Proto (which Sandstorm uses).
All of the machines in the video are running Linux, with GNOME 3 as the window manager.
> Camlistore
Yep, I've talked to Bradfitz about Camlistore a few times.
Sandstorm actually gives each app instance its own directory on the filesystem for storage, so that it can use any storage format it wants. So an app could use Camlistore. It may also make sense for Camlistore to act as an independent app which other apps connect to. But in any case, Sandstorm is agnostic to these things.
29 comments
[ 2.8 ms ] story [ 56.2 ms ] threadhttps://blog.sandstorm.io/news/2014-07-25-sharelatex.html
Most of our existing apps are actually ports of existing open source Linux-based servers. Most were ported by Jason Paryani over the last month, which tells you how easy it is. This includes apps written in PHP, Python, Node, etc.
Note that the dev tools (and all the platform source code) are available to anyone. There are some WIP docs on the wiki:
https://github.com/sandstorm-io/sandstorm/wiki
It's probably actually harder to port _away_ from Sandstorm simply because you have to add a login system and other such features. Adding is a lot harder than removing. :)
Anyway, this is wonderful. One more question - web2py?
It's actually incredibly easy to move your data between Sandstorm hosts. Unlike, say, Google apps, Sandstorm apps are not tied to the hosting, so you can change hosts while still using exactly the same app. You can also run Sandstorm on your own machine if you want (it's open source).
Porting data from a Sandstorm version of an app to a non-Sandstorm version is also relatively easy. If you click the "download a backup" button, it's just a zip file that contains the app's database in whatever format it uses. But, how exactly to set up the app outside of Sandstorm depends entirely on the app.
Regarding web2py: I don't know much about it, but I assume it runs on Linux so it should work fine for writing a Sandstorm app...
Nice implementation.
Not really true. Sandstorm acts as a proxy in front of the apps and adds headers which identify the user. It takes very little code on the app side to read these headers and use them rather than use the app's own authentication.
That said, we do have some changes which we'll have to rebase from time to time. Eventually we hope that app developers will be interested in maintaining these changes themselves and explicitly targeting Sandstorm, but that is probably a ways away still. In the meantime, maintaining the edits doesn't seem to painful.
While at Google, he initiated the Jeff Dean Facts prank [2], and in his spare time part-designed his own LAN-party-optimized house. [3]
So, you know, if he thinks he can provide a differentiating hosting platform, I'd have a glance.
[1] http://kentonv.github.io/capnproto/ [2] https://plus.google.com/+KentonVarda/posts/TSDhe5CvaFe [3] http://kentonsprojects.blogspot.com/2011/12/lan-party-optimi...
Just need to correct one thing: I didn't invent Protobufs (Jeff Dean and his colleague Sanjay Ghemawat did), but I did write most of version 2 and was responsible for open sourcing it.
That said, my googler friends say they much prefer ProtoBufs v2 ;)
[1] https://www.indiegogo.com/projects/sandstorm-io-personal-clo...
We still need your help to get this into production. Please check out the crowdfunding campaign: http://igg.me/at/sandstorm
Over the course of yesterday we had 747 demo users launch 952 app instances.
Currently we're on GCE for price reasons, though we're looking for good non-Google, non-Amazon options for our eventual release.
Sandstorm tends to be memory-bottlenecked since we're launching separate instances of each app for each user, although we make up a lot of it by shutting down instances when they aren't in active use.
I was super-paranoid about the demo totally breaking under load (having not tried this before), so I upped the server to GCE's largest-memory instance at 104GB of RAM.
That turned out to be completely unnecessary. It never used more than 6GB (and CPU was mostly negligible). Doh. Oh well, better to have spent too much than too little.
We also have a separate front-end server which does our SSL, so that our private keys don't reside on the same server as apps, just as an extra precaution. That machine is just a regular n1-standard-1 instance. CPU usage was in the single digit percentages there as well.
You can indeed run your own Sandstorm instance. There are installation instructions in our Github readme: https://github.com/sandstorm-io/sandstorm
Feel free to send me questions.
At any rate, sounds like my 4 core gbuplink hetzner box with 16gb ram should have no problem working with this (obviously overkill for a personal playpen, but its good to know I have lots of headroom...).
Are you planning to stay with meteor going forward? The other components seem much less complicated to get up and running, and more sys.adm. friendly. An example being the need (?) for node/npm and meteor's bundled node/npm...
We do plan to stick with Meteor. It's a pretty awesome framework. I find Meteor is incredibly easy to get running, but this is again due to its binary installer script which maybe you don't like. :)
I believe that Meteorite (the "mrt" tool which you had to install separately using npm) is slated to go away before Meteor 1.0 as that functionality will be rolled into Meteor itself. So, that part will get better, at least. For now, you actually can (and arguably should) install it using Meteor's bundled npm rather than the system npm.
I was indeed installing from source, as that is a requisite for contributing to the project (if I can't build it how could I possibly submit patches).
I have no problem with you citing any one distribution/version as supported however -- it would be madness to demand support for "everything" from an alpha source release! And I think aiming for cross-platform-distribution comparability would be wasted effort at this point.
With Vagrant from wheezy-backports, everything came up fine -- just had to(?) increase the vms ram to 512 mb for ipython notebook to work.
Now my only problem is figuring out how to get the hacker cms to publish anything (to users without login). I've understood that the cms should be stuffing files under /var/www (in my vagrant/virtualbox vm) -- so I'll have to check if "publish" pushes anything there -- and if there appear to be an nginx-instance set up to serve files from /var/www...).
As for meteor -- I have a few reservation of their architecture in general -- but my main problem is the complexity of the setup -- especially how hart it seems to be to build everything needed from source. But then again, they're not yet at 1.0, are they?
(Still seeing as how easy nodejs is to build from source, it is a shame meteor manage to somehow mangle that process...)
[edit: That is, I couldn't find any docs on how to publish from hacker cms, but I did find this for devs:
https://github.com/sandstorm-io/sandstorm/wiki/Publishing-to... ]
OT: I always take a closer look when a computer screen appears in a video - in this case I couldn't recognize what you're running there. My guess would be a a tiling window manager? But the other screen shows Windows so I'm curious.
The code on the right window looks like Haskell due to the one liners.
https://dl.dropboxusercontent.com/u/19746944/kv.png
(Also, its kinda surprising/refresing that Sandstorm isn't written in Go or Node... )
That's me running Qt Creator (my personal favorite IDE) on Linux. You may be disappointed to learn that the code is C++. :) Sandstorm's lower layers are all C++ since we interface closely with Linux syscalls, though I think in that shot I'm actually editing Cap'n Proto (which Sandstorm uses).
All of the machines in the video are running Linux, with GNOME 3 as the window manager.
> Camlistore
Yep, I've talked to Bradfitz about Camlistore a few times.
Sandstorm actually gives each app instance its own directory on the filesystem for storage, so that it can use any storage format it wants. So an app could use Camlistore. It may also make sense for Camlistore to act as an independent app which other apps connect to. But in any case, Sandstorm is agnostic to these things.