HN Explorer

Who's using Google App Engine?

51 points by micaelwidell ↗ HN
I'm looking for a scalable web hosting provider which is free or cheap at entry level, and which isn't a VPS (I don't want to bother with server maintenance). Google App Engine is the only one that comes to mind.

What big sites use Google App Engine? Have you tried it? Is it flexible enough to run big, serious projects on?

70 comments

[ 3.0 ms ] story [ 125 ms ] thread
gr366
I'm using it for small, very low-traffic projects, so I'd also be interested in hearing about larger sites using GAE. Also curious whether they're using Python, Java or something running on the JVM like Clojure.
gte910h
I've found it just fine for whatever clients want to do....EXCEPT....when they need to contact a server over a non-http port. It just can't do that. So for my client who wanted to do apple push notifications for his iPhone app, we needed to use both app-engine and dumb endpoint nodes on rackspace 11 buck a month linux servers.

Amazon Ec2/S3 works well as well and has the highest security bar of all the cloud offerings.

Rackspace is pretty good, but everyone seems very dis-satisfied with the S3 competitor, cloudfiles, so just use them for servers. Don't put anything super valuable there though, they know your root passwords (as does google, if you think about it, on GAE).

The users are on python. Apologies for the poor formatting/spelling, having to rush out the door.

mattdennewitz
cloudfiles could certainly do a bit more to catch up to s3, but i've never been left wholly unsatisfied.
gte910h
I've had 2 people who had horrible issues uploading media. Unusable levels of can't upload things.
jeff18
That is no longer a limitation as of the 1.3.2 release. I quote:

URLFetch API - We’ve expanded the number of ports you can access with the URLFetch API. You can now access ports 80-90, 440-450, and 1024-65535.

gte910h
Well that will be nice that I can then talk to 2195
derefr
> I'm looking for a scalable web hosting provider which is free or cheap at entry level, and which isn't a VPS

Heroku! Basically the Ruby equivalent of GAE—you get a Postgres database instead of Bigtable, though (whether that's a good or bad thing is up to you.)

nir
Heroku's great, but in money/performance ratio it's not really comparable to GAE. The free Heroku package doesn't get you very far as far as scaling goes (though if you're willing to pay it's a great alternative to hosting your own Ruby site)
derefr
I'd question how far you can really manage to scale something before you have to start considering it a commercial project requiring investment. My experience is that the free Heroku and GAE levels are both fine at the prototype/closed beta stage, unless your "closed beta" is the size of Gmail. Both services have heavy caching infrastructure in front of them, and you get as much of that as you need at any level—you only need to worry about scaling for actual CPU-intensive tasks, which should always be handled gracefully client-side to begin with (in case of network latency &c.)
Qz
Is Heroku free or just cheap?
derefr
It's free given the base configuration: http://heroku.com/pricing#blossom-1-0

That config is basically only for development/prototypes, though (it has a 5MB database cap); their default tiny-ish plan is $15/month, which is quite decent for anything that runs Ruby, let alone on a grid.

Turning on all the knobs and levers on that page gave me "$5089/month", which is a slight bit terrorizing—but I can't imagine the kind of business that would grow to need that scale without becoming a multinational with its own IT.

evandavid
I am loving Heroku for the exact reason described in this post: no sysadmining. At first I thought that wasn't possible and expected all kinds of issues with gems etc, but so far everything I've tried really does Just Work™.

This may warrant a separate question, but I've seen various posts here and there about running Ruby/Rails on GAE using JRuby. Does anyone have any experience as to just how realistic that is as a deployment option? I can imagine it being full of headaches, but I would love to hear from someone who has either tried it or is actually running it on a production app.

skybrian
I've not run it myself, but a neighboring team tried it and switched to Java due to latency issues. Remember that anything you do at startup will affect latency for some requests when App Engine has to spin up a new JVM. It's to your advantage to make startup times as fast as possible.
mattdennewitz
i've used appengine from time to time @ pitchfork, and it's been great. with the recently relaxed restrictions on uploads, result sets length, and the task queue, and ability to read and process larger uploads, and xmpp and incoming mail handling, etc, i'd say it's in good shape for bigger, more "serious" projects. absolutely capable for smaller projects, too.
chipsy
I'm considering working with GAE to make a persistent browser-based game. Not serious about it yet, but maybe in another month or two.

I have some doubts about staying within the CPU quotas, since I have to take on a processing load in the tick-update loop of game time, but I think I can design it to be spread out somehow.

psawaya
I looked into this a while ago, and it seems like GAE still doesn't have a real way for the server to push to the browser (Comet). There's a ticket open for it (http://code.google.com/p/googleappengine/issues/detail?id=37...), and it's a feature that's "on deck" (http://code.google.com/appengine/docs/roadmap.html), but who knows when they'll get around to implementing it. As I'm sure you know, regular polling is usually too slow for a game, and counts against your app engine HTTP request quota.
bd
I did a very simple game on GAE in its early days and the biggest trouble were very costly writes.

Basically you could forget about applications that needed to write game state to datastore in each step.

Even the simplest possible writes were already triggering CPU quota warnings [1].

I didn't touch GAE recently, so maybe they already relaxed it a bit, but when it started, it was usable practically only for read-heavy applications (for which Google strongly suggested to use memcached wherever possible).

Also moving larger amounts of data between your computer and GAE datastore was massive pain (e.g. backups or initial state load). You had to be very careful not to blow over your quotas, as such imports were doing a lot of writes.

---

[1] For example, in early days they counted CPU use in "megacycles":

  * memcached based ajax request costed tens of megacycles
  * datastore-read based ajax request costed hundreds of megacycles 
  * datastore-read-write ajax request costed thousands of megacycles
grandalf
this has gotten much better. Also, they now let you specify what level of consistency you want, which makes writes go a lot faster if you're just putting in a bunch of data or don't need to wait to be sure it's consistent before you put in the next item.
johnnytee
I built a small app on it Tweetc.com using python and webapp framework. It was my 1st gae and python app. So far no problems. I know Twibes is built on it which gets moderate traffic. I like it can I can quickly build and test ideas for free.
adora
I talked with one of the guys that works for Google App Engine, and he confirmed that BuddyPoke is one of the biggest (if not THE biggest) apps that run on it. That app serves millions of DAUs on various social networks and runs just fine.

I've set up things on app engine and amazon as well. I found GAE just to be much, much easier and saved me a lot of time and money (yes I know EC2 is cheap but it's not cool to see your bill growing when you're just testing things).

I actually took the time to switch to EC2/S3 for a couple days because I was worried that GAE was not really serious about making it a productive place to run serious applications. Portability was a big factor too. But after a while I figured that all this worry was nonsensical. If I ever have to move things over, it must be a good sign and either someone will have built a tool for it or I just spend a week building a tool for it.

Also, re: SSL. You can actually do it right now on GAE but it involves a few tricky workarounds. That said, the GAE team just pushed a more straightforward feature for this to the top of the roadmap. Beforehand they had said they weren't going to work on it anytime soon, but presumably after so many people complained they decided to go ahead with it -- another good sign that at least they're responsive to developers' needs.

kaffeinecoma
I've been using GAE+GWT for about 6 months now, mostly for small, "do it in a weekend" type projects. I've also used it to host content for my iPhone apps, so that I could update the content (mostly ebooks delivered as PDFs) without going through the awful App Store update process.

It's been great for me. I love how quickly I can bootstrap new ideas, put up a prototype, and then just hand out the URL for to people to try.

A couple of recent apps:

http://create-a-password.appspot.com http://slimpoints.appspot.com http://mealminder.appspot.com

That last one was mostly a proof-of-concept in using Google's account and payment infrastructure. Really handy if you decide that you'll want your customers to pay you some day.

I guess the biggest hurdle is getting used to the Datastore, which is not really intuitive if all you're used to is SQL. Overall, a very positive experience for me.

araneae
Meal Minder is cool, but you might want to work on the automatic-filling-in bit. It read my directions as an ingredients list... AND as directions. I listed my ingredients with bullet points (*) which is a pretty common format, and they weren't recognized at all.
snprbob86
Although we aren't a big site by any measure; we're using it and loving it for http://www.classlet.com
clemesha
Great site you have going there. Went through the whole thing, and then demo. Clean and straightforward.

Quick question, a bit off topic. Here: http://twitter.com/Classlet/status/3854900110 you say "We finally rescued our Twitter user name from a squatter!". How did you manage to do this?

snprbob86
Thanks!

As per some instructions on the web, we created a "classlet2" Twitter account. When logged in, there is a help form where you can open a ticket. It took a few weeks, but they eventually released the account to us.

[deleted]
(comment deleted)
waleedka
I use app engine for several projects, a couple of which have decent load. My biggest one is networkedblogs.com, which I started on php/mysql and recently migrated big parts of it's back-end to the app engine. My experience with it has been good, and I always build new projects on it, unless there is a good reason that prevents it. It has it's own pros and cons, but overall it's a good choice:

- It's easy to start with and fun to use. You shouldn't need more than one day to learn it.

- If you like to focus on coding and not worry about managing the server then you'll love it. But if you're a control freak and like to have access to low-level settings then you'll not.

- If you have prior experience with relational databases such as mysql, then you'll need an adjusting period to forget much of what you learned about databases and start thinking in new and different ways. Some things that are really easy in mysql, such as getting the number of records in a table or joining tables, become hard on app engine and require writing code.

- When it goes down, you'll feel helpless that you can't do anything to bring your site back up, but you'll also feel lucky that a Google employee is carrying that beeper and have to wake up and fix it.

- It scales, but it's not super fast. Due to its distributed nature, individual requests rarely finish in less than 50ms and usually take 100 to 150ms (that' my experience, yours will likely differ). But it can run a huge amount of requests at the same time.

- It solves the scaling problem, but it's not magic. You still need to optimize the heavily used parts of your code for scaling. But at least you do it with code, rather than having to fiddle with server and cache settings.

- It's a paradigm shift that will eventually eliminate most of database and system admin jobs if it catches it (I think it will).

- It's very different from Amazon's EC2. Amazon gives you servers that you have to manage, Google manages everything for you. With ec2 you pay per server hour, with App Engine you pay for resources you use: CPU Cycles, storage, bandwidth, ...etc.

- It's free to start with, but if you have heavy load it's a bit more expensive than hosting dedicated servers. I think it's worth it, though, because you'll be saving a lot of your own time.

- It's good for the majority of projects, but if your project has special not-too-common requirements, then it might not. Check first.

- It only supports python and Java right now.

If you're in Silicon Valley and if you need more in-depth information, you can find me at the next App Engine meetup at Google on the 6th.

rmason
Actually it also supports CFML, remember CFML is built atop Java.

There's an open source version known as Open BlueDragon that runs on the GAE (and EC2) and has made the cloud their specialty http://www.OpenBlueDragon.org

Here's a link on the section of their wiki on the GAE: http://wiki.openbluedragon.org/wiki/index.php/Category:Googl...

jeff18
I've been using it for about a year for http://www.wolfire.com and several hobby projects and am a big fan. If you have any specific questions, I am familiar with most aspects of it. It certainly has a lot of limitations, but if your app happens to coincide with what App Engine currently offers, it is a pretty sweet deal.
micaelwidell
About where do you usually hit the limit for free usage? Are we talking hundreds of daily visitors? Thousands? Tens of thousands? I know this varies heavily depending on your code, but does anyone have a good rule of thumb?
jeff18
We rarely go over the free limit -- I think we've stayed under $10-20 for the entire year, while serving up several million pages. Most of this is due to bandwidth overages from our comic: http://www.wolfire.com/comic

The times we are charged tend to be for burst traffic when something gets picked up on Digg or Reddit. Recently, one of our blog posts was featured on Digg, Reddit, Slashdot, etc. at the same time and Google dutifully served up a couple hundred thousand visits over the course of a few days. They charged us about a nickel! Granted everything is nicely cached so we were not doing much heavy processing.

It definitely depends on your web app. If you can cache it, App Engine is essentially free. If you are doing something CPU intensive for every visitor, it's going to be expensive.

eplanit
Sounds like from all these postings that its appropriateness for "big, serious projects" just isn't there. Now that it's over 2 years old now, I'm underwhelmed (yet not at all surprised) by it's lack of adoption/popularity.
dhess
While we're on the subject, does anyone here have experience using Clojure with GAE? I'm considering that configuration for a useful-but-not-mission-critical application, and wondering if Clojure on GAE is robust enough for that.
DTrejo
Yes, someone please tell us about their experiences?
jeandenis
I've been using Clojure w/ GAE to build an enterprise knowledge-base webservice for about 3 months now. It's my first serious project with Clojure on the GAE although I've been playing around with it since last summer.

(1) On GAE you don't get to use some of the good parts of Clojure because you can't spawn any threads. So the rationale for Clojure has to be something else than agents and the Clojure concurrency model. I'd say use Clojure if you want functional programming and all of the Lispy goodness: code as data, macros...

For me Lispyness is a huge + so it was a no brainer.

(2) One of the things I really like about traditional Clojure web-development (using Ring or Compojure) is that if you set things up properly you can experiment with and patch a live system using the REPL. I haven't been able to set that up just yet on the GAE.

On your development server using the GAE jars, however, you can do it (see http://www.hackers-with-attitude.com/2009/08/intertactive-pr... for some guidance). There is nothing like being a C-c C-c away from updating your development server app :)

I can't emphasize enough taking the time to get Slime, your GAE development server, lein, and Clojure well-integrated. Your development cycle will make good ol' Java GAE development using the Eclipse plug-in (which I used to think was pretty sweet) feel really really clunky.

(See also http://elhumidor.blogspot.com/2009/04/clojure-on-google-appe... for some pointers on setting up Clojure on GAE)

(3) GAE's datastore works extremely well with Clojure because it's schema-less. You can store arbitrary hashmaps and structmaps in GAE with zero effort -- it's a match made in heaven. There are a couple of branches of appengine-clj (http://github.com/duelinmarkers/appengine-clj is the original, http://github.com/r0man/appengine-clj is a pretty good branch) that you can look at to get a feel what I mean.

(4) With GAE's transaction support and automatic index generation (it's good enough most of the time), you can dynamically create types (kinds), put them in the datastore, and query on them without having to do any of the traditional NoSQL stuff (thinking queries first, denormalizing ahead of time, etc.). It's very sweet. I've done some Clojure+Cassandra and Clojure+GAE is a lot more painless (as long as you can live with the other limitations of the GAE) -- although things are getting much easier in Cassandra world with Hector + automatic index generators.

(5) In terms of performance and robustness, nothing about Clojure's held me up. But since I haven't launched yet, that doesn't mean anything.

I don't know of any "high-traffic" Clojure+GAE apps out there -- if there are any, I'd love to get some numbers on costs (CPU, especially) to see if there is a significant difference with similar java projects.

(Actually, the guys behind https://the-deadline.appspot.com, see http://www.slideshare.net/smartrevolution/how-a-clojure-pet-..., just launched three days ago -- you may want to shoot them an email, they seem nice enough)

My guess is that this is a non-issue. Clojure holds up really well against Java and JVM languages like Scala in terms of performance (especially if you give type-hints), and I'd expect the same thing to be true on GAE.

(6) General Clojure limitation with web development: Clojure web frameworks are light. You don't get to start with anything as nice as Scala's Lift. You'll have to work and build your own tools -- the good thing is that Clojure is a...

fuzzythinker
This is the most detailed and helpful post I read on HN. I wish I can up vote it multiple times.
jeandenis
Thanks, I appreciate it. First post for me after being a longtime reader so it's nice to know it was helpful to some.
smartrevolution
Hi. I am from above mentioned TheDeadline-Team!

We can definitly recommend Google App Engine. The SDK is really cool and you don't have to care about managing servers. Scaling is easy. Its (almost) automatic. The datastore is a natural fit for Clojure datastructures, but you have to learn how to structure your data for a distributed key-value store: When you structure your data, you have to think about what has to be in the same transaction (entity-group) and at the same time you have to think about reducing data-dependencies to minimize contention. But when you have lots of data, this is what you have to do anyway. For me, it is much more fun than using an RDBMS. (Read this: http://code.google.com/intl/de-DE/appengine/docs/python/data...)

We did the whole system in Clojure, so you really can write large systems using Clojure/GAE. It works! (Although the UI is simple, TheDeadline is a complex piece of software e.g. containing a self-written rule-based expert-system, a production system, very similar to LISA http://lisa.sourceforge.net/.)

The functional paradigm means less code and makes it easier to build powerful abstractions. You don't have to write lots of boiler-plate code like in Java and Clojure is more concises and much more powerful than Python. (Stop! I don't want to start a programming language discussion now! :) ).

We will soon post some Clojure examples and some free software libraries for GAE.

If you are interested, just follow our blog H.W.A. (http://www.hackers-with-attitude.com) and/or try out http://the-deadline.appspot.com to get the "feeling". ;)

frognibble
The one problem with using Clojure on App Engine is that the time for a "loading request" is at least 5 seconds. It makes your app look bad for the random user stuck on the loading request.
jeandenis
I'll interpret that as a dig against the App Engine and not Clojure... even though it's clearly aimed at either Clojure alone or the combination with GAE.

Anyway, are you being serious (i.e., in your experience is clj+gae noticeably slower than java+gae or python+gae?) or just funny? Since I've never done any serious profiling of Clojure on GAE, I'll bite and do some brief performance tests and report back.

frognibble
I am serious when I say that the GAE loading request time for a Clojure app is noticeably longer than the time for Java or Python. I searched for some benchmarks and found this: http://github.com/garyburd/app-eng-cold-start-test See http://googleappengine.blogspot.com/2009/12/request-performa... for more info on loading request perf.
jeandenis
Both of those are about "loading requests" which happen pretty rarely in the larger scheme of things (on upload, new server for app, too much traffic so need to scale jvm). On my current project, it's 3.5-4 seconds. Definitely not ideal, but it's not that common an event.

I'd like to run some benchmarks for normal requests, where I doubt there'll be any real difference.

Thanks for the search.

dhess
Your reply was much more detailed than I had any right to expect, so thanks for taking the time to write it. You've eliminated any doubts I had about using Clojure with GAE.

I encourage you to write more about your Clojure/GAE work, and, if you do so, to reply here so that those of us who posted in this thread know when you've got something up. I guarantee you at least one reader. :)

okaq
GAE is ideal for nano web apps. I wrote an HTML5 Canvas Plasma Benchmark using GAE/P:

http://speed.okaq.com/

It clocks in under 10kB compressed, is very fast, very accurate and I use it to test devices all the time.

Under current quotas, I'm able to serve ~200k users per day at no cost and am actively developing 2 more nano web tools.

The development environment is certainly restrictive, but the risk for web app prototyping is absolutely minimal.

grandalf
If you're disciplined about how you design your app it's an amazing platform.

I recommend doing a fairly trivial project in your spare time to get the hang of it. Try out the following features before deciding:

- large data set related stuff like cursors, paging lots of records - transactions - GQL (look at the various limitations of it)

GAE gives you a set of solid conventions for writing a highly scalable app that you'd eventually have to deal with if you used a more traditional approach. Some can be a bit annoying at first but it's fairly obvious why each choice was made.

You might also check out jinja2, django, etc. But personally I love the simple webapp framework. It's clean, obvious, stays out of the way, and it's very fast.

olsonjeffery
I used webapp pretty faithfully, myself, until I came across bottle.py .. I haven't looked back, since.
grandalf
interesting. was it easy to set up on app engine?
camperman
I saw a note in the bottle docs that it's easy to set up with GAE but no further info. Any tutorials out there?
[deleted]
(comment deleted)
[deleted]
(comment deleted)