Something not mentioned in the notes is the addition of ndb to the python library. Located in google/appengine/ext/ndb directory. I am definitely liking the direction it is going rather than how the original db module works.
Now that we have ep.io, Gondor, and all the other PaaS things that support Python, the ability to upload an application and scale it as much as we want isn't limited to App Engine anymore. Whereas back when it came out, it was the only service of its kind for Python. (Heroku might not have even been out for Ruby, though my memory's a bit fuzzy at that point.)
And pretty much all the other vendors use plain old SQL databases and don't have everything as a platform-specific API like App Engine does. So when the prices went up, my guess is that people realized, "Hey, why should I lock myself in to this one vendor that could raise the prices again, when I can go with other vendors that would be easier to migrate between [or switch to hosting myself later]?"
Are there good python PaaS that doesn't lock you in? I'm looking for one for our next project (or migrate the current project away from App Engine). I guess I just don't trust Google any more.
I have had an amazing experience with ep.io. The invite list is a mile long, but if you have something that you already know you want to build, you can e-mail the team and they'll let you jump the queue (I think it's more they simply haven't sat down and powered through their list yet than any lack of capacity on their part).
The team is very friendly, and they have an incredibly generous free quota as compared to other services (2 GB of files/database, 5 GB/month of bandwidth, and 16 MB of Redis - per app). They use plain old WSGI for the server, Postgres for the database, and they'll install any packages you put in a requirements.txt file, so no lockin. Haven't used Django with it, but so far my experiences with Flask are great.
Awesome, thanks for the reminder. I've been waiting for ep.io to get back to me as I asked for an invite a while ago, but apparently Gondor is out of beta now and anyone can sign up!
I don't mind App Engine that much though, to be honest. The price changes don't really affect me and I've always been happy with their infrastructure. The only thing that bugs me about using it is vendor lock-in.
I'm taking a "wait and see" approach. App Engine has been a disappointment for its entire existence, but now it seems to be one of Page's chosen hyperxylopygian arrows. I tried launching one project on the platform a couple years ago, and it didn't work out, but I'm willing to try again for my next one.
Almost every developer I speak with has developed a passionate hatred of Google for a variety of reasons. And I don't know a single person who would trust Google enough to run their whole app infrastructure.
Google has gone to extreme lengths to make every company on the internet their enemy, and they've succeeded.
After a year of working on a midsize AppEngine project targeted at large enterprises, I can honestly say that AppEngine's only truly redeeming quality is its Google integration. Our product is an add-on for Google Apps; AppEngine runs on the same infrastructure; big companies like that.
If this doesn't apply to you, choose an up-to-date, cheaper, more versatile, far less tedious deployment platform. I've spent the past 10 years developing, deploying, and maintaining web applications of most sizes and I've never had a harder time "scaling" usage of mere GB of stored data. Maybe if I had 100TB then AppEngine would make the "impossible possible"; for the rest of the 99.999999% of the Internet, it makes the formerly trivial insanely tedious.
On deployments is not just a matter of updating the code, many times you have to update the data too and "appcfg.py update" really does not take care of that.
On one of my apps the only thing I have to do is literally "git push production release" and that takes care of everything.
I don't understand the scaling comment either.
How is it not scaling for you?
On scaling it is tedious because things we take for granted are suddenly hard. Some example of that, but not limited to:
---
The transactional support in GAE is really awful. Something which I take for granted when working in Django or Rails with a RDBMS: I want the writes I did to be reverted in case my request failed for some stupid reason and triggers an exception. It's nothing much, but I want this basic functionality that should be a no-brainer.
Transactions in GAE work on trees. Basically GAE puts a lock on a parent entity (every entity can have a parent entity) and every operation done on an entity with that parent will be inside the transaction (and reverted in case of rollback). But THE PROBLEM is that you cannot abuse this functionality, because once a lock is made on parent, the whole tree is completely blocked. Which means that concurrent requests to the same entities from another request will wait until the transaction holding the lock finishes, with a freaking timeout and an exception if it fails.
Even SQLite is smarter than that. Programming like it's freaking 1970.
---
Even on GAE you do have to keep a normalized copy of your data, unless your data is really dumb. That's because you do not know the kind of queries you will do in advance and it is better to have a single-source of truth that is normalized and then build views asynchronously around that for whatever queries you want ... but in doing so, adding a single dumb view to your app is a huge PITA.
And even denormalization is a problem, because surely you can store the comments on an article for a stupid blog in a single entity as a list, or tags, or whatever, but then GAE places hard limits on the amount of data stored in a single entity (1 MB or something). Every step of the way you'll have to think about the query you want and about how big your dataset is. You just can't throw the data in there and optimize it later.
In case of MySQL missing an index your app will just be slower. But in case of GAE, it won't run.
---
Many simple queries become really complex, like in case you want to do a radius search, for which with MYSQL you can just do a stupid haversine-based search like lat BETWEEN (LatX, LatY) AND lon BETWEEN (LonX, LonY), but this is not possible to optimize on GAE and instead have to go and experiment with bounding box searches.
And even if you have a perfectly optimizeable query, GAE places hard limits on what you can retrieve. For example you can retrieve only the first 5000 items of a dataset (that number is up from 1000 last time I tried it). This is again problematic -- with MySQL sometimes I want the first 20000 items and it isn't a problem. What if you want to update all items in an entity somehow? What if you want not the first 5000 items in an index, but the last 5000, something which isn't a problem with a classical RDBMS? Yeah, you have to do special gymnastics.
Go ahead and try building a classic admin, like the one Django provides, in which you have a paginated view of all the data you have in a single table, with filters applicable and ordering on any column you want. Something that is trivial in a classical environment will suck your soul out until your eyes will bleed.
---
GAE puts hard limits on the amount of time allocated for each request or background task. And surely they raised those limits but it is not enough.
The fact of the matter is that you have absolutely no guarantee that processing from 1 to X items from the datastore will succeed in the time allocated. When I had to solve this, a background task sometimes processed 500 entities in a row, sometimes it managed to ...
I think bad_user covers most of it, (although to be fair misses the mark on some things too; there is no 5000-entity limit per query if you use the query as an iterator rather than calling fetch(), which is a much better idea generally) but it's not that deployment itself is difficult--that's the only easy part, frankly.
What's tedious is development, due to all the restrictions and extra concerns necessary. They may make total sense for Google but make zero sense for the rest of the world's applications.
As one simple example, everything you need to be consistent must be handled in a manual transaction but transactions are slow and prone to collisions with insane time-outs (e.g. a request takes 45 seconds because a transaction couldn't commit the first time). Don't even get me started on cross-entity transactions or parent/child relationships, both of which you can use to completely destroy all semblance of performance in an application. Something you take for granted every day on any other platform (transparent, fast consistency guarantees) is of constant consideration and concern on AppEngine--and the APIs presented for it are pedestrian.
These are not intractable problems; none of AppEngine's shortcomings are, really. But the huge number of man-hours spent dealing with its terrible APIs, restrictions, and astounding pre-optimization requirements are hours that could have been spent self-managing a much more user-friendly platform.
I was skeptical of the new pricing plan at first, but then I realized it was introducing more accurate price signals to users. This helped a lot of people realize that their apps were doing really stupid things. I argue this here:
Python 2.7 may be present in the SDK (which is great!), but it's still in experimental mode:
"Unfortunately, being on the bleeding edge means that we may make backwards-incompatible changes. We will inform the community once the Python 2.7 runtime is no longer experimental."
From my POV google app engine is still the only PaaS offering that can scale the data-tier. Scaling the application instances is a lot easier and most other PaaS providers struggle with that too.
I think if you look at the big picture GAE is as good as it gets at the moment and there is quite a gap to the second place when it comes to automatic scaling.
Am I missing something? Disclaimer: we are very happy and excited GAE customers with https://blossom.io
I'm a GAE user for a personal app -- basically I use it for serving comments on my personal blog, similar to whatever Disqus is doing but I prefer the control. I also built on it 2 other personal apps that I later shut down.
IMHO, GAE is great for quick prototypes. Even for apps built for personal usage is great as the free quota is still reasonable, and compared to shitty PHP hosting services it's heaven-like. As far as the free quota goes it's also a lot better than Heroku.
However, I feel very restricted on GAE. My current project at work that pays my bills would never fit into it. Even if it did, because of the stuff we do asynchronously and because of how Google charges for API calls, the bills would go through the roof.
the only PaaS offering that can scale the data-tier
Yes it is. However, you can also scale the data-tier by yourself. Yes, it takes engineering, but it really isn't that difficult if you're planning ahead a little, which you have to do with GAE anyway. I think this old article from 2009 about how Friendfeed used MySQL to do sharding is a classic by now, but in case you haven't read it: http://bret.appspot.com/entry/how-friendfeed-uses-mysql
My biggest problem with GAE is that it is hard to migrate from when it no longer suits your needs. Not impossible, not extremely difficult, but difficult nonetheless taking away precious resources you'd rather invest in something else.
But as far as PaaS goes if you're in to that, it works great.
20 comments
[ 5.5 ms ] story [ 23.3 ms ] threadAnd pretty much all the other vendors use plain old SQL databases and don't have everything as a platform-specific API like App Engine does. So when the prices went up, my guess is that people realized, "Hey, why should I lock myself in to this one vendor that could raise the prices again, when I can go with other vendors that would be easier to migrate between [or switch to hosting myself later]?"
The team is very friendly, and they have an incredibly generous free quota as compared to other services (2 GB of files/database, 5 GB/month of bandwidth, and 16 MB of Redis - per app). They use plain old WSGI for the server, Postgres for the database, and they'll install any packages you put in a requirements.txt file, so no lockin. Haven't used Django with it, but so far my experiences with Flask are great.
http://devcenter.heroku.com/articles/python
There's even extra love for Django specifically:
http://devcenter.heroku.com/articles/django
Absolutely no lock in -- Heroku runs any WSGI app with a completely standard Python/virtualenv/pip runtime. It uses Python 2.7.2 currently.
And of course, the entire Heroku ecosystem is available, such as PostgreSQL and Redis databases, New Relic analytics, etc.
I don't mind App Engine that much though, to be honest. The price changes don't really affect me and I've always been happy with their infrastructure. The only thing that bugs me about using it is vendor lock-in.
Google has gone to extreme lengths to make every company on the internet their enemy, and they've succeeded.
If this doesn't apply to you, choose an up-to-date, cheaper, more versatile, far less tedious deployment platform. I've spent the past 10 years developing, deploying, and maintaining web applications of most sizes and I've never had a harder time "scaling" usage of mere GB of stored data. Maybe if I had 100TB then AppEngine would make the "impossible possible"; for the rest of the 99.999999% of the Internet, it makes the formerly trivial insanely tedious.
appcfg.py update .
I don't understand the scaling comment either. How is it not scaling for you?
On one of my apps the only thing I have to do is literally "git push production release" and that takes care of everything.
On scaling it is tedious because things we take for granted are suddenly hard. Some example of that, but not limited to:---
The transactional support in GAE is really awful. Something which I take for granted when working in Django or Rails with a RDBMS: I want the writes I did to be reverted in case my request failed for some stupid reason and triggers an exception. It's nothing much, but I want this basic functionality that should be a no-brainer.
Transactions in GAE work on trees. Basically GAE puts a lock on a parent entity (every entity can have a parent entity) and every operation done on an entity with that parent will be inside the transaction (and reverted in case of rollback). But THE PROBLEM is that you cannot abuse this functionality, because once a lock is made on parent, the whole tree is completely blocked. Which means that concurrent requests to the same entities from another request will wait until the transaction holding the lock finishes, with a freaking timeout and an exception if it fails.
Even SQLite is smarter than that. Programming like it's freaking 1970.
---
Even on GAE you do have to keep a normalized copy of your data, unless your data is really dumb. That's because you do not know the kind of queries you will do in advance and it is better to have a single-source of truth that is normalized and then build views asynchronously around that for whatever queries you want ... but in doing so, adding a single dumb view to your app is a huge PITA.
And even denormalization is a problem, because surely you can store the comments on an article for a stupid blog in a single entity as a list, or tags, or whatever, but then GAE places hard limits on the amount of data stored in a single entity (1 MB or something). Every step of the way you'll have to think about the query you want and about how big your dataset is. You just can't throw the data in there and optimize it later.
In case of MySQL missing an index your app will just be slower. But in case of GAE, it won't run.
---
Many simple queries become really complex, like in case you want to do a radius search, for which with MYSQL you can just do a stupid haversine-based search like lat BETWEEN (LatX, LatY) AND lon BETWEEN (LonX, LonY), but this is not possible to optimize on GAE and instead have to go and experiment with bounding box searches.
And even if you have a perfectly optimizeable query, GAE places hard limits on what you can retrieve. For example you can retrieve only the first 5000 items of a dataset (that number is up from 1000 last time I tried it). This is again problematic -- with MySQL sometimes I want the first 20000 items and it isn't a problem. What if you want to update all items in an entity somehow? What if you want not the first 5000 items in an index, but the last 5000, something which isn't a problem with a classical RDBMS? Yeah, you have to do special gymnastics.
Go ahead and try building a classic admin, like the one Django provides, in which you have a paginated view of all the data you have in a single table, with filters applicable and ordering on any column you want. Something that is trivial in a classical environment will suck your soul out until your eyes will bleed.
---
GAE puts hard limits on the amount of time allocated for each request or background task. And surely they raised those limits but it is not enough.
The fact of the matter is that you have absolutely no guarantee that processing from 1 to X items from the datastore will succeed in the time allocated. When I had to solve this, a background task sometimes processed 500 entities in a row, sometimes it managed to ...
What's tedious is development, due to all the restrictions and extra concerns necessary. They may make total sense for Google but make zero sense for the rest of the world's applications.
As one simple example, everything you need to be consistent must be handled in a manual transaction but transactions are slow and prone to collisions with insane time-outs (e.g. a request takes 45 seconds because a transaction couldn't commit the first time). Don't even get me started on cross-entity transactions or parent/child relationships, both of which you can use to completely destroy all semblance of performance in an application. Something you take for granted every day on any other platform (transparent, fast consistency guarantees) is of constant consideration and concern on AppEngine--and the APIs presented for it are pedestrian.
These are not intractable problems; none of AppEngine's shortcomings are, really. But the huge number of man-hours spent dealing with its terrible APIs, restrictions, and astounding pre-optimization requirements are hours that could have been spent self-managing a much more user-friendly platform.
https://plus.google.com/116396240707733722472/posts/EHpaKDWM...
"Unfortunately, being on the bleeding edge means that we may make backwards-incompatible changes. We will inform the community once the Python 2.7 runtime is no longer experimental."
http://code.google.com/appengine/docs/python/python27/newin2....
I think if you look at the big picture GAE is as good as it gets at the moment and there is quite a gap to the second place when it comes to automatic scaling.
Am I missing something? Disclaimer: we are very happy and excited GAE customers with https://blossom.io
IMHO, GAE is great for quick prototypes. Even for apps built for personal usage is great as the free quota is still reasonable, and compared to shitty PHP hosting services it's heaven-like. As far as the free quota goes it's also a lot better than Heroku.
However, I feel very restricted on GAE. My current project at work that pays my bills would never fit into it. Even if it did, because of the stuff we do asynchronously and because of how Google charges for API calls, the bills would go through the roof.
Yes it is. However, you can also scale the data-tier by yourself. Yes, it takes engineering, but it really isn't that difficult if you're planning ahead a little, which you have to do with GAE anyway. I think this old article from 2009 about how Friendfeed used MySQL to do sharding is a classic by now, but in case you haven't read it: http://bret.appspot.com/entry/how-friendfeed-uses-mysqlMy biggest problem with GAE is that it is hard to migrate from when it no longer suits your needs. Not impossible, not extremely difficult, but difficult nonetheless taking away precious resources you'd rather invest in something else.
But as far as PaaS goes if you're in to that, it works great.
Are they serious?