This looks incredibly slick. Speaking as someone who is implementing all the ceremony (security, logging, etc) around a new API right now I would use this in a heartbeat.
Of course, in a couple years, assuming success, the AWS lockin will suck. But given the odds of success I think I'd take the chance.
You could always use an open source gateway like Tyk (https://tyk.io) or API Umbrella (full disclosure: I run Tyk.io, and this news is... disheartening).
Think of it as Amazon validating your business! It's yet another selling point (specially for orgs that can't or don't want to use AWS mind-boggling myriad of products).
Consider that AWS has always been incredibly expensive. If you can't compete with AWS on price, then sure, it should be disheartening, but then you should take a long hard look at why.
Otherwise let AWS grow the market, and look for opportunities to serve people who don't want to pay the AWS premium and/or to plug feature holes.
Azure has tended to be price comparable to AWS, but does offer some compelling options for storage, that I wish other providers did too.
My biggest complaint about MS/Azure is they can't seem to get single signon/off working right... Login to one account for personal email, another for work email, and another still for my personal Azure account... Sometimes gets stuck in a loop, just doesn't work, whatever... I often keep another browser, that I don't mind wiping all cookies on just to get the damned thing to work. Let alone the other day where my payment didn't go through for some reason, and my azure account was locked, as was my live account, that I had to go through a convoluted recovery process that was different because of 2fa, only then to have my email (application password) revoked... grrr.
Despite account issues, I've really liked actually working with Azure.. though I think their node tooling could use some TLC, and better documentation... wound up writing my own wrapper (azure-storage-simple) to have a cleaner interface... also wrote mssql-ng as a friendly abstraction on top of mssql.
The analytics on Tyk look amazing, definitely a step up from the internal tools I've built.
Is Tyk HIPAA compliant? I'd definitely like to use something like Tyk, but I manage an API that transmits health data, so lots of tools have turned out to be off-limits because of HIPAA issues.
I shouldn't see why not. Tyk has a pretty small footprint and supports SSL end-to-end so transmission of data is safe. It obfuscates API keys so they can't be pulled if you keystore is compromised and all analytics also use the obfuscated keys so that security is ensured in log data.
However, we'd probably need to do an audit. Worth a check though.
Thanks for posting this, Tyk actually looks great (and easier to understand than Kong or Amazon API Gateway).
We've been considering building something like this ourselves or looking for a good existing solution, but hadn't found anything yet. I'm definitely checking this out.
Flexibility mostly. Pricing you can just bake into you're business model.
Flexibility however is always something you run into with platforms. And while this is a bit different then what folks typically think of as a platform (Django, Rails, Play, Spring) it looks like it would suffer the same problems. As soon as you need to do something weird, you're stuck fighting the platform or porting out instead of making your project better. This is why I prefer platforms that are more a collection of libraries with a preferred way of doing then as opposed to rigid box box style things.
That makes sense. I'm usually a fan of AWS services, but this does look like the kind of thing that could land you in a world of hurt down the line, when there's something you need to do and you just can't without, say, moving completely off the API platform.
Flexibility and being able to easily add extra functionality is the underlying philosophy of Kong (https://github.com/Mashape/kong), of which I am one of the core contributors.
We are accepting pull-requests from the community for new Plugins (http://getkong.org/plugins), and I can see Kong running on top of the AWS Gateway -> Lambda integration to provide functionality that is not necessarily bound to the AWS realm.
API is an acronym, and the product is called "Amazon API Gateway". This submission title is bugging me more than it should. Sorry for the meta-comment.
Edit: The submission title has been changed since this comment was written.
Actually, its an initialism (an abbreviation from first letters where each letter is pronounced as a letter) rather than acronym (an abbreviation from first letters pronounced as if it were a word.) Were it an acronym, many style guides (particularly British ones, IIRC) would support presenting it as "Api" rather than "API".
I have used Apigee a bit. Amazon's product does seem to me a direct competitor to Apigee. For anyone already on Amazon's infrastructure it's a logical step to use another of their products than buy a service from a third party provider. Amazon's pay-as-you-go service is also very appealing as you might pay for a lot of features you don't use with a provider like Apigee.
I wish Lambda would allow listening to a socket [it helps binaries communicate with node]. This would move our team to use this without any further doubt.
I see this more of a replacement for parse which does a similar thing with cloud code both as far as I can tell there isn't a good place to keep the code / API layouts elsewhere.
The blog post mentions that you can describe your API using Swagger and then import that. I suppose it means you can keep your Swagger definitions under version control.
Yo! We're building this at https://stackhut.com . You can see what an example API looks like @ https://github.com/StackHut/pdf-tools . You `stackhut deploy` this and it pushes it to an endpoint you can call over JSON-RPC or HTTP. Would love to hear your feedback (or ping me at leo@stackhut.com if you want me to walk you through anything!)
API Managmeent market is quite big, makes sense for Amazon to get on board.
Imagine this: You are BigCorp USA Inc. and have a bunch of internal SOAP-based web services that are 10 years old and cost a fortune to maintain.
You hear someone will pay you to use them, but SOAP sucks. So you could either go and change your APIs (and probably break a bunch of internal processes) OR you could use a gateway to sit in front of them and translate in bound / outbound requests. Now you can charge people for your APIs!
Scenario 2: If you're an indie dev/ startup but dont want to run EC2 infrastructure, Lambda + Gateway = Cloud service, without any heavy lifting, servers and cost effective. It's development lego. Kind of like Parse.
Scenario 3: You have an API for your app, now you wnat third party devs to use it.Your API is not set up for that kind of access (multi-user security, throttling, quotas, analytics etc.) - solution: shove a gateway in front of it.
#1 is exactly what I'm working on right now; turns out there's a huge market for this sort of thing. We're using an API gateway (Axway/Oracle) to build an "API Mullet" (all SOAP in the back, all REST in the front), and while I have some qualms with the tooling, the idea of an API gateway is very sound.
Can't speak for the parent but having tackled similar efforts before I find that almost all APIs regardless of the implementation flavor (including SOAP and RPC) are readily understood as CRUD systems creating/reading/updating/deleting things), and the mapping from CRUD to REST is straightforward. In practice this generally means mapping from a SOAP call like `SetPostAuthor` to a REST call like `PUT /post/:post_id/author`. There are always some gnarly bits along the way, but the basic structure usually comes together pretty easily.
Yup, petty much. If you're willing to be a little flexible in your application of RESTful concepts and are okay with your abstractions leaking a bit, it's not especially tough.
A harder case is when you have a "service" layer which also has a large collection of queries at various levels of abstraction.
These map poorly into resources, where you want a single level of abstraction per resource. And they map badly to legacy consumers, who've grown comfortable to having a collection of special knobs and dials perfectly shaped for their exact needs.
I can't take credit for it, alas. It was from an Axway blog post. It's sort of backwards too, if you think about it - business in the back and party in front.
I don't know, the "front" is all business - ordered, structured, conforms to the standards. The back is a jumbled, messy, non conforming. It's a party back there.
It does suck a bit, I've also sent Layer 7 into production which is just as bad as Axway. Typical Vendorware.
The AWS Gateway is pretty enticing. Solves lots of the API Gateway problem space very quickly and super cheap. We could run our production instance for about $2 a month I reckon, vs a 6 figure sum for the vendor tech.
Gateway is pretty non-specific if you're talking Axway. It could mean XFB Gateway, Gateway Interchange or API Gateway... and I'm not even sure that's the whole list.
I meant XFB. It was the only product that was referred internally just as "Gateway". "Gateway Interchange" was "Interchange" and "API Gateway" was "API Gateway".
API management is not big. Is under $500M/year [1] - plus most of the features are becoming commodities. Take a look at the open source KONG API gateway [2]
Agreed that it's moving down the value chain, many other open source projects out there doing similar things. There's also Tyk Open Source API Gateway (https://tyk.io) and API Umbrella (http://apiumbrella.io/), and APIAxle, though that's been around for a while now.
Is small from a VC prospective. Especially in 2015-2020. Everything under 1B is tiny. Plus, when there are soo many players you will end up fighting like crazy just the get bread crumbs.
It's still an interesting proposition for new development... front end published to S3/Cloudfront, with the backend running in Lambda responders, and an API layer using this for a communication bridge... full scale flexibility with very low administrative overhead...
At a larger size, it may be more expensive than a roll your own, but for an independent idea that you just want to build something as a software/service and if successful be able to scale and handle extra load, it isn't a bad idea.
This is really the missing piece to Lambda in my mind... though being able to act as a bridge for legacy SOAP/WS-* services is pretty nice too. I've written a lot of pretty small node service that act as a bridge to SOAP based services the past few years... though most of those have been for internal use, and not sure how well that translates here.
Let's say you had 10 or so APIs that you wanted to monetise. You might want to bundle these up into groups that you sell. You might want institute different pricing tiers for different rates. To have to bundle all that into each application (along with communicating things like rate usage between them) might not be very appealing.
This removes infrastructure from the equation entirely. You already had AWS Lambda which could run small pieces of Java or Javascript code when certain events were fired from other services (all without running a server yourself). The missing piece was something that catches HTTP requests and passes them to a Lambda function. This service seems to fill that gap.
It means you do not need to think about x instances that serve your API anymore. You just upload your Lambda function, wire some API path to it and pay per request. For smaller APIs this will be probably be negligible. Could be a gamechanger!
Yeah that's why I'm excited about this. I have a small instance that just runs a sort of "node proxy" to lambda, it's cheap ($11 / month or something), but annoying to run just to be able to connect to lambda. Now I can get rid of that.
I don't see why they'd be obsoleted. If I have infrastructure already setup on say, Azure, and Azure comes out with something like this, I'm going to use that instead of AWS, simply because it's familiar territory (not to mention LAN-level latencies between the gateway and my other API servers)
Because they invested a lot of engineering effort in making an API gateway and judging by the speed of development for internal projects versus AWS development soon most if not all of its functionality will he surpassed by the AWS version.
And because everything for them is already built on top of AWS.
It depends on the features of the gateway, but you can add or centralize various functions and not have to make them a part of every API ever. So you save in not having to duplicate that code as well as being able to change it everywhere rather than having to fix one API at a time. Okay, sure, maybe you had a library or something that each API used, but this does make managing it very simple and simple is good, especially if you have hundreds or thousands of APIs.
In the API Gateway I work with allows you to do authentication within the gateway and then delegate responses to the back end API (which can still have its own security, if you really want it, but usually is just hidden on an internal network). It works pretty much the same for other things like rate limiting. It also gives you monitoring for each API, again cutting down on the things you have to add to every single API by encapsulating the function in the gateway.
Also, you can separate your internal API and your external APIs. Maybe you have to always support certain operations to the outside world, but internally you'd really like to change something. By having them separate, you can create a policy within the gateway to change how the internal calls are done without changing the interface as far as the outside world is concerned.
You can also do various types of conversion (e.g. REST->SOAP) or even more exotic things so that outside devs don't have to care just how weird your damned mainframe is or other aspects of your internal architecture that they shouldn't know about.
The target market is, effectively, anyone who wants to expose internal APIs to the outside world.
Source: I work on an API Gateway and this is how our product works.
Seems like a great product to quickly get started with a mBaaS in a powerful cloud like AWS. The concept looks really similar to StrongLoop's loopback [0] with a big difference: vendor lock-in. I like the openness that StrongLoop is bringing on this front... IMO the best solution is one that allows you to move your containerised API from cloud to another cloud.
That being said, having this as an option in AWS is pretty cool and potentially time-saving. I'll probably give it a shot soon.
I was looking for something like this. Lambda functions are amazing but restricted because they weren't easily consumable externally. This is they key.
In onl fly in the ultra-low cost ointment is caching
To put the minimum cache in front of your API costs more than running an EC2.micro
For my projected use where my volume are going to be so low that I will fall under the Free lamda tier and the very first API tier I don't need to worry about caching but for those that do the price will start to rack up.
At the AWS Re:Invent in Las Vegas today, Amazon Web Services today announced the deprecation of Elastic Compute Cloud as it shifts toward lighter-weight, more horizontally scalable services. Amazon announced that it was giving customers the opportunity to migrate toward what it claims are lower cost "containers" and "Lambda processes".
"We believe that the cloud has shifted and customers are demanding more flexibility," tweeted Jeff Barr, AWS Spokesperson. "Customers don't want to deal with the complexities of security and scaling on-instance environments, and are willing to sacrifice controls and cost management in order to take advantage of the great scale we offer in the largest public cloud."
Barr went on to add that since their acqui-hire of Heroku last year, AWS has decided that the future of the cloud was in Platform as a Service (PaaS) and is now turning its focus to user-centric SSH key and user management services like https://Userify.com.
Amazon (AMZN) stock was up $0.02 on the latest announcements.
Okay, at Google I found that
IAM abbreviates Amazon's
Identity and Access Management.
So, the OP has
an undefined three letter
acronym.
Suspicions confirmed: The OP
is an example of poor
technical writing.
Why do I care? I could be
interested in Amazon Web Services
(AWS) for my startup. But so far by
a very wide margin what has
been the worst problem in my
startup? And the candidates are:
(1) Getting a clear statement of
the business idea.
(2) Inventing the crucial,
defensible, core technology.
(3) Learning the necessary programming
languages.
(4) Installing software.
(5) System backup and restore.
(6) Making sense out of
documentation about computer
software.
May I have the envelope please
(drum roll)? And the winner is,
(6) Making sense out of
documentation about computer
software.
And the judges have decided that
uniquely in the history of this
competition, this selection deserves
the Grand Prize, never before given
and to be retired on this first
award, for the widest margin of
victory ever.
No joke, guys: The poorly written
documentation,
stupid words for really simple
ideas, has cost me literally
years of my effort. No joke.
Years. No exaggeration. Did
I mention years?
At this point, "I'm reticent.
Yes, I'm reticent." Maybe
Amazon has the greatest stuff
since the discovery and
explanation of the
3 degree K background
radiation, supersonic
flight, atomic power,
the microbe theory of
disease, electric power,
mechanized agriculture,
and sex, but if Amazon
can't do a good job,
and now I insist on a very
good job, documenting their
work, which is to be
yet another layer of documentation
between me and some microprocessors,
then, no, no thanks, no way,
not a chance, not even
zip, zilch, zero.
What might it take me to
cut through bad Amazon
documentation of AWS,
hours, days, weeks,
months, years, then
from time to time,
more hours, days, or weeks,
and then as AWS evolves,
more such time? What would
I need to keep my startup
progress on track,
500 hours a day? More?
All just to cut through
badly written documentation
for simple ideas, and worse
documentation for complicated
ideas?
First test: Any cases of
undefined terms or acronyms?
Result: About three such cases,
and out'a here. Gone. Done.
Kaput. I don't know what AWS
has, but I don't need it.
Sorry, AWS, to get me and my
startup as a user/customer,
you have to up your game
by several notches. The first
thing I will look at is
the quality of the technical
writing in your documentation.
And, I have some benchmarks for
comparison from J. von Neumann,
P. Halmos, I. Herstein, W. Rudin,
L. Breiman, J. Neveu, D. Knuth.
Amazon, for now, with me, from
your example of writing here,
you lose. Don't want it.
Can't use it. Not even for free.
I'm not going to invest my
time and effort trying to
cut through your poor technical
writing. And, the next time
I look at anything from AWS,
the first undefined term
and I'm out'a here again.
Yes, I'm hyper sensitive about
bad technical writing -- couldn't
be more sensitive if my fingers
and arms were burned off. Whenever
possible, I will be avoiding any
chance of encountering bad technical
writing, ever again. Period.
Clear enough?
More generally, my view is that
bad technical writing is the
worst bottleneck for progress
in computing.
Come on, AWS, up your game.
I don't want to run a server
farm and would like you to do
that for me, but neither do I
want to cut through more
bad technical writing -- for that
work, my project is already
years over budget.
OP is writing a technical blog on AWS, primarily aimed at existing AWS users. Defining what IAM means in there is about as pointless as you needing to define what OP meant in your post.
I'm just saying that there is
bad technical writing, that there's no
good reason for it, and that it's
darned costly.
If the that OP blog post was just
for existing, old AWS customers, then
find, good, okay, sure, of course,
but make that point clear, up front.
Else, Jeff Bezos, I'm building
a Web site; I've got the production
quality, reasonably high capacity,
scalable software written and running;
HP has offered me servers initially
for free;
I plan, intend, and
hope my Web site becomes a big thing;
I should consider a cloud instead
of my own server farm; AWS should
be a candidate; I'm close enough
now to going live that I should
be looking into AWS; I read the
OP to get started; I came away
from that post and its poor technical
writing torqued and wanting to
set aside AWS even if I was
offered it for free (for me
to work through enough
more AWS
bad technical writing
for my Web site
would be much more costly than
free, making a free service
still too costly),
that's
a bad situation for Jeff Bezos,
Amazon, AWS, and me.
There's a lot of absurdly
bad technical
writing in computing
for absurdly poor reasons
with absurdly high costs.
I should have been torqued
decades ago; I should be torqued
now, and I am.
I just want better technical writing,
in particular to save my
time, effort, and money.
For this instance, the blog just
needed a common statement
about "Who this information
is intended for".
For AWS to put out information about
their services with bad writing
that torques off
a highly serious and experienced
software developer is bad business
for both AWS and me.
I am just objecting to the costly
bad technical writing. Simple.
Should not be difficult,
complicated, or
controversial.
You seem to be mistaking your lack of knowledge of a particular set of technologies with bad writing. I read it and it was an absolutely clear article to me.
I want an article explaining how Amazon's API gateway works, not what all of Amazon's services are, what API gateways are, what REST is etc. There's already plenty of those out there. Not every article needs to explain every concept from scratch.
They even have a link to a page on what IAM is for those who don't know (admittedly it's not on the first mention, but that hardly makes the entire article badly written).
The issue here is not
me but some bad technical writing.
The OP failed to explain that
a prerequisite to reading the
post was being a
long time AWS user.
Then, the OP was bad
technical writing. Done.
QED.
Without such a warning about
prerequisites, the undefined
acronym was bad technical
writing.
> your lack of knowledge
of a particular
set of technologies
That whole concept is big mistake
common in computing.
The "technologies" in question
are, for at least a good first
description, just dirt simple
with meager prerequisites.
None of us in computing should
try to carry the whole pile of
simplistic trivia around between
our ears. For a related lesson,
after a while in academics, it
becomes obvious that only fools
try to carry the whole research
library around between their
ears -- the lesson is much stronger
for the less deep, less good
material in practical computing.
With good technical writing, nearly
anything in practical computing
can be explained plenty well enough
to nearly any of us.
I don't have a relevant "lack"
of anything in computing.
It wasn't undefined. There was a hyperlink to a page all about it.
You failed to explain what OP meant, and didn't explain that your comment was aimed at people who knew this kind of terminology. Does that make your comment bad writing?
No one is suggesting that people should carry every bit of technology around in their heads. Google is a great source for finding things you don't know. The alternatives - explaining in every single blog post the complete set of terms that you need to understand to read it - would make for massive, unreadable posts and would be 90% useless for most of the audience it is aimed at.
For my using OP in my post,
well, here on Hacker News (HN),
that is a standard abbreviation.
But, yes, good technical writing
would not do that without, say,
original post (OP) or some such.
> There was a hyperlink
Sorry, clearly that's just not
good enough since the first use
of the acronym did not have the link.
The standard in technical writing
is to take some steps to be
adequately clear on terminology
at or before the first, say, significant
or material, use of that technology
in the piece of writing.
> Google is a great source for finding things you don't know. The alternatives - explaining in every single blog post the complete set of terms that you need to understand to read it - would make for massive, unreadable posts and would be 90% useless for most of the audience it is aimed at.
Of course it would, but I've
suggested no such thing.
Instead, I was clear:
"With good technical writing, nearly anything in practical computing can be explained plenty well enough to nearly any of us."
Did I mention, my concern is
good technical writing?
I thought that somewhere in my
posts here I mentioned that
my concern was good technical
writing. To be more clear,
let me say, my concern is
good technical writing.
I want to be fully sure:
Did I say that my concern was
good technical writing?
For just what constitutes good
technical writing, I won't try
to give a course here and
will leave that
to other sources.
But, the good work in the
fields of science, technology,
engineering, and math (commonly
called the STEM fields)
has some really good examples
of good technical writing.
E.g., for science, see the
freshman physics texts by
Sears, etc. For technology,
see D. Knuth, if only his
The TeXBook. For engineering,
see, say, D. Luenberger, Optimization
by Vector Space Methods.
For math, see, say,
any of the more respected
texts for freshman calculus
going back at least 50 years.
Also see the texts of W. Rudin.
The core
problem I am addressing here
is that the technical
writing in practical computing
is way too often really badly
done with high costs for
nearly everyone involved,
really high costs.
Some of the comments here
defending the original post
are straining to defend
the bad technical writing,
making excuses for no good reason.
Let me say, likely the first
good rule in good technical writing
is to strain never but never
have undefined terms, acronyms,
and jargon; instead, on or before
the first use of any such,
have, for definitions, motivations,
discussions, examples, explanations
or at least links. E.g.,
write "Amazon Web Services (AWS)".
Just do that -- always for
the first use of the acronym AWS in
anything at all about AWS.
Just do it, always. Spend the extra
three words. Be clear. Remove
all doubt. Reassure the reader that
in the piece of writing they
just will not face undefined
gibberish. Assure the reader
that they don't need Google
searches to get prerequisites
for the piece.
Be easier to read.
For acronyms, the example
of API abbreviates both
application programming
interface and American
Petroleum Institute is
right on target -- for nearly
any three letters, there
are from several to many
three words they could abbreviate.
And, if there is no reasonable
way for the piece of writing
to be for a broad audience,
then up front say what the
intended audience and prerequisites
are.
This is dirt simple stuff. To
accept these lessons it should
be sufficient just to want
to communicate instead of intimidate.
Was chuckling at Herstein and Neveu above, but I must interject about Rudin here - I wasn't aware "terseness" was the largest principal component of your analysis of "good technical writing" - how about Apostol instead?
Congrats on some insight
into Rudin! He got less
terse over time. His
Real and Complex Analysis
is terrific -- hardly terse at all.
The only question
I had was, why regular Borel
measures? Sure, if I went back
and studied again where he used
the hypotheses I'd see it.
But in Rudin's favor, e.g.,
for his Principles,
it's
all there, and crystal clear,
very precise, and perfectly correct.
Yes, have to draw own figures,
at least between ears, maybe
on paper.
Students might be told that
one of the purposes of the
early material in Rudin's
Principles is to build a
really solid foundation for
discussing continuity and
uniform continuity. So, for
that, he wants compactness.
He uses that work to show that,
with continuity on a compact
set and, thus, uniform
continuity, the
Riemann (Riemann-Stieltjes)
integral exists.
Then near the end of the book
he shows that the uniform
limit of continuous functions
is continuous.
But, also he goes ahead and
shows that a function has
a Riemann integral if and only
if it is continuous everywhere
except on a set of Lebesgue
measure 0, without really
saying much about Lebesgue
measure.
I don't have much from Apostol.
I'm sure he could write a
book that competes with
Rudin's Principles and might
be easier for ugrads to read.
It happens that two weeks or
so I got out Rudin's Principles,
the third edition, and looked up
something and was struck at
just how beautifully, elegantly,
sparsely done the work actually is.
>For my using OP in my post, well, here on Hacker News (HN), that is a standard abbreviation
And for AWS, IAM is a standard abbreviation. That's my point.
>Sorry, clearly that's just not good enough since the first use of the acronym did not have the link.
I've already acknowledged that, but one minor error - putting the hyperlink on the second use of a term that you could locate in 5 seconds in Google is hardly going to make the whole thing unintelligible is it?
>Did I say that my concern was good technical writing?
You did. And you've backed it up with pretty poor non-technical writing and a single complaint about the lack of hyperlink on the first occurrence of a single acronym that 99% of those reading an AWS technical blog would already know.
> And for AWS, IAM is a standard abbreviation. That's my point.
That point fails: HN has a much
larger and more broad audience
than AWS. Still, actually, HN should
define OP -- I still don't have
a good source to know what it means
and have been only guessing. That
AWS has a smaller audience than
HN does not excuse omitting
definitions.
Again you are assuming that
readers of that OP blog
are "99%" experienced AWS
users, and that's not good.
If the blog were only for
such users, then I should
say so.
That I could look around to
unwind the acronym, later
in the text, elsewhere on
the page, on some other
pages of the blog,
elsewhere at AWS, at Google,
doesn't excuse anything.
I mentioned some authors with
some writing that is astoundingly
technical, and there terms
are defined as I described.
Computing is awash in bad
technical writing; I should
be torqued at it, and I am;
one of the worst problems
is poor handling of technical
terminology, jargon, and
acronyms; the blog post was
an example: Either (A)
define the acronyms on or
before first usage or
warn the audience that
there are prerequisites.
When you were a child you started out learning simple one syllable words, and speaking one word sentences.
When you were toddler you started stringing those words together, and learning how to form extremely basic sentences.
And eventually you learned to form complex sentences with complex ideas.
The same thing applies to learning computing terms. AWS is now 9 years old, and has 9 years of development and features to catch up on if you are just starting. Yes, it will take some effort to learn everything, and if you don't see the benefit in doing so then you don't have to.
But if you want to be an "adult" engineer rather than a "toddler" engineer it is necessary to be willing to learn some things, even if it isn't easy.
I strongly disagree: I'm under
no obligation to have
followed AWS for the past nine
years. Instead, in each field,
we should write in ways as self
contained, and make the minimum
assumptions, possible. In particular,
in technical writing, we should
not have undefined terms.
The research libraries are awash
with single texts that start
with meager prerequisites and
get to some quite advanced
and challenging material.
AWS can do so, also.
I've been in computing for a long
time, but there's no hope
for me to know all the jargon
of all parts of computing.
Mostly jargon is not conceptually
difficult, and the solution to the
challenge of jargon is simple --
just define it.
Parts of computing have long
enjoyed having a closed
priesthood that spoke in
three letter acronym gibberish.
IBM used to do this with
PCP, MFT, MVT, MVS, JCL,
DCB, IMS, DB2, CMS, SQL,
and many more. And there
were some four letter acronyms --
CICS, DASD, REXX.
The concepts
were mostly simple; the gibberish
was a secret language.
Once I gave a grad seminar
on computing and had a lot
of guests from industry.
IBM talked in their gibberish,
but none of the other vendors
did. The students caught on:
IBM wanted to intimidate people
and otherwise invite people
into their priesthood where they,
too, would understand the gibberish.
IBM was the least popular of
the vendors.
AWS has documentation at varying levels. If you are looking for documentation that starts at the absolute basics, without acronyms then you are probably wanting the "Getting Started" section of their documentation site: http://aws.amazon.com/documentation/
The linked article is basically a changelog for devs who have already been using the ecosystem for years, so no its not newbie friendly.
They would have been doing better
for their business if they
had made that point clear.
Again, my gripe is, better writing
is needed. Why? The writing now
too often results in absurdly high
costs for absurdly poor reasons.
The costs to me have me torqued.
I am not a Programmer. I put up simple websites, and like technology, and enjoy this field. That said, I probally shouldn't comment, but from day one of my computing journey; My biggest gripe with this field has been terrible technical writing. (I put up with it because it was all so new, and figured it will eventually get better.)
I can still remember picking up those 500 page computing books. Flipping through the phone book and putting it back on the shelf.
As to the overuse of acronyms; I think they are overused in this profession. It almost seems fashionable?
Jump to today, things have gotten better. I'm not sure the writing has gotten better, or I just have more presenters of the information(Internet)?
I don't know why you got down voted? The level of technical writing in this field is atrocious? I understand "these are tough concepts!". Yes, in many ways it's like learning a language! I get it. In most cases I'm glad someone took the time to write something down. Yes--I'm grateful.
I do have a problem with entities, with a lot of money/resources, whom publish terrible technical writting. It does seem like a big company, like Amazon, could have done a little better job conveying this product?
Much like for Multics, Algol, Smalltalk, Unisys, Plan-9, Modula-3, various LISPs, etc, the kinds of people who care about bad technical writing probably have a budget that allows them to pay for more than AWS.
Smart move for AWS, but they are not innovating nothing here, just following. Late.
Companies like Apigee/Mashape/3scale/Mulesoft have been doing cloud API management in various forms since 2008. Even Microsoft Azure has an API management offering since two years.
Nowadays all those API gateway features are commodities and doesn't make sense to pay for it anymore. Indeed Open Source projects such as KONG [1] are getting tremendous tractions. Same things happened in search with all cloud solutions and then ElasticSearch came out and was game over.
This is very interesting, and I am surprised it didn't happen a long time ago. The Lambda function integration opens up lots of new ideas when building API backends ready to be consumed by clients apps like, for example, a client-side Javascript application.
On the other side it seems like other extra functionality is limited and very AWS-oriented. If you are looking for an open source gateway that can be expanded with extra functionality, and potentially sit on top of the AWS-> Lambda integration, take a look at https://github.com/Mashape/kong (I am a core maintainer too, so feel free to ask me any question). Kong is accepting plugins from the community (http://getkong.org/plugins), which can be used to replace or extend functionality beyond any other gateway including AWS Gateway.
I think AWS is going after the Parse / Heroku market here, the combination of Lambda and this gateway make it more suited for that kind of implementation, they are missing out on the larger features that enterprise needs - the blog post doesn't really go into detail on how SOAP conversion works, and I doubt it's a cakewalk.
It's in legacy API services that places like Mashery and Apigee make all their real money, and I can't see large enterprises shifting their legacy services to EC2. It's quite forward looking. The satire on how EC2 may be deprecated is really not far off from where things are heading.
I always find it quite funny when only single OSS projects are mentioned as alternatives, there's so many more full-featured projects out there: Tyk (https://tyk.io) API Umbrella (http://apiumbrella.io/) and ApiAxle (http://apiaxle.com/) to name some. For more fine grained approaches there's also Vulcan (http://vulcanproxy.com/)
vulcan is not for api management and beside that quality > quantity
What do you mean ? With vulcand you can implement your own middleware similar to kong. Also, it is written in go and you can use all existing go libraries.
Also, how can you gracefully reload kong if you need to add/remove/change a plugin ? With vulcand you just replace the binary, and send an USR2 signal to the running process. It will fork, wait for all connections to drain, and remove the old process
For deployment, again, you need to sync all lua files, with vulcand you just ship a compiled binary.
Kong doesn't have a notion of servers for each API, you need to forward requests to a haproxy or another load balancer for this. Also, I can see that backends are added by their DNS hostname. In order to achieve HA (backend redundancy) is there any way you can do it, assuming that nginx is caching the upstream dns values ?
About performance, I can see you are advertising about 1000 r/s using kong, and you need 3 machines for this (kong, cassandra, haproxy). I benchmarked vulcand and obtained about 12000 r/s on a more modest hardware.
"What do you mean? With vulcand you can implement your own middleware similar to kong."
Nginx is not an API management layer, right? It doesn't have the necessary features, but it's extensible so someone could build them. Mashape did, and they released the result as Kong.
Vulcand is in the same position as nginx. It could be a good choice for building an API management layer on, but it doesn't provide one out of the box.
> How can you gracefully reload kong if you need to add/remove/change a plugin?
You can gracefully reload Kong by executing "kong reload" (http://getkong.org/docs/0.3.2/cli/#reload). It pretty much works the same way: fork, wait for existing connections to drain, and remove the old master process.
> For deployment, again, you need to sync all lua files, with vulcand you just ship a compiled binary.
Kong has a few distribution options (rpm, deb, Docker, etc) that simplify the process (http://getkong.org/download/), so unless it's being built from source, the deployment is straightforward.
> Kong doesn't have a notion of servers for each API, you need to forward requests to a haproxy or another load balancer for this. Also, I can see that backends are added by their DNS hostname. In order to achieve HA (backend redundancy) is there any way you can do it, assuming that nginx is caching the upstream dns values ?
You are correct. Kong supports either a DNS or IP address for a backend service, and starting from 0.5.0 it will be possible to add more DNS addresses or IPs per each API. That will make Kong work in load-balancing mode (https://github.com/Mashape/kong/issues/157).
> About performance, I can see you are advertising about 1000 r/s using kong
Kong is basically built on top of nginx (a very solid core), and it can achieve a similar performance to nginx because all the data requested from the external datastore is being cached in memory. So pretty much the latency it's going to be nginx + Lua execution overhead (which is minimal, thanks to LuaJIT). I will run a better benchmark and write a blog post soon, trying to cover both single-node and multi-datacenter setups.
> and you need 3 machines for this (kong, cassandra, haproxy)
An external datastore (Cassandra - with plans to support PostgreSQL) is required because Kong has been built to scale multi-node and multi-datacenter in order to handle pretty much every use case and plugin/middleware. In very simple single-node use cases, Cassandra can live in the same instance along with Kong. I would say that in this regard Kong is inspired by a different philosophy (start small, grow big, even supporting hybrid cloud/bare-metal setups). Having an external datastore also means that Kong could support some pretty cool plugins, like multi-datacenter service discovery, API billing coordination, multi-region health checks, etc. For example today it is possible to rate-limit requests in an eventually consistent replication fashion across multiple data-centers.
I have worked with L7 for a client which offers something similar. The whole thing was painful in a way.
The SOAP to JSON conversion is easy for baby SOAP or JSON requests using transformation. For larger ones they turned out to be complex.
Once you start with client started with using the "SOAP" converted to JSON apis, there were new feature kind of requests. Now the problem is somebody needs to be dig into that old code mess of SOAP web services. Or write new services and split requests between old and new services.
There will be performance issues because of the no of transformation doing on top of the request. Ultimately has to write plugins on top write performant transformation in a single piece.
Personally SOAP to JSON conversion will not be as simple as move to AWS, although I will assume Amazon will do far far better job on this than others.
Therein lies the rub - if you are working with AWS' key-value store, and then also their Lambda system, then you don't really develop server side - you develop everything client side (mobile app) and just use the cloud as a dumb logic store.
Well, you are still free to write resource consuming operations and deploy it to an EC2 instance and call it from Amazon Gateway. Lambda is just an example.
Moreover I feel Lambda can do pretty good stuff. I am sure it will integrate with other AWS services pretty soon and once there is connectivity to an RDS instance, then it solves the use case for half the developer population out there.
One of the other benefits of using Cloudfront based endpoints is that your app servers behind it can avoid the TCP handshakes that add some latency. Amazon did an interesting presentation at re:Invent on the performance improvement from using Cloudfront ahead of dynamic requests that was eye opening.
They say "If you already utilize OAuth tokens or any other authorization mechanism, you can easily setup API Gateway not to require signed API calls and simply forward the token headers to your backend for verification." It would be nice if AWS would stand up an authentication service that could handle oauth. Or do they already have such a thing?
They say we can use Amazon Cognito[0]. I hope they put some examples with ACL and Role management (I wonder if this could live in lambda functions using packages like node_acl)
Please can people give examples of what they're using lambda for. Everything I've seen has been really basic (like image scaling), but most things I think of require a database.
Compiling LilyPond scores (http://lilybin.com). Usually we have at most 1 user at any time, but we can also handle bursts, like when a classroom of 30 students all use it at the same time. Ends up even cheaper than running one VM that can only handle 2–3 simultaneous users.
Image processing primarily: user uploads an image to S3, a thumbnail version gets created automatically. Now, we'll probably put it behind some of our APIs that see huge swings in traffic. It'll be nice to not pay to be running even a single x-large instance at night when the API is only seeing a few hundred hits per hour.
For heavy users of AWS services (not just EC2, but fancy SaaS/PaaS stuff) do you ever regret being locked in to a hosting provider? Does it restrict your ability to develop locally? Have you been bitten by problems that you can't resolve because you don't own the product? Or do you pretty much just love it?
We had cases where failing MySQL queries[1] would leave behind GigaBytes of garbage data that we (as users of Amazon's RDS) have to pay for and have no option of removing (besides killing the entire database and rebuilding a new one, which would cost us more than just living with the garbage). Having our own small EC2 instance with MySQL on it would have saved us quite a few hours of work/downtime.
[1] Have a database of a few gigabytes of sizes, start an ``ALTER TABLE`` and wait until your pre-allocated disk space is filled up. The query dies and you just fucked yourself. Can happen unexpectedly: We tried to remove a non-existing index on a table and MySQL decided to FIRST copy the entire table and THEN report that there is no such index.
FWIW, we absolutely loved it at the last place I worked. We were all too happy to let AWS host our Postgres instance (RDS), our static assets (S3 and CloudFront), and a bunch of other things.
We weren't particularly locked in, either. We always had the option to run our own Postgres cluster (it just didn't make sense to spend the manpower on it for us). We could have hacked together our own CDN, but that wasn't our core business focus.
For small companies with low manpower, build on the services and get things done fast and as well as possible. It may cost more upfront than doing it on your own, but you are probably going to come out ahead by minimizing your maintenance/security/administration surface area.
It sounds like you were using relatively loosely coupled AWS services. RDS doesn't change your programming interface to the database, it just hosts it. If you needed to, you could switch to a non-RDS Postgres server with only a configuration change to your core application.
Contrast this with using AWS DynamoDB, which would likely be a major overhaul if you needed to switch to something that could be used outside of AWS.
Haha, I just posted a nearly identical reply to another commenct. We actually just selected SimpleDB over DynamoDB for a new project specifically because it supports JPA annotations instead of a custom SDK. We know we're going to have to transition to a private datacenter in the near future so we have to avoid lock-in. SimpleDB beat out RDS just on price.
> Do you ever regret being locked in to a hosting provider?
> Does it restrict your ability to develop locally?
No, we use Vagrant so that we can run our docker cluster either locally or on Amazon. We have stubbed alternatives for all AWS services, including SNS and SQS, allowing us to run the entire backend locally even without an internet connection.
> Have you been bitten by problems that you can't resolve because you don't own the product?
I'd say its kind of like coding in Ruby on Rails. If you are bit by a problem you can't resolve there is a high chance you are doing something wrong, or at least not the "one true way".
Sometimes we have issues when exploring the edges of what is possible on the AWS ecosystem (we've been using the fresh-out-of-beta ECS, and running into a lot of limits there), but generally the basic stuff like S3, SNS, SQS, Elasticache, etc just works, and doesn't require any attention. It's just a working, hands off gear in your system.
> Do you pretty much just love it?
Sure. It's a lot better than racking physical hardware, and the deep integration between things like autoscaling groups, EC2, ELB, etc makes it easy to automate all the things. In the end the more our service is a machine that can run itself and repair itself (replacing its own instances as needed) the more I love it.
We moved our whole stack from manually managed Linode instances to AWS Opsworks, Elasticache, RDS, etc. We are currently somewhat tied now to the AWS stack, though through a large collection of our own Chef scripts I imagine we could move to a manually setup chef setup ourselves.
Largely we've been very pleased with AWS, but the black box nature of a number of products (particularly Elasticache and RDS), has caused us some minor grief. Also, the higher frequency of single instance failures (which our stack can handle pretty gracefully) is somewhat annoying.
EDIT: We were also able to fairly significantly lower our bill with reserved instances while also significantly increasing our ability to handle peak capacity with AWS.
I would think Elasticache and RDS would be the least of your worries since they're just fully managed versions of readily available software. You can always host your own Redis and SQL database. Something like DynamoDB would be way more complicated.
Really the issue with Elasticache and RDS is that when they go down and fail over to new hardware, we have no idea what happened to cause an (admittedly brief) outage.
Basically, we just open a support ticket and say "what happened", and they just say "oh, the machine failed and was rebooted"
I can't help but notice that this looks more like an enterprise integration tool (think mulesoft) than API management (think apogee.. or I think that is what they do).
Speaking somewhat from experience (webmethods, caml, spring-integration and various other enterprise integration tools) they always want you to use "their" DSL which is often not even stored on the developers filesystem (ie webmethods... you submit the code to the "broker" or "rules engine" or "router"... lots of words for thing that does the work). Which leads to very awkward development.
Consequently I wonder if they will have git integration because writing code even snippets in a web form no matter how nice gets old fast.
I am surprised that Amazon did not add support to more API description formats like RAML or apiblueprint. It is such a key feature. If I wanted to use this service in front of existing APIs, even only one API, I would not want to go through the work of having to redefine all my endpoints through a web form!
Shameless plug: after working on several API projects, I have been researching ways to not have to "code" over and over again what goes into creating endpoints, it became so repetitive. Lately, I turned to RAML (Yaml for REST) and, with 4 other developers, we created an opensource project called Ramses. It creates a fully functional API from a RAML file. It is a bit opinionated but having to "just" edit a Yaml file when building a new API simplified my life. As a bonus, I also get a documentation and a javascript client generated from the same file.
swagger 2.0 spec and RAML are almost the same thing. There's some minor differences but I would guess you could import the RAML into AWS and it would work
198 comments
[ 47.6 ms ] story [ 3521 ms ] threadOf course, in a couple years, assuming success, the AWS lockin will suck. But given the odds of success I think I'd take the chance.
Otherwise let AWS grow the market, and look for opportunities to serve people who don't want to pay the AWS premium and/or to plug feature holes.
My biggest complaint about MS/Azure is they can't seem to get single signon/off working right... Login to one account for personal email, another for work email, and another still for my personal Azure account... Sometimes gets stuck in a loop, just doesn't work, whatever... I often keep another browser, that I don't mind wiping all cookies on just to get the damned thing to work. Let alone the other day where my payment didn't go through for some reason, and my azure account was locked, as was my live account, that I had to go through a convoluted recovery process that was different because of 2fa, only then to have my email (application password) revoked... grrr.
Despite account issues, I've really liked actually working with Azure.. though I think their node tooling could use some TLC, and better documentation... wound up writing my own wrapper (azure-storage-simple) to have a cleaner interface... also wrote mssql-ng as a friendly abstraction on top of mssql.
Is Tyk HIPAA compliant? I'd definitely like to use something like Tyk, but I manage an API that transmits health data, so lots of tools have turned out to be off-limits because of HIPAA issues.
However, we'd probably need to do an audit. Worth a check though.
We've been considering building something like this ourselves or looking for a good existing solution, but hadn't found anything yet. I'm definitely checking this out.
Flexibility however is always something you run into with platforms. And while this is a bit different then what folks typically think of as a platform (Django, Rails, Play, Spring) it looks like it would suffer the same problems. As soon as you need to do something weird, you're stuck fighting the platform or porting out instead of making your project better. This is why I prefer platforms that are more a collection of libraries with a preferred way of doing then as opposed to rigid box box style things.
We are accepting pull-requests from the community for new Plugins (http://getkong.org/plugins), and I can see Kong running on top of the AWS Gateway -> Lambda integration to provide functionality that is not necessarily bound to the AWS realm.
Edit: The submission title has been changed since this comment was written.
Actually, its an initialism (an abbreviation from first letters where each letter is pronounced as a letter) rather than acronym (an abbreviation from first letters pronounced as if it were a word.) Were it an acronym, many style guides (particularly British ones, IIRC) would support presenting it as "Api" rather than "API".
I doen't have experience with their product, but on surface they look similar.
http://blog.0x82.com/2014/11/24/aws-lambda-functions-in-go/
What tools are there to allow me to keep my code/API layouts in source control when uploading to this?
I'm sure they exist somewhere, so mostly curious about pointers. (A sample node project using this would probably go a long way)
I've got an API already running. What does this buy me?
Caching? I can see some benefit there it's read heavy.
Auth and access control? Feels like that's part of my app code but maybe there's a benefit I'm missing
A lot of the other benefits also feel like it would be hard to cleanly separate them from my app code.
What's the elevator pitch and who's the target market?
Imagine this: You are BigCorp USA Inc. and have a bunch of internal SOAP-based web services that are 10 years old and cost a fortune to maintain.
You hear someone will pay you to use them, but SOAP sucks. So you could either go and change your APIs (and probably break a bunch of internal processes) OR you could use a gateway to sit in front of them and translate in bound / outbound requests. Now you can charge people for your APIs!
Scenario 2: If you're an indie dev/ startup but dont want to run EC2 infrastructure, Lambda + Gateway = Cloud service, without any heavy lifting, servers and cost effective. It's development lego. Kind of like Parse.
Scenario 3: You have an API for your app, now you wnat third party devs to use it.Your API is not set up for that kind of access (multi-user security, throttling, quotas, analytics etc.) - solution: shove a gateway in front of it.
How do you go about proxying REST style requests to a SOAP backend?
A harder case is when you have a "service" layer which also has a large collection of queries at various levels of abstraction.
These map poorly into resources, where you want a single level of abstraction per resource. And they map badly to legacy consumers, who've grown comfortable to having a collection of special knobs and dials perfectly shaped for their exact needs.
It does suck a bit, I've also sent Layer 7 into production which is just as bad as Axway. Typical Vendorware.
The AWS Gateway is pretty enticing. Solves lots of the API Gateway problem space very quickly and super cheap. We could run our production instance for about $2 a month I reckon, vs a 6 figure sum for the vendor tech.
[1]https://www.forrester.com/Sizing+The+Market+For+API+Manageme...
[2] https://github.com/mashape/kong
Also API market is intended to grow to $660 million in US by 2020 and over a billion worldwide (http://blogs.forrester.com/michael_yamnitsky/15-06-07-the_ap...). So it's definetely not small.
At a larger size, it may be more expensive than a roll your own, but for an independent idea that you just want to build something as a software/service and if successful be able to scale and handle extra load, it isn't a bad idea.
This is really the missing piece to Lambda in my mind... though being able to act as a bridge for legacy SOAP/WS-* services is pretty nice too. I've written a lot of pretty small node service that act as a bridge to SOAP based services the past few years... though most of those have been for internal use, and not sure how well that translates here.
It means you do not need to think about x instances that serve your API anymore. You just upload your Lambda function, wire some API path to it and pay per request. For smaller APIs this will be probably be negligible. Could be a gamechanger!
Rather than running an API server 24/7, you can write your API as a set of functions that cost you money only when they're actually invoked.
If you have a large cloud platform, an API gateway is priceless.
Especially if you have single sign on and therefore can use auth tokens for everything.
And because everything for them is already built on top of AWS.
In the API Gateway I work with allows you to do authentication within the gateway and then delegate responses to the back end API (which can still have its own security, if you really want it, but usually is just hidden on an internal network). It works pretty much the same for other things like rate limiting. It also gives you monitoring for each API, again cutting down on the things you have to add to every single API by encapsulating the function in the gateway.
Also, you can separate your internal API and your external APIs. Maybe you have to always support certain operations to the outside world, but internally you'd really like to change something. By having them separate, you can create a policy within the gateway to change how the internal calls are done without changing the interface as far as the outside world is concerned.
You can also do various types of conversion (e.g. REST->SOAP) or even more exotic things so that outside devs don't have to care just how weird your damned mainframe is or other aspects of your internal architecture that they shouldn't know about.
The target market is, effectively, anyone who wants to expose internal APIs to the outside world.
Source: I work on an API Gateway and this is how our product works.
That being said, having this as an option in AWS is pretty cool and potentially time-saving. I'll probably give it a shot soon.
[0] https://strongloop.com/node-js/api-platform/
I was looking for something like this. Lambda functions are amazing but restricted because they weren't easily consumable externally. This is they key.
To put the minimum cache in front of your API costs more than running an EC2.micro
For my projected use where my volume are going to be so low that I will fall under the Free lamda tier and the very first API tier I don't need to worry about caching but for those that do the price will start to rack up.
I'm not sure about the costs involved with the caching options... just the same, I think this option is pretty cool.
[1] https://cloud.google.com/endpoints/
1. What are the differences between this + AWS lambda and parse? Is there additional functionality or freedom with this route? Is it cheaper?
2. What kind of savings could one expect hosting an API with this vs a heroku standard dyno?
November 3, 2017, SEATTLE
At the AWS Re:Invent in Las Vegas today, Amazon Web Services today announced the deprecation of Elastic Compute Cloud as it shifts toward lighter-weight, more horizontally scalable services. Amazon announced that it was giving customers the opportunity to migrate toward what it claims are lower cost "containers" and "Lambda processes".
"We believe that the cloud has shifted and customers are demanding more flexibility," tweeted Jeff Barr, AWS Spokesperson. "Customers don't want to deal with the complexities of security and scaling on-instance environments, and are willing to sacrifice controls and cost management in order to take advantage of the great scale we offer in the largest public cloud."
Barr went on to add that since their acqui-hire of Heroku last year, AWS has decided that the future of the cloud was in Platform as a Service (PaaS) and is now turning its focus to user-centric SSH key and user management services like https://Userify.com.
Amazon (AMZN) stock was up $0.02 on the latest announcements.
So, the OP has an undefined three letter acronym.
Suspicions confirmed: The OP is an example of poor technical writing.
Why do I care? I could be interested in Amazon Web Services (AWS) for my startup. But so far by a very wide margin what has been the worst problem in my startup? And the candidates are:
(1) Getting a clear statement of the business idea.
(2) Inventing the crucial, defensible, core technology.
(3) Learning the necessary programming languages.
(4) Installing software.
(5) System backup and restore.
(6) Making sense out of documentation about computer software.
May I have the envelope please (drum roll)? And the winner is,
(6) Making sense out of documentation about computer software.
And the judges have decided that uniquely in the history of this competition, this selection deserves the Grand Prize, never before given and to be retired on this first award, for the widest margin of victory ever.
No joke, guys: The poorly written documentation, stupid words for really simple ideas, has cost me literally years of my effort. No joke. Years. No exaggeration. Did I mention years?
At this point, "I'm reticent. Yes, I'm reticent." Maybe Amazon has the greatest stuff since the discovery and explanation of the 3 degree K background radiation, supersonic flight, atomic power, the microbe theory of disease, electric power, mechanized agriculture, and sex, but if Amazon can't do a good job, and now I insist on a very good job, documenting their work, which is to be yet another layer of documentation between me and some microprocessors, then, no, no thanks, no way, not a chance, not even zip, zilch, zero.
What might it take me to cut through bad Amazon documentation of AWS, hours, days, weeks, months, years, then from time to time, more hours, days, or weeks, and then as AWS evolves, more such time? What would I need to keep my startup progress on track, 500 hours a day? More? All just to cut through badly written documentation for simple ideas, and worse documentation for complicated ideas?
First test: Any cases of undefined terms or acronyms?
Result: About three such cases, and out'a here. Gone. Done. Kaput. I don't know what AWS has, but I don't need it.
Sorry, AWS, to get me and my startup as a user/customer, you have to up your game by several notches. The first thing I will look at is the quality of the technical writing in your documentation. And, I have some benchmarks for comparison from J. von Neumann, P. Halmos, I. Herstein, W. Rudin, L. Breiman, J. Neveu, D. Knuth.
Amazon, for now, with me, from your example of writing here, you lose. Don't want it. Can't use it. Not even for free. I'm not going to invest my time and effort trying to cut through your poor technical writing. And, the next time I look at anything from AWS, the first undefined term and I'm out'a here again.
Yes, I'm hyper sensitive about bad technical writing -- couldn't be more sensitive if my fingers and arms were burned off. Whenever possible, I will be avoiding any chance of encountering bad technical writing, ever again. Period. Clear enough?
More generally, my view is that bad technical writing is the worst bottleneck for progress in computing. Come on, AWS, up your game.
I don't want to run a server farm and would like you to do that for me, but neither do I want to cut through more bad technical writing -- for that work, my project is already years over budget.
Either that, that is, really poor business thinking, or, as I concluded, some poor technical writing. I don't believe the first.
Are you saying that you don't believe the Amazon technical blog is aimed primarily at existing AWS users?
If the that OP blog post was just for existing, old AWS customers, then find, good, okay, sure, of course, but make that point clear, up front.
Else, Jeff Bezos, I'm building a Web site; I've got the production quality, reasonably high capacity, scalable software written and running; HP has offered me servers initially for free; I plan, intend, and hope my Web site becomes a big thing; I should consider a cloud instead of my own server farm; AWS should be a candidate; I'm close enough now to going live that I should be looking into AWS; I read the OP to get started; I came away from that post and its poor technical writing torqued and wanting to set aside AWS even if I was offered it for free (for me to work through enough more AWS bad technical writing for my Web site would be much more costly than free, making a free service still too costly), that's a bad situation for Jeff Bezos, Amazon, AWS, and me.
There's a lot of absurdly bad technical writing in computing for absurdly poor reasons with absurdly high costs. I should have been torqued decades ago; I should be torqued now, and I am.
I just want better technical writing, in particular to save my time, effort, and money. For this instance, the blog just needed a common statement about "Who this information is intended for".
For AWS to put out information about their services with bad writing that torques off a highly serious and experienced software developer is bad business for both AWS and me.
I am just objecting to the costly bad technical writing. Simple. Should not be difficult, complicated, or controversial.
I want an article explaining how Amazon's API gateway works, not what all of Amazon's services are, what API gateways are, what REST is etc. There's already plenty of those out there. Not every article needs to explain every concept from scratch.
They even have a link to a page on what IAM is for those who don't know (admittedly it's not on the first mention, but that hardly makes the entire article badly written).
The OP failed to explain that a prerequisite to reading the post was being a long time AWS user. Then, the OP was bad technical writing. Done. QED.
Without such a warning about prerequisites, the undefined acronym was bad technical writing.
> your lack of knowledge of a particular set of technologies
That whole concept is big mistake common in computing. The "technologies" in question are, for at least a good first description, just dirt simple with meager prerequisites.
None of us in computing should try to carry the whole pile of simplistic trivia around between our ears. For a related lesson, after a while in academics, it becomes obvious that only fools try to carry the whole research library around between their ears -- the lesson is much stronger for the less deep, less good material in practical computing.
With good technical writing, nearly anything in practical computing can be explained plenty well enough to nearly any of us.
I don't have a relevant "lack" of anything in computing.
You failed to explain what OP meant, and didn't explain that your comment was aimed at people who knew this kind of terminology. Does that make your comment bad writing?
No one is suggesting that people should carry every bit of technology around in their heads. Google is a great source for finding things you don't know. The alternatives - explaining in every single blog post the complete set of terms that you need to understand to read it - would make for massive, unreadable posts and would be 90% useless for most of the audience it is aimed at.
> There was a hyperlink
Sorry, clearly that's just not good enough since the first use of the acronym did not have the link. The standard in technical writing is to take some steps to be adequately clear on terminology at or before the first, say, significant or material, use of that technology in the piece of writing.
> Google is a great source for finding things you don't know. The alternatives - explaining in every single blog post the complete set of terms that you need to understand to read it - would make for massive, unreadable posts and would be 90% useless for most of the audience it is aimed at.
Of course it would, but I've suggested no such thing. Instead, I was clear:
"With good technical writing, nearly anything in practical computing can be explained plenty well enough to nearly any of us."
Did I mention, my concern is good technical writing? I thought that somewhere in my posts here I mentioned that my concern was good technical writing. To be more clear, let me say, my concern is good technical writing. I want to be fully sure: Did I say that my concern was good technical writing?
For just what constitutes good technical writing, I won't try to give a course here and will leave that to other sources.
But, the good work in the fields of science, technology, engineering, and math (commonly called the STEM fields) has some really good examples of good technical writing. E.g., for science, see the freshman physics texts by Sears, etc. For technology, see D. Knuth, if only his The TeXBook. For engineering, see, say, D. Luenberger, Optimization by Vector Space Methods. For math, see, say, any of the more respected texts for freshman calculus going back at least 50 years. Also see the texts of W. Rudin.
The core problem I am addressing here is that the technical writing in practical computing is way too often really badly done with high costs for nearly everyone involved, really high costs.
Some of the comments here defending the original post are straining to defend the bad technical writing, making excuses for no good reason.
Let me say, likely the first good rule in good technical writing is to strain never but never have undefined terms, acronyms, and jargon; instead, on or before the first use of any such, have, for definitions, motivations, discussions, examples, explanations or at least links. E.g., write "Amazon Web Services (AWS)". Just do that -- always for the first use of the acronym AWS in anything at all about AWS. Just do it, always. Spend the extra three words. Be clear. Remove all doubt. Reassure the reader that in the piece of writing they just will not face undefined gibberish. Assure the reader that they don't need Google searches to get prerequisites for the piece. Be easier to read.
For acronyms, the example of API abbreviates both application programming interface and American Petroleum Institute is right on target -- for nearly any three letters, there are from several to many three words they could abbreviate.
And, if there is no reasonable way for the piece of writing to be for a broad audience, then up front say what the intended audience and prerequisites are.
This is dirt simple stuff. To accept these lessons it should be sufficient just to want to communicate instead of intimidate.
Are we communicating now?
The only question I had was, why regular Borel measures? Sure, if I went back and studied again where he used the hypotheses I'd see it.
But in Rudin's favor, e.g., for his Principles, it's all there, and crystal clear, very precise, and perfectly correct. Yes, have to draw own figures, at least between ears, maybe on paper.
Students might be told that one of the purposes of the early material in Rudin's Principles is to build a really solid foundation for discussing continuity and uniform continuity. So, for that, he wants compactness. He uses that work to show that, with continuity on a compact set and, thus, uniform continuity, the Riemann (Riemann-Stieltjes) integral exists. Then near the end of the book he shows that the uniform limit of continuous functions is continuous.
But, also he goes ahead and shows that a function has a Riemann integral if and only if it is continuous everywhere except on a set of Lebesgue measure 0, without really saying much about Lebesgue measure.
I don't have much from Apostol. I'm sure he could write a book that competes with Rudin's Principles and might be easier for ugrads to read.
It happens that two weeks or so I got out Rudin's Principles, the third edition, and looked up something and was struck at just how beautifully, elegantly, sparsely done the work actually is.
On elegance, Neveu is my favorite.
And for AWS, IAM is a standard abbreviation. That's my point.
>Sorry, clearly that's just not good enough since the first use of the acronym did not have the link.
I've already acknowledged that, but one minor error - putting the hyperlink on the second use of a term that you could locate in 5 seconds in Google is hardly going to make the whole thing unintelligible is it?
>Did I say that my concern was good technical writing?
You did. And you've backed it up with pretty poor non-technical writing and a single complaint about the lack of hyperlink on the first occurrence of a single acronym that 99% of those reading an AWS technical blog would already know.
That point fails: HN has a much larger and more broad audience than AWS. Still, actually, HN should define OP -- I still don't have a good source to know what it means and have been only guessing. That AWS has a smaller audience than HN does not excuse omitting definitions.
Again you are assuming that readers of that OP blog are "99%" experienced AWS users, and that's not good. If the blog were only for such users, then I should say so.
That I could look around to unwind the acronym, later in the text, elsewhere on the page, on some other pages of the blog, elsewhere at AWS, at Google, doesn't excuse anything.
I mentioned some authors with some writing that is astoundingly technical, and there terms are defined as I described.
Computing is awash in bad technical writing; I should be torqued at it, and I am; one of the worst problems is poor handling of technical terminology, jargon, and acronyms; the blog post was an example: Either (A) define the acronyms on or before first usage or warn the audience that there are prerequisites.
Simple.
So it's HN's fault, not yours, that you didn't define a term that you chose to use?
And you're using a term that you don't know what it means?
> I should be torqued at it, and I am
What is "torqued" meant to mean in this context? I know what torque means, but it makes no sense when applied to this sentence.
When you were toddler you started stringing those words together, and learning how to form extremely basic sentences.
And eventually you learned to form complex sentences with complex ideas.
The same thing applies to learning computing terms. AWS is now 9 years old, and has 9 years of development and features to catch up on if you are just starting. Yes, it will take some effort to learn everything, and if you don't see the benefit in doing so then you don't have to.
But if you want to be an "adult" engineer rather than a "toddler" engineer it is necessary to be willing to learn some things, even if it isn't easy.
The research libraries are awash with single texts that start with meager prerequisites and get to some quite advanced and challenging material. AWS can do so, also.
I've been in computing for a long time, but there's no hope for me to know all the jargon of all parts of computing. Mostly jargon is not conceptually difficult, and the solution to the challenge of jargon is simple -- just define it.
Parts of computing have long enjoyed having a closed priesthood that spoke in three letter acronym gibberish. IBM used to do this with PCP, MFT, MVT, MVS, JCL, DCB, IMS, DB2, CMS, SQL, and many more. And there were some four letter acronyms -- CICS, DASD, REXX.
The concepts were mostly simple; the gibberish was a secret language. Once I gave a grad seminar on computing and had a lot of guests from industry. IBM talked in their gibberish, but none of the other vendors did. The students caught on: IBM wanted to intimidate people and otherwise invite people into their priesthood where they, too, would understand the gibberish. IBM was the least popular of the vendors.
The linked article is basically a changelog for devs who have already been using the ecosystem for years, so no its not newbie friendly.
They would have been doing better for their business if they had made that point clear.
Again, my gripe is, better writing is needed. Why? The writing now too often results in absurdly high costs for absurdly poor reasons. The costs to me have me torqued.
We're just talking about bad writing.
I can still remember picking up those 500 page computing books. Flipping through the phone book and putting it back on the shelf.
As to the overuse of acronyms; I think they are overused in this profession. It almost seems fashionable?
Jump to today, things have gotten better. I'm not sure the writing has gotten better, or I just have more presenters of the information(Internet)?
I don't know why you got down voted? The level of technical writing in this field is atrocious? I understand "these are tough concepts!". Yes, in many ways it's like learning a language! I get it. In most cases I'm glad someone took the time to write something down. Yes--I'm grateful.
I do have a problem with entities, with a lot of money/resources, whom publish terrible technical writting. It does seem like a big company, like Amazon, could have done a little better job conveying this product?
Companies like Apigee/Mashape/3scale/Mulesoft have been doing cloud API management in various forms since 2008. Even Microsoft Azure has an API management offering since two years.
Nowadays all those API gateway features are commodities and doesn't make sense to pay for it anymore. Indeed Open Source projects such as KONG [1] are getting tremendous tractions. Same things happened in search with all cloud solutions and then ElasticSearch came out and was game over.
[1] https://github.com/mashape/kong
There's value in integration and a lot of people want that.
On the other side it seems like other extra functionality is limited and very AWS-oriented. If you are looking for an open source gateway that can be expanded with extra functionality, and potentially sit on top of the AWS-> Lambda integration, take a look at https://github.com/Mashape/kong (I am a core maintainer too, so feel free to ask me any question). Kong is accepting plugins from the community (http://getkong.org/plugins), which can be used to replace or extend functionality beyond any other gateway including AWS Gateway.
It's in legacy API services that places like Mashery and Apigee make all their real money, and I can't see large enterprises shifting their legacy services to EC2. It's quite forward looking. The satire on how EC2 may be deprecated is really not far off from where things are heading.
I always find it quite funny when only single OSS projects are mentioned as alternatives, there's so many more full-featured projects out there: Tyk (https://tyk.io) API Umbrella (http://apiumbrella.io/) and ApiAxle (http://apiaxle.com/) to name some. For more fine grained approaches there's also Vulcan (http://vulcanproxy.com/)
Also, how can you gracefully reload kong if you need to add/remove/change a plugin ? With vulcand you just replace the binary, and send an USR2 signal to the running process. It will fork, wait for all connections to drain, and remove the old process
For deployment, again, you need to sync all lua files, with vulcand you just ship a compiled binary.
Kong doesn't have a notion of servers for each API, you need to forward requests to a haproxy or another load balancer for this. Also, I can see that backends are added by their DNS hostname. In order to achieve HA (backend redundancy) is there any way you can do it, assuming that nginx is caching the upstream dns values ?
About performance, I can see you are advertising about 1000 r/s using kong, and you need 3 machines for this (kong, cassandra, haproxy). I benchmarked vulcand and obtained about 12000 r/s on a more modest hardware.
Nginx is not an API management layer, right? It doesn't have the necessary features, but it's extensible so someone could build them. Mashape did, and they released the result as Kong.
Vulcand is in the same position as nginx. It could be a good choice for building an API management layer on, but it doesn't provide one out of the box.
> How can you gracefully reload kong if you need to add/remove/change a plugin?
You can gracefully reload Kong by executing "kong reload" (http://getkong.org/docs/0.3.2/cli/#reload). It pretty much works the same way: fork, wait for existing connections to drain, and remove the old master process.
> For deployment, again, you need to sync all lua files, with vulcand you just ship a compiled binary.
Kong has a few distribution options (rpm, deb, Docker, etc) that simplify the process (http://getkong.org/download/), so unless it's being built from source, the deployment is straightforward.
> Kong doesn't have a notion of servers for each API, you need to forward requests to a haproxy or another load balancer for this. Also, I can see that backends are added by their DNS hostname. In order to achieve HA (backend redundancy) is there any way you can do it, assuming that nginx is caching the upstream dns values ?
You are correct. Kong supports either a DNS or IP address for a backend service, and starting from 0.5.0 it will be possible to add more DNS addresses or IPs per each API. That will make Kong work in load-balancing mode (https://github.com/Mashape/kong/issues/157).
> About performance, I can see you are advertising about 1000 r/s using kong
Kong is basically built on top of nginx (a very solid core), and it can achieve a similar performance to nginx because all the data requested from the external datastore is being cached in memory. So pretty much the latency it's going to be nginx + Lua execution overhead (which is minimal, thanks to LuaJIT). I will run a better benchmark and write a blog post soon, trying to cover both single-node and multi-datacenter setups.
> and you need 3 machines for this (kong, cassandra, haproxy)
An external datastore (Cassandra - with plans to support PostgreSQL) is required because Kong has been built to scale multi-node and multi-datacenter in order to handle pretty much every use case and plugin/middleware. In very simple single-node use cases, Cassandra can live in the same instance along with Kong. I would say that in this regard Kong is inspired by a different philosophy (start small, grow big, even supporting hybrid cloud/bare-metal setups). Having an external datastore also means that Kong could support some pretty cool plugins, like multi-datacenter service discovery, API billing coordination, multi-region health checks, etc. For example today it is possible to rate-limit requests in an eventually consistent replication fashion across multiple data-centers.
The SOAP to JSON conversion is easy for baby SOAP or JSON requests using transformation. For larger ones they turned out to be complex.
Once you start with client started with using the "SOAP" converted to JSON apis, there were new feature kind of requests. Now the problem is somebody needs to be dig into that old code mess of SOAP web services. Or write new services and split requests between old and new services.
There will be performance issues because of the no of transformation doing on top of the request. Ultimately has to write plugins on top write performant transformation in a single piece.
Personally SOAP to JSON conversion will not be as simple as move to AWS, although I will assume Amazon will do far far better job on this than others.
any plans to add pluggable backend ie not use cassandra? i've no experience managing cassandra and would rather not worry about it.
For testing check out http://kongdb.org/, but don't use it in production.
[1] http://www.meetup.com/The-Mashape-API-Developer-Community/ev...
Moreover I feel Lambda can do pretty good stuff. I am sure it will integrate with other AWS services pretty soon and once there is connectivity to an RDS instance, then it solves the use case for half the developer population out there.
[1] https://github.com/mashape/kong [2] http://getkong.org/plugins/
[0] http://aws.amazon.com/cognito/
[1] Have a database of a few gigabytes of sizes, start an ``ALTER TABLE`` and wait until your pre-allocated disk space is filled up. The query dies and you just fucked yourself. Can happen unexpectedly: We tried to remove a non-existing index on a table and MySQL decided to FIRST copy the entire table and THEN report that there is no such index.
We weren't particularly locked in, either. We always had the option to run our own Postgres cluster (it just didn't make sense to spend the manpower on it for us). We could have hacked together our own CDN, but that wasn't our core business focus.
For small companies with low manpower, build on the services and get things done fast and as well as possible. It may cost more upfront than doing it on your own, but you are probably going to come out ahead by minimizing your maintenance/security/administration surface area.
Contrast this with using AWS DynamoDB, which would likely be a major overhaul if you needed to switch to something that could be used outside of AWS.
No, we use Vagrant so that we can run our docker cluster either locally or on Amazon. We have stubbed alternatives for all AWS services, including SNS and SQS, allowing us to run the entire backend locally even without an internet connection.
> Have you been bitten by problems that you can't resolve because you don't own the product?
I'd say its kind of like coding in Ruby on Rails. If you are bit by a problem you can't resolve there is a high chance you are doing something wrong, or at least not the "one true way".
Sometimes we have issues when exploring the edges of what is possible on the AWS ecosystem (we've been using the fresh-out-of-beta ECS, and running into a lot of limits there), but generally the basic stuff like S3, SNS, SQS, Elasticache, etc just works, and doesn't require any attention. It's just a working, hands off gear in your system.
> Do you pretty much just love it?
Sure. It's a lot better than racking physical hardware, and the deep integration between things like autoscaling groups, EC2, ELB, etc makes it easy to automate all the things. In the end the more our service is a machine that can run itself and repair itself (replacing its own instances as needed) the more I love it.
Largely we've been very pleased with AWS, but the black box nature of a number of products (particularly Elasticache and RDS), has caused us some minor grief. Also, the higher frequency of single instance failures (which our stack can handle pretty gracefully) is somewhat annoying.
EDIT: We were also able to fairly significantly lower our bill with reserved instances while also significantly increasing our ability to handle peak capacity with AWS.
Basically, we just open a support ticket and say "what happened", and they just say "oh, the machine failed and was rebooted"
Speaking somewhat from experience (webmethods, caml, spring-integration and various other enterprise integration tools) they always want you to use "their" DSL which is often not even stored on the developers filesystem (ie webmethods... you submit the code to the "broker" or "rules engine" or "router"... lots of words for thing that does the work). Which leads to very awkward development.
Consequently I wonder if they will have git integration because writing code even snippets in a web form no matter how nice gets old fast.
Shameless plug: after working on several API projects, I have been researching ways to not have to "code" over and over again what goes into creating endpoints, it became so repetitive. Lately, I turned to RAML (Yaml for REST) and, with 4 other developers, we created an opensource project called Ramses. It creates a fully functional API from a RAML file. It is a bit opinionated but having to "just" edit a Yaml file when building a new API simplified my life. As a bonus, I also get a documentation and a javascript client generated from the same file.
EDIT: forgot the url https://github.com/brandicted/ramses and url of a quick start tutorial: https://realpython.com/blog/python/create-a-rest-api-in-minu...
http://i.imgur.com/wSEKeVb.png
It would be nice to have one place to edit routes with a good UI, better than what Amazon released today.