I haven't used many of the new AWS services. Someone tell me, what's the quality level? I'll be surprised if all the new AWS stuff works that well given how divided their focus must be now.
(I WORK FOR AWS)
Each service has it's own dedicated service team. I've used some (not all) of the new services and I believe there is room to improve but I didn't think they were unstable or anything like that.
I know there's no obligation to invest in services that don't make money, so please don't take this as some kind of expectation of that. A dedicated team doesn't always mean there's an equal distribution of quality and innovation. You can have a dedicated team that's so small they only have time to do bug fixes or a team with less talent/experience because everyone wants to move to teams working on the projects with the most organizational momentum.
TBH I think you have a point there: the hot new thing is serverless and lambda so everyone I know really wants to work on those teams. I see where you're coming from for sure.
There are still a lot of talented and passionate people at Amazon who want to execute on other products. The S3 team is still growing 10 years later and they're still doing (IMO) cool stuff. I'm a devangelist though so I don't always get to see the inside of every team (not enough time). I do know that we all drink the customer focused kool-aide and we genuinely believe in it. Sorry if this all sounds like marketing.
You say that like their quality doesn't suffer today because of how divided their focus has been in the past :)
AWS's crustiness is exactly what I would expect from an agglomeration of adversarial teams each looking to minimize their own liability at the expense of everyone else. Given that they're still organized that way, the surprise would be if they turned the trend around, not if it continued.
Ive been pretty impressed by the latest AWS services. There are a ton of them though. I wouldnt expect anything different from a system designed to accomodate arbitry infrastructure. I'm considering deploying to Lightsail next, which looks like refreshing simplicity for smaller projects.
It depends, but the quality is often MVP level. Some new services are introduced but then virtually abandoned or developed at a snail's pace (Service Catalog, ECS). Sometimes I wish they'd spend more time making the core awesome instead of constantly introducing new services.
Having worked at AWS for a number of years, I had difficulty keeping track of all the new features/services when I was there. If I were still there, I would have given up :-) But I do think that overall, it's better for both the customers and the company.
As to some concerns about diluted focus, service teams operate with great deal of autonomy. From what I have seen, features are driven by teams, and often as a result of conversations with customers.
AWS makes me wonder what the future of open source infrastructure will look like. I'm sure they contribute back at least a little but providers have little reason to contribute back the distributed/scale modifications they make. They can take any project with a friendly license and essentially siphon away its future user base by turning it into a service. It's basically what a lot of open core companies do but it's different since those companies are usually the creators/maintainers so they have a vested interest in preserving and maintaining it.
I'm probably over thinking it but big bang announcements of multiple products does make me wonder.
If I were developing a new project I'd probably go with an AGPL style license. It just seems appropriate for anything that could be hosted. Not sure what the downsides are.
As someone who writes AGPL software, I will say to most of us it isn't a downside: I would conceptialize that scenario as "your loss" and hope that my work helps give companies who are more friendly to the idea of community software at least a little bit of a competitive advantage over your company. In all seriousness: if you are intending to not contribute changes back (which is the only legitimate reason to avoid GPL variants: because you want to use someone else's work while hoarding your own), your gracious "using" of my software has absolutely no value to me...
Totally agree that it is indeed "our loss". No argument there.
I don't want to appear a kook, so I won't relitigate what I said in another reply. But essentially we do want to give back. We just aren't always able to give back everything (at least not right now).
AGPL, like GPL, is all or nothing in that way. I don't get the option of giving back something, then giving back more later (OSS is threatening to some people - I have frequent discussion at work re why we give anything at all back).
Your software, your rules. You have every right to take this all or nothing position! More power to you.
But assuming that anyone that doesn't like AGPL does so because they "want to use someone else's work while hoarding [their] own" is, respectfully, a simplistic and stereotyped point of view.
I wonder if a modified license that set an explicit fixed boundary would help. "You must contribute back any changes to this software that you make but anything that interacts with it at using its API is exempt from the license." So for example if you took the NFS server source and modified it to be distributed and highly available across multiple servers you'd have to contribute that back but the UI or management software you use build for customer interaction with it wouldn't be "infected".
The worry, places I've worked, is that the GPL would spread to proprietary stuff, like a kind of legal infection.
They worry that if they use an AGPL javascript library for the video player on their front page, they'll have to opensource the whole web application (I've even sat in on debates over weather using GPL3 programs means you have to opensource any source data you use them to process and publish the result).
A lot of this is silly, but a lot of it isn't. A large company I worked for stopped using Linux after they were forced by a court to opensource a custom video processing microchip because they ran Linux on the box and drove the chip through a kernel driver.
The chip design. It was a PVR box and an activist group wanted to opensource the DRM system they were using because they knew some important codes were baked into the chip so they took them to court. The court felt that the functionality was shared between the GPL kernel driver and the chip, so the chip fell under GPL.
Plenty of companies would also use the non-AGPL software without any strings attached, and not contribute patches anyway. You may be lucky enough to get bug reports, though (and, in effect, become a free labor source for a corporation). So what's the point? Most of the time, this kind of position just comes across as a vague, empty posturing, to be quite honest. "I could definitely be a good friend to you, you see... if you cooperate and help out your new friend".
It sounds like a very obvious choice when you phrase it as "we wouldn't contribute patches to you", but that's just a way of holding out a carrot and hoping for people to bite, phrasing it as a "your loss" scenario. At least it is, to me. What am I losing, exactly? My code is right there. I haven't lost anything at all, except maybe some vague, hypothetical possibility of a little extra help -- which, in general, seems to rarely materialize in the way people imagine anyway, for a host of reasons (companies are slow, they're lazy, their hacks are incomplete and unsuitable for an upstream, they're anti-OSS contribution in general and only disallow using the GPL, but disallow contributing to anything, etc).
There is plenty of software that, very arguably IMO, should be available under non-GPL terms for a host of reasons, it's true (even the FSF agrees with this, to help support and promote e.g. interoperable, royalty-free codecs, algorithms, which benefit from public domain availability). And licensing is also a complicated legal, political and, at some level, personal topic as we pick licenses that we believe represent our goals and values. We should treat it that way.
But phrasing the scenario in such a manner, where there's a narrative that "The only thing the AGPL will do is ensure I write 0 patches (even though I would have likely written 0 patches in any case), so uhhh, think about that", feels like an attempt to poison the well, even if you didn't intend it to be. We could just as easily turn it around: if you literally aren't going to contribute patches, to return some of the benefit bequeathed to you by the virtue of using my code -- why should I even care a tiny bit about you or your company, or whether you "need" to use my software to achieve your own goals? Clearly, you don't need it so badly that you're willing to agree to the rules. Why bother? So you can just freeload on the people who do agree to the rules?
If this isn't your intent or I'm being overly presumptuous, I apologize. But even then, I still dislike the narrative that pops up around these arguments, even if I may be reading too much into your particular comment.
I believe, respectfully, that you're reading posturing into my position where none exists.
In a typical scenario for us, we use some open source software for our SaaS platform. We need to extend it in ways that weren't envisaged, so we (a) add some hooks to allow extensions and (b) add the extensions themselves.
Our preference is to contribute both (a) and usually (b) back to the project. But sometimes for business reasons we can't contribute (b), or at least not until more people internally have their heads around it.
AGPL stops us from contributing (a) and not (b). So we just don't use AGPL software.
I wasn't trying to poison the well, I was just indicating what our decision making process is. Some people, like us, have valid business reasons for not using AGPL software, even though we love OSS.
I find it weird that this narrative thing pops up here. There's no narrative. If I was the owner of said AGPL software, I probably wouldn't give a rat's arse if anyone said they weren't going to use my software.
But my original post was a specific response to someone who said "Not sure what the downsides are." [of using AGPL].
So let me try again. Here's one possible downside - *you may miss out on some patches that you would have otherwise got, since your license is incompatible with some people's business needs".
I know they do a tiny bit here and there (there was a presentation here at the re:Invent conference about it that I didn't get a chance to attend) but their usage of it internally probably dwarfs those contributions.
It makes business sense even if it's a shitty thing to do. The stronger an open source project is that provides the core functionality that your managed service provides the easier it is for a competitor to build the same thing. Amazon isn't as cutthroat as Oracle but they're certainly much less OSS friendly than many of their competitors.
Part of it is the cereal business model. You introduce more flavors not because they are going to sell well or to increase volume, but just to take shelf space at the supermarket away from your competitors. In this case the shelf space is mindshare, media coverage and "feature coverage" in researcher's market comparisons.
If your new cereal favor doesn't increase sales, wouldn't the supermarket find a brand that does? Otherwise, the super market isn't optimizing shelf space.
If you're interested in distributed system tracing there's a lot going on.
As a starting point, I would recommend reading Google's paper on their project "Dapper." [1] It's essentially the core of most distributed tracing systems. At least those I've encountered.
There's a lot of tooling out there that take their cues from Dapper. I've recently been looking into integrating OpenZipkin[2] with our systems. I see at as a more viable alternative (no tie in!) to Yet Another Propriety Thing in AWS (YAPTA). There's other as well, like AppDash.
Recently there has been a push towards an open-standard for the collection side called OpenTracing[3]. I came across it when investigating LightStep[4]. Ideally that means no vendor lock-in, which of course has lots of knock-on effects.
If you know of any, I'd love to be pointed in the direction of _different_ and not just divergent techniques.
My masters thesis is X-trace derived, as well as a paper we published at ICPE a few years ago. I'm on mobile so don't have a link handy. You can probably guess my name from my username.
Tracelytics was a startup I co-founded based on X-Trace, which ended up as AppNeta's Traceview and is now part of the Solarwinds monitoring cloud suite of products (joining Librato, Pingdom, Papertrail). So look out for more tracing from us as well.
Request IDs and their variants are only giving you correlated events, they don't encode enough information to connect causality and ordering of events. Basically: with request IDs you get a unordered bag of events, with proper tracing (trace id/span id/parent id) you get a DAG of events.
In practice, the analysis a DAG allows you to perform are way more powerful. So if you're in the process of setting that request ID stuff up, consider doing the tracing way.
We've been instrumenting Zipkin in our environment over the last couple of months and it has been excellent to work with. While there are a couple of snags here and there, it is _amazing_ at how quickly it can bring bottlenecks and bad network calls to light.
Stackdriver Trace [1] is Google's external implementation of Dapper with additional analysis features. It's available for free, even for workloads not running on Google Cloud Platform.
This is awesome! Can Google update the docs so that the links for external zipkin integrations work? They currently link in circles and there's no info on how to integrate into OpenTracing/Zipkin
Yep, an issue with the anchor links that'll be updated soon. It's supposed to just link to a lower part of the page.
If you scroll down you'll see a top-level header also titled "Configure Zipkin tracers" that describes how to configure the Brave tracer. You don't really need to do anything special here - just point your Zipkin tracers at the Stackdriver Zipkin Collector rather than your existing one.
Since you work on Stackdriver, could you explain to me how did the pricing change last week? Previously I thought $8/instance/month was expensive, but it seems like now there is a cheaper basic tier. What is a good use case for the basic tier and when will I need the premium?
Been using OpenTracing with LightStep and they have a really solid product, and a super well thought out API design in OpenTracing. Highly recommend checking them out, especially since OpenTracing saves you from vendor lock-in and works using AppDash/Zipkin/etc.
The similarity to Dynatrace is that - like the PurePath - X-Ray enables distributed tracing. But the approach is different: whereas Dynatrace instruments your application using an agent^, X-Ray needs you to instrument an application using the provided SDK. Moreover, X-Ray seems to be limited to applications communicating via HTTP(S).
Incredible. My book talks about writing tests to help build reliable systems but having a dynamic run time tool like this is an astounding addition to a developer's toolset.
Lol yeah, I think it's more about it being Re:invent. FWIW, I've been receiving a lot of pings on this stuff since I work on OpenTracing. So I think it's believable that this reached the top on its own.
Ugh. Of course I read the comments first to see if the article is worth reading and it takes me 10 minutes to realize this is not Amazon's 3D rendering as a service, service..
Honest question: why is there so much AWS news up today? I mean, X-Ray is particularly exciting for me, but there are currently 5/30 stories on the front page which are basically just product announcements. I'm pretty new, but is this normal? I thought the basic upvote criteria was meant to be that we should focus on upvoting articles of some depth as well as interest.
Google, Apple, Microsoft, and Amazon each dominate the frontpage for several days when their respective developer conferences are ongoing or just completed.
There's a big Amazon event (AWS re:Invent) and they are announcing an extraordinary number of new tools. Think an Apple or Google event but one where they announce a new product every 15 minutes. People with many different kinds of jobs are excited for different reasons. It's double-Christmas for people who work extensively with AWS.
Today is the second day of AWS Re:Invent. So just like when Apple releases products and the site is slammed with Apple news, these two days are Amazon days. Don't worry, tomorrow will be back to normal.
Significant blow to all APM players in the space. Approximately 50% of New Relic's revenue comes from AWS. Datadog just announced their distributed tracing system at $25/host. X-Ray is orders of magnitude cheaper than either - and per trace pricing is something that would be very tough for anyone else to do and plays well to Lambda efforts.
X-Ray seems like a pretty basic service at this point but in my opinion the writing is on the wall for other APM providers. Amazon chose this announcement for a major slot so I expect they'll be investing in the service. As Bezos is often quoted as saying "your margin is my opportunity."
Turns out X-Ray is sampling service, it drops data. You can use it to debug recurring problems, but it's no use when debugging a particular incident with the customer on the phone. Bummer.
To provide a performant and cost-effective experience, X-Ray does not collect data for every request that is sent to an application. Instead, it collects data for a statistically significant number of requests. X-Ray should not be used as an audit or compliance tool because it does not guarantee data completeness.
Yeah, there's a big difference between something that's pure distributed tracing (this) and something that does metrics/monitoring as well (TraceView, New Relic, ...). Sampling is a valid way to make distributed tracing scalable and performant, but it can also limit the use-cases for the data.
I imagine this being used for episodic debug cases, where one could turn up the sample rate and pay only for the traces captured/queried during an incident. But you would need to do your monitoring and trending separately.
I wouldn't be surprised if they eventually start integrating this better with CloudWatch for that reason, though it doesn't seem to be doing any of that today.
Disclosure: I work on TraceView (traceview.solarwinds.com) which is distributed tracing based APM product. We're inspired by Google Dapper and x-trace, both mentioned elsewhere in this thread.
I was speaking to a more general monitoring approach where you might want to know p99 latency, request volume, error rate, etc, for each service to use in alerting and trending. This is a common use-case for application monitoring that isn't addressed well by a pure tracing approach.
If you're looking specifically for the 100% audit trail case, I'd look at DynaTrace or potentially Instana.
With Zipkin and OpenTracing you can force spans to be recorded despite the sampling, so if you know something errors (middelware, catches etc.) you can almost guarantee that you'll record your trace if you set things up correctly.
Some have mentioned is this full APM and does this compete with New Relic, Dynatrace, Stackify, Appdynamics, App Insights, etc.
Those products are primarily based on code profiling. For example, at Stackify we automatically profile key methods for dozens of common dependencies and frameworks to understand their usage and performance. Every SQL, NoSQL, caching, queuing providers and many other things. Plus app errors, logs, etc.
So the best I can tell from the AWS blog and docs is the answer is no its not a full APM. It appears to track how long a web request takes and any usage of the AWS services via their SDK. More of a lightweight service mapping of AWS services. So SQL database or HTTP calls probably aren't tracked.
In the future could they expand it? Sure. But for now it seems limited compared to a full blown APM product. Although this could be help for identifying performance problems with AWS services.
103 comments
[ 2.8 ms ] story [ 166 ms ] thread> AWS X-Ray supports tracing for applications that are written in Node.js, Java, and .NET.
Edit: The blog post mentions they will be adding this to the other SDK's.
I'll be talking about this on the https://twitch.tv/aws stream at 12:30 pacific if you guys want to ask questions / learn more.
(I WORK AT AWS)
If there's no Python support - that shocks me.
There are still a lot of talented and passionate people at Amazon who want to execute on other products. The S3 team is still growing 10 years later and they're still doing (IMO) cool stuff. I'm a devangelist though so I don't always get to see the inside of every team (not enough time). I do know that we all drink the customer focused kool-aide and we genuinely believe in it. Sorry if this all sounds like marketing.
AWS's crustiness is exactly what I would expect from an agglomeration of adversarial teams each looking to minimize their own liability at the expense of everyone else. Given that they're still organized that way, the surprise would be if they turned the trend around, not if it continued.
As to some concerns about diluted focus, service teams operate with great deal of autonomy. From what I have seen, features are driven by teams, and often as a result of conversations with customers.
I'm probably over thinking it but big bang announcements of multiple products does make me wonder.
That might or might not be a downside to you.
I don't want to appear a kook, so I won't relitigate what I said in another reply. But essentially we do want to give back. We just aren't always able to give back everything (at least not right now).
AGPL, like GPL, is all or nothing in that way. I don't get the option of giving back something, then giving back more later (OSS is threatening to some people - I have frequent discussion at work re why we give anything at all back).
Your software, your rules. You have every right to take this all or nothing position! More power to you.
But assuming that anyone that doesn't like AGPL does so because they "want to use someone else's work while hoarding [their] own" is, respectfully, a simplistic and stereotyped point of view.
They worry that if they use an AGPL javascript library for the video player on their front page, they'll have to opensource the whole web application (I've even sat in on debates over weather using GPL3 programs means you have to opensource any source data you use them to process and publish the result).
A lot of this is silly, but a lot of it isn't. A large company I worked for stopped using Linux after they were forced by a court to opensource a custom video processing microchip because they ran Linux on the box and drove the chip through a kernel driver.
It sounds like a very obvious choice when you phrase it as "we wouldn't contribute patches to you", but that's just a way of holding out a carrot and hoping for people to bite, phrasing it as a "your loss" scenario. At least it is, to me. What am I losing, exactly? My code is right there. I haven't lost anything at all, except maybe some vague, hypothetical possibility of a little extra help -- which, in general, seems to rarely materialize in the way people imagine anyway, for a host of reasons (companies are slow, they're lazy, their hacks are incomplete and unsuitable for an upstream, they're anti-OSS contribution in general and only disallow using the GPL, but disallow contributing to anything, etc).
There is plenty of software that, very arguably IMO, should be available under non-GPL terms for a host of reasons, it's true (even the FSF agrees with this, to help support and promote e.g. interoperable, royalty-free codecs, algorithms, which benefit from public domain availability). And licensing is also a complicated legal, political and, at some level, personal topic as we pick licenses that we believe represent our goals and values. We should treat it that way.
But phrasing the scenario in such a manner, where there's a narrative that "The only thing the AGPL will do is ensure I write 0 patches (even though I would have likely written 0 patches in any case), so uhhh, think about that", feels like an attempt to poison the well, even if you didn't intend it to be. We could just as easily turn it around: if you literally aren't going to contribute patches, to return some of the benefit bequeathed to you by the virtue of using my code -- why should I even care a tiny bit about you or your company, or whether you "need" to use my software to achieve your own goals? Clearly, you don't need it so badly that you're willing to agree to the rules. Why bother? So you can just freeload on the people who do agree to the rules?
If this isn't your intent or I'm being overly presumptuous, I apologize. But even then, I still dislike the narrative that pops up around these arguments, even if I may be reading too much into your particular comment.
In a typical scenario for us, we use some open source software for our SaaS platform. We need to extend it in ways that weren't envisaged, so we (a) add some hooks to allow extensions and (b) add the extensions themselves.
Our preference is to contribute both (a) and usually (b) back to the project. But sometimes for business reasons we can't contribute (b), or at least not until more people internally have their heads around it.
AGPL stops us from contributing (a) and not (b). So we just don't use AGPL software.
I wasn't trying to poison the well, I was just indicating what our decision making process is. Some people, like us, have valid business reasons for not using AGPL software, even though we love OSS.
I find it weird that this narrative thing pops up here. There's no narrative. If I was the owner of said AGPL software, I probably wouldn't give a rat's arse if anyone said they weren't going to use my software.
But my original post was a specific response to someone who said "Not sure what the downsides are." [of using AGPL].
So let me try again. Here's one possible downside - *you may miss out on some patches that you would have otherwise got, since your license is incompatible with some people's business needs".
Almost from 2 years ago:
https://news.ycombinator.com/item?id=9358843
But gist is, Amazon employees are generally discouraged from contributing to open source at all.
It makes business sense even if it's a shitty thing to do. The stronger an open source project is that provides the core functionality that your managed service provides the easier it is for a competitor to build the same thing. Amazon isn't as cutthroat as Oracle but they're certainly much less OSS friendly than many of their competitors.
> You can use X-Ray with applications written in Java, Node.js, and .NET that are deployed on these services. Support for AWS Lambda is coming soon.
As a starting point, I would recommend reading Google's paper on their project "Dapper." [1] It's essentially the core of most distributed tracing systems. At least those I've encountered.
There's a lot of tooling out there that take their cues from Dapper. I've recently been looking into integrating OpenZipkin[2] with our systems. I see at as a more viable alternative (no tie in!) to Yet Another Propriety Thing in AWS (YAPTA). There's other as well, like AppDash.
Recently there has been a push towards an open-standard for the collection side called OpenTracing[3]. I came across it when investigating LightStep[4]. Ideally that means no vendor lock-in, which of course has lots of knock-on effects.
If you know of any, I'd love to be pointed in the direction of _different_ and not just divergent techniques.
[1]: http://static.googleusercontent.com/media/research.google.co...
[2]: http://zipkin.io
[3]: http://opentracing.io
[4]: http://lightstep.com - Impressive team behind this.
[5] https://github.com/sourcegraph/appdash
and their follow on work, Pivot Tracing: http://pivottracing.io/
(which got a best paper award at SOSP'15.)
https://twitter.com/dhh/status/795732523640766464
In practice, the analysis a DAG allows you to perform are way more powerful. So if you're in the process of setting that request ID stuff up, consider doing the tracing way.
(Disclosure: I work on it)
[1]: https://cloud.google.com/trace/
It doesn't show zipkin configuration :)
If you scroll down you'll see a top-level header also titled "Configure Zipkin tracers" that describes how to configure the Brave tracer. You don't really need to do anything special here - just point your Zipkin tracers at the Stackdriver Zipkin Collector rather than your existing one.
(I work on it)
The similarity to Dynatrace is that - like the PurePath - X-Ray enables distributed tracing. But the approach is different: whereas Dynatrace instruments your application using an agent^, X-Ray needs you to instrument an application using the provided SDK. Moreover, X-Ray seems to be limited to applications communicating via HTTP(S).
^ in the most cases
https://azure.microsoft.com/pl-pl/services/application-insig...
https://reinvent.awsevents.com/
Google, Apple, Microsoft, and Amazon each dominate the frontpage for several days when their respective developer conferences are ongoing or just completed.
X-Ray seems like a pretty basic service at this point but in my opinion the writing is on the wall for other APM providers. Amazon chose this announcement for a major slot so I expect they'll be investing in the service. As Bezos is often quoted as saying "your margin is my opportunity."
To provide a performant and cost-effective experience, X-Ray does not collect data for every request that is sent to an application. Instead, it collects data for a statistically significant number of requests. X-Ray should not be used as an audit or compliance tool because it does not guarantee data completeness.
https://aws.amazon.com/xray/faqs/
I imagine this being used for episodic debug cases, where one could turn up the sample rate and pay only for the traces captured/queried during an incident. But you would need to do your monitoring and trending separately.
I wouldn't be surprised if they eventually start integrating this better with CloudWatch for that reason, though it doesn't seem to be doing any of that today.
Disclosure: I work on TraceView (traceview.solarwinds.com) which is distributed tracing based APM product. We're inspired by Google Dapper and x-trace, both mentioned elsewhere in this thread.
I don't see the distinction you're making, as neither seems to record all traces for accurate audit. Am I missing it?
If you're looking specifically for the 100% audit trail case, I'd look at DynaTrace or potentially Instana.
The service works via the SDK's in various languages, that report tracing information to a local daemon that runs on the host, over UDP.
The daemon then batches the data, applies sampling (Which is configurable, all the way to 100% - report everything), and sends it en-masse to AWS.
Edit: See Sampling Rules section here - http://docs.aws.amazon.com/xray/latest/devguide/xray-sdk-nod...
I concur, it was specifically stated that 100% sampling is an option (and is by default?)
Those products are primarily based on code profiling. For example, at Stackify we automatically profile key methods for dozens of common dependencies and frameworks to understand their usage and performance. Every SQL, NoSQL, caching, queuing providers and many other things. Plus app errors, logs, etc.
So the best I can tell from the AWS blog and docs is the answer is no its not a full APM. It appears to track how long a web request takes and any usage of the AWS services via their SDK. More of a lightweight service mapping of AWS services. So SQL database or HTTP calls probably aren't tracked.
In the future could they expand it? Sure. But for now it seems limited compared to a full blown APM product. Although this could be help for identifying performance problems with AWS services.
Matt - Founder of Stackify