Not really understanding the down votes (maybe the double question mark??), but oh well.
This is a valid comment considering that the EC2 instance mentioned is a 2 core box, and there is a possibly that the node implementation didn't take advantage of both cores.
- Go has multi-core support out of the box
- Node.js has to be shown a little love, and has limitations. But this would have been a great use-case for Cluster.
If the Node implementation was using all cores, then I wouldn't have asked this question. As it stands I don't really know what we are comparing even with all the graphs.
Also we are still missing a little info on how they handled the requests. When you have a endpoint on a node server that is not a simple one, it is best to queue those before handling them (yes a additional queue). This prevents the process from choking because it just opened up NNN requests to S3.
Most developers are not working with hundreds of HEAVY requests a minute, but in Node.js this usually has to be handled with care, and does not work out of the box.
But if you want to pick up a shinier hammer then Go for it.
Why would I want to use a language that requires bolt-ons just for bare minimum acceptable performance, meanwhile dealing with callback/half-promise soup, no types, etc?
Also, I strongly suspect that an app that has been refactored multiple times by people well versed in node.js is probably already doing something to leverage multi-core...
Personal opinion: Because comments like your original one are annoying as hell.
They never fail to show up. Someone post an interesting article one how they solved a particular problem and someone always have to show up with these pointless comments. Not everyone will know about, in this case Cluster, and it's not relevant without an explanation.
Article and blog posts about Go seems to attract comments like: You might like Rust/Elm/Elixir and "that problem could have been solved using X/Y/Z package or framework".
More often than not it's missing the point, which often is people, with little or no Go experience are managing to solve really complex problems within very short time frames.
The interesting question is why Node isn't. It's supposed to be fast because everything's non-blocking, right? So why is this particular use problematic? What colour are the emperor's clothes, anyway?
Agree! It makes you wonder why they didnt try python 3.5 with async stuff, as most of their stack is based on python. Yes its not multi threaded but if the service was rewritten it might have actually performed better. Otoh: sometimes you need a good excuse to test some other technologies.
Care to explain why python async would have performed better? I don't see a single way in which python async would be a better alternative for the given job. I'm a python guy myself but, when it comes to async stuff, I find nodejs much better and easier to understand/follow through (even with cb madness yes). For the described job, I would have went with Elixir or Go as well yes.
Doesn't really make me wonder. I haven't really seen anything about python's new async stuff that indicates it's on par with node's (reply here if you found anything, I've been looking for this information!). And if python is caught up with node, it would be at very high risk of performing the same and having another random performance gotcha. Golang is one of the natural choices to replace a nodejs application for performance reasons. It's low-level, faster, uses less memory.
That's not the message of the article. The original application wasn't poorly written. In fact it was written by people who know Javascript and refactored multiple times. The new application is written by someone that picked up Go in a few weeks.
I don't see it as a criticism Node.js either, it's just that Node wasn't the right tool for the job, in this case. If you pick the correct tools and know the limitations of your tools, then it becomes less important how proficient you are with those tools (not irrelevant of cause).
I would like to know more about the implementation.
I have had node do 12000 requests a minute with no problem (on a single core). In this case they had 4 two core boxes, so they needed each one to at least perform 42 requests a second under max load.
Its evident from the flamegraph that the architecture is fundamentally different. Why are the stacks so deep in the javascript example? Why aren't they using a similar worker pool in the node implementation? Are they just throwing everything onto the event queue and treating that as a naive worker pool?
If you know the limitations of your tools then you can work around them. This was clearly not done in this case.
When a request could block the server, for example a request which download or upload a big file, Node it not for you. Any concurrent language would be better. Node is perfect when a lot or requests work together, but each one in a small time or even a long time but step by step. that is my experience and I hope it will help to choose the right tool for the task.
That misses the point of the article, and misses what they learnt. The point isn't that a request was blocking the server. That's exactly not what happened. The point was that they couldn't handle the timeout of the non-blocking request they were making because there were so many other non-blocking events being processed before the timeout callback could be handled.
They could probably have solved this by scaling out, but I'm not going to fault their diagnosis or response.
This. It sounds like the author used a queue and worker pool with the Go version. There's nothing stopping you from doing the same thing in Node.
I think a lot of people forget this sometimes and just spawn off unlimited multiple simultaneous requests in Node apps. Doesnt matter what language you're using, that's never going to turn out well
> They could probably have solved this by scaling out, but I'm not going to fault their diagnosis or response.
Just want to add that EC2 instances aren't exactly cheap either when it comes down to it. I'm sure budgets are tight at Digg. Far better to first try to solve the problem with refactoring than throwing more hardware at it.
Every single "Why we switched from Node.js to Go" article I've ever read is rooted in misunderstanding of Node.js - This one is no exception. The problem was that the codebase was bloated and probably time to scale up or scale out. I bet that if an experienced Node.js developer rewrote the entire service again in Node, you would also get 2x speedup.
I can't believe the CTO of digg bought this but I guess ultimately, it's also about making engineers happy.
Eventually Go will also reach a point where it will start lagging and then some smart engineer will conclude that you need to switch to C++! Why can't developers just admit that they don't know how to scale a service?! It's not the language that's the problem, it's you; it's always you!
I'm not a node expert, but here is what seems like the crux of the problem:
> Consequently, when any request timeouts happened, the event and its associated callback was put on an already overloaded message queue. While the timeout event might occur at 1 second, the callback wasn’t getting processed until all other messages currently on the queue, and their corresponding callback code, were finished executing (potentially seconds later).
Seems pretty specific to node, no? I mean, just looking at what the article says (again, I'm not a Node or Go expert) it seems like this problem was directly alleviated by Go and the speedup wasn't because Go is closer to the metal, but ultimately had a different method of handling many async network calls.
I don't think it was alleviated directly by Go, I think it was alleviated directly by improved parallelism... By the descriptions given, their go implementation uses a worker pool to help split active work across threads, while their Node implementation did not (though in Node you'd shard across processes rather than threads) - or at least it doesn't seem like it did (despite admitting to spending a bunch of time doing processing in JS between s3 requests).
Another nit: The article also implies that the height of the flame graph would indicate a performance issue, which I argue is inaccurate: Since the height of a flame graph measures stack depth, I argue it also represents abstraction. So long as most of those stacks are are equal-width, then the abstraction is low overhead (and probably has value to justify it). Generally with a flame graph you want to be making comparisons within the graph, comparing the time spent in each call (so the relative widths); not commenting on its shape in-general (though it does always make for a pretty blog post).
Yes it is, Go is truly concurrent, NodeJS is not. Go has green threads in userland, NodeJS does not. A Go program can actually perform multiple operations at the same time since Go coroutines are implemented at the language level, NodeJS cannot. Obviously Javascript is more expressive but Go programs are easier to read and maintain. With Javascript anything goes.
> Go has multithreaded concurrency. Node JS has multiprocess concurrency.
Go also has "multi-process" concurrency. Go is both concurrent and parallel, Go coroutines will spawn on every processor available by default. NodeJS needs an external tool like Redis to enable "multi-process" communication. Go does not.
Nonsense. You don't need redis for multiprocess communication in Node - child processes spawned have a communication channel with the parent whereby they can efficiently send messages (in the form of json objects) back and forth. (Or you could open your own shared file descriptor, like how you could do multiprocess message passing in any lower-level language.) There is little difference in usability between message-passing goroutines and message-passing child processes in node - just less syntactic sugar.
Go makes it more frictionless to write parallel code than in Node, but its certainly not impossible to write good parallel code in Node.
I wouldn't blame you nor the article author for not understanding the distinction - most people hear "Node is single threaded" and just accept that at face value. Coming from Ruby or Python, where the fork-process-to-be-parallel scheme is common, I am familiar with this variety of parallelism.
I guess you're probably not familiar with the whole "Why I switched from Node.js to Go" rhetoric - This article is just one of many.
I've heard this "The Node.js event loop gets saturated" argument so many times - It's the #1 argument used in all articles which promote switching from Node.js to Go.
The real reason why the event loop is saturated is because your Node.js process is doing too much work! The solution is to split your process into two (use the Cluster module or a load balancer - It's super easy).
With goroutines and the like, all Go actually does is delay the inevitable point when you have to split your main process into two... It doesn't solve the problem, it just delays it...
I don't know how Go handles the situation where a process gets too many requests... Maybe it just drops requests (instead of adding latency like Node.js)? I don't know... But once a program maxes out on total available CPU, bad things will happen! It doesn't matter what language/engine is being used; whether requests will start lagging + timing out or that they will start to get rejected outright; programs cannot transcend CPU constraints.
I don't mind people switching to Go from Node.js because "they feel like it", but I don't think it's fair to write an article about it claiming that Node.js is unfit for technical reasons.
There is no meat behind this argument at all and I'm just tired of seeing it resurface again and again and again on HN. It's starting to look like a propaganda campaign.
> With goroutines and the like, all Go actually does is delay the inevitable point when you have to split your main process into two... It doesn't solve the problem, it just delays it...
Yes, but Go's concurrency model is far easier to reason about. "I _want_ to do this thing", and the runtime handles the _how_ and the doing. No need to draw up some IPC messaging protocol to handle trivial tasks safely. It's a language feature (ugh sounds so jingoistic). The cluster module is nothing like goroutines in concept or in practice.
You're making some mighty big assertions there, all the while disregarding the obvious weakness of a _single_threaded_ event-loop. I'm a total Node.JS fanboy since 0.2, but as was mentioned elsewhere in the comments, every language is making trade-offs: substituting shiny new concerns for it's own achilles heel.
You're giving Node devs too much credit. Most Node devs know nothing beyond `fs` and `express`. For some strange reason they refuse to learn about the standard library beyond the very basics.
Go and Javascript/Node.js are quite different languages in their feature lists and main implementations.
To mention a big one, Go is mostly statically and strongly typed, in stark contrast to Javascript. Also it has significantly fewer misdesigns (such as the equality operator being weird). Are you claiming that such features don't matter for the effectivity of the platform?
The syntax of your language doesn't change its performance when running.
I can appreciate that not everyone likes JavaScript - it's all a matter of personal preference I guess - but saying that Go is a better language because of the syntax is like saying bananas taste better because of their color.
They are two very different languages with each their strengths and weaknesses. Does JavaScript have a few design flaws? Absolutely! More so than many other languages? Quite possibly! Does it make me write worse software if I'm aware of those quirks? Most certainly not.
> The syntax of your language doesn't change its performance when running.
No need to bring up syntax. It's a superficial feature and has nothing to do with the actually important features I listed.
> Does it make me write worse software if I'm aware of those quirks? Most certainly not.
Is there some limit where good language features stop mattering? I guess we would obviously agree that creating web software with x86 assembler wouldn't be a good idea -- even if we could imagine a comparable library/framework situation.
> The syntax of your language doesn't change its performance when running.
No but the difference between static typing and dynamic typing actually does. Go compiler performs a lot of optimizations that no Javascript engine can.
Those you mention no. There are tools for typechecking (flow) and to be honest in years of JS programming only rarely have I had bugs caused by a variable of the wrong type being passed into a function. As for the "weird" equality operator, again, not an issue, any linter will force you to use === and then you are good to go.
If you are talking about the callback queue getting too full and causing delays you may have a point, but the objections that you have raised are quite petty.
>Those you mention no. There are tools for typechecking (flow) and to be honest in years of JS programming only rarely have I had bugs caused by a variable of the wrong type being passed into a function.
Are you certain that that is true, or do you think it might just be that you're so used to the dynamic woes that you nearly ignore them?
Writing code that type-checks whatever input is coming into your functions may only take time, but what really are woes that you will have fewer of the stronger the type system, is that you have to design the code that runs when those checks fail.
That designing requires thinking, and thinking takes your attention away from whatever your goals are, whether you are used to it or not.
Strong type systems come with their own things that require extra thinking, but I think it is less than what is required to write robust dynamically typed code.
Many projects don't need truly robust code. Then, the dynamic language may still win.
> Strong type systems come with their own things that require extra thinking, but I think it is less than ...
Well, that's your bias. As I said in practice I don't see a problem. And also, as I said, there are static analysers, and you really HAVE to have strong type you can always use typescript.
Perhaps I'm doing a strawman here, sorry in advance.
But you seem to imply that you implement strong type checking into all(?) your functions yourself and think that is a better solution than having your platform do it for you automatically and systematically. Or is it about the freedom of not doing it?
> Eventually Go will also reach a point where it will start lagging and then some smart engineer will conclude that you need to switch to C++!
Go is much better suited for large codebases than Node.js though, so when it becomes bloated again, it'll be easier to fix problems with/optimize the existing code and rewrites can be avoided.
Long flame graphs in node vs. golang is probably 50% refactor. The other 50% is the async calling pattern in node that leads to deep stack frames.
The overloaded event queue? If that is true the not the queue is overloaded but there are too many entries that are ready to run and at least one takes a lot of time running to completion. So there could be system level problems (flood incoming data) or a code problem. There could also be a low level i/o problem in node showing up under high load. It would have been interesting to learn the true root cause.
Considering the time already spent and the results they got and the cost of investment of switching to golang I would say the decision is rationale. The whole point of micro-services is to maintain the tooling freedom. You are trading off the need of expensive support contracts with the bounded risk of re-engineering a small service.
> I bet that if an experienced Node.js developer rewrote the entire service again in Node, you would also get 2x speedup.
Maybe, but why not try Go on a small piece and see how it works.
> I can't believe the CTO of digg bought this...
You make it seem like the CTO was somehow being conned. Like Go was a bad choice. Maybe it was as simple as wanting to see what another language offered and this was a good time to try it out.
> Why can't developers just admit that they don't know how to scale a service?! It's not the language that's the problem, it's you; it's always you!
Oddly enough, this is a language feature built into Go from the get-Go. The language is designed such that a mid level programmer can ramp up and write performant Go code in a short amount of time without needing years (or even months) of experience, theory, classes, third party libraries, tooling, etc. It's a bold experiment and it may not succeed. Has it for Digg?
Building in C++ is completely rational after you've hit such scale that server costs outstrip the engineering costs of using a low level language. It's not like fb, google and amazon don't know how to scale a service.
>> Two weeks later, after my initial crash course introduction to Golang,
we had a brand new Octo service up and running.
>> you would also get 2x speedup.
The win wasn't the 2x speedup, it was not choking under heavy load.
Success..? I'm assuming this woman is a payed employee. She claims to have worked with nodejs for months on end, she attempts and fails many times to fix a bug, or possibly refactor a simple utility, blames the technology and decides she should spend her companies time and money learning a new language and rebuilding the service, slowly, with a brand new language.
In other words: She couldn't solve the issue because she couldn't find a tutorial. The only tutorial she could find used Go, so she switched to Go. Brilliant success.
As someone who wrote Node for roughly 5 years, it's VERY hard to write robust Node code, I would never trust it to mission critical work personally. Go isn't perfect, and it'll likely remain that way but it's much better suited to this sort of thing.
I've just started learning Ruby on Rails but Go I'm thinking about Go more often and am curious about working with it.
Are there Rails-like frameworks for Go? Why I try to Google that, I find lots of stories about people switching from Rails to Go, but not a lot of frameworks discussions.
There are a couple vaguely Rails like frameworks but Go definitely doesn't lend itself to immersing itself into a web framework like that. Overall they tend to be much lighter and sit closer the the Go core language itself.
I switched from ruby (and rails) to golang this year as my main language.
If you're thinking "I'm hesitating between ruby on rails and golang", you should probably go with ruby on rails (no pun intended).
Golang is more low level, and it expects you prefer conceptual simplicity over abstraction [EDIT: this is why you won't see prominent golang rails-like framework]. This also means you're supposed to already know how to architecture a web application (if that's what you want to do) and to know the techs behind it.
You mention you started to learn RoR, but do you have previous and consequent experience with web development? If not, definitely use RoR, it will provide clear guidelines about how to build a web app.
> Are there Rails-like frameworks for Go? Why I try to Google that, I find lots of stories about people switching from Rails to Go, but not a lot of frameworks discussions.
The definitive answer is no. It's no because given Go's type system it is impossible to create a framework like Rails while staying type safe. One could create something that involves a lot of reflection tricks but you'd loose type safety. The alternative would be code generation + DSL but why use Go at first place if you end up relying on a complex DSL that needs to be compiled to Go first ? it doesn't make sense.
The best you'll get with Go are Sinatra or Flask clones. But Rails? no way.
If you want the Rails experience use Rails. Go is more verbose and more rigid than Ruby anyway.
When I think of event loop blocking, the first thing I think of is GC. I find it strange that the author doesn't mention or consider GC. They might easily have been creating too many closures in a loop (a closure per data item, instead of a single handler for all data items), or millions of long lived pointers (using a Javascript object as a hash table for caching, instead of a flat buffer-based hash table).
Did the author tune the size of the libuv threadpool or did they leave it at the default of 4 threads?
Also, the response times given both for Node and their new system in Go are really not that great. The results for Go are an anti-climax considering it was a rewrite. They could probably undercut both times with a better design, regardless of language.
Node is a great control plane, and you can always use C++ as your data plane.
Step 3 - Learn a new technology which doesn't have that achilles heel.
Step 4 - Goto Step 2.
To break this cycle, you need to appreciate that every language and library has its strengths and weaknesses. Learning what they are, and tailoring your architecture appropriately is better than jumping from one place to another every time you hit a new problem.
What does this even mean? She doesn't understand how Node, or even JavaScript work, spends forever creating "one-offs" and gives up, finds a relevant tutorial in Go, which worked, proving that Go was the best solution, because Go is better than Node in this situation and probably all situations, and she is good at her job and not wasteful, this article is relevant and a great read.
Most all major problems with scale are due to single-process threads. Node.js had it's moment with single-process thread. Blocking will happen even with events but @ scale. He solved his issue with multiple queue-workers + messaging between. This alleviated his blocking, and some people spoke to the fact that node had a cluster workflow which could have also alleviated this issue as well.
Some languages make this workflow more obvious, and some people get stuck in the general way of thinking about problems. I think this is more of an engineering problem and thinking out of the box and ultimately his direction in looking for resources & help within the general community.
I think when you run into problems like these, you need to try to contact the community at large to ask about scaling, and ways scaling is generally done with these types of tasks. Thinking you're the only one who ever experiences trouble scaling is very juvenile and expresses inexperience & ultimately ego. Yes, he solved a problem, but he felt he had to uproot himself in the code already there to do it.
I guess he solved his problem, but he had a quicker path very possibly, he just didn't know it existed.
This article is about not being able to analyze problems and only being able to express shallow solutions in terms of permutations of existing solutions without actually thinking about the underlying problem.
Whether the author knows it or not.
Hence, this does not describe a good argument for switching to Go. It describes a good argument for better education, training and mentoring of developers so that they become programmers rather than "random assemblers of bits and pieces".
70 comments
[ 3.2 ms ] story [ 142 ms ] threadOriginal thread based on Medium post.
https://nodejs.org/api/cluster.html#cluster_cluster
This is a valid comment considering that the EC2 instance mentioned is a 2 core box, and there is a possibly that the node implementation didn't take advantage of both cores.
- Go has multi-core support out of the box
- Node.js has to be shown a little love, and has limitations. But this would have been a great use-case for Cluster.
If the Node implementation was using all cores, then I wouldn't have asked this question. As it stands I don't really know what we are comparing even with all the graphs.
Also we are still missing a little info on how they handled the requests. When you have a endpoint on a node server that is not a simple one, it is best to queue those before handling them (yes a additional queue). This prevents the process from choking because it just opened up NNN requests to S3.
Most developers are not working with hundreds of HEAVY requests a minute, but in Node.js this usually has to be handled with care, and does not work out of the box.
But if you want to pick up a shinier hammer then Go for it.
Also, I strongly suspect that an app that has been refactored multiple times by people well versed in node.js is probably already doing something to leverage multi-core...
Personal opinion: Because comments like your original one are annoying as hell.
They never fail to show up. Someone post an interesting article one how they solved a particular problem and someone always have to show up with these pointless comments. Not everyone will know about, in this case Cluster, and it's not relevant without an explanation.
Article and blog posts about Go seems to attract comments like: You might like Rust/Elm/Elixir and "that problem could have been solved using X/Y/Z package or framework".
More often than not it's missing the point, which often is people, with little or no Go experience are managing to solve really complex problems within very short time frames.
I don't see it as a criticism Node.js either, it's just that Node wasn't the right tool for the job, in this case. If you pick the correct tools and know the limitations of your tools, then it becomes less important how proficient you are with those tools (not irrelevant of cause).
I have had node do 12000 requests a minute with no problem (on a single core). In this case they had 4 two core boxes, so they needed each one to at least perform 42 requests a second under max load.
I want to know more :)
http://marcio.io/2015/07/handling-1-million-requests-per-min... etc
If you know the limitations of your tools then you can work around them. This was clearly not done in this case.
They could probably have solved this by scaling out, but I'm not going to fault their diagnosis or response.
Don't spawn off heavy requests without a throttle / queue, or else you will overload your single core.
We normally don't think about this because most requests are just tiny queries / small http requests.
I think a lot of people forget this sometimes and just spawn off unlimited multiple simultaneous requests in Node apps. Doesnt matter what language you're using, that's never going to turn out well
Just want to add that EC2 instances aren't exactly cheap either when it comes down to it. I'm sure budgets are tight at Digg. Far better to first try to solve the problem with refactoring than throwing more hardware at it.
I can't believe the CTO of digg bought this but I guess ultimately, it's also about making engineers happy.
Eventually Go will also reach a point where it will start lagging and then some smart engineer will conclude that you need to switch to C++! Why can't developers just admit that they don't know how to scale a service?! It's not the language that's the problem, it's you; it's always you!
> Consequently, when any request timeouts happened, the event and its associated callback was put on an already overloaded message queue. While the timeout event might occur at 1 second, the callback wasn’t getting processed until all other messages currently on the queue, and their corresponding callback code, were finished executing (potentially seconds later).
Seems pretty specific to node, no? I mean, just looking at what the article says (again, I'm not a Node or Go expert) it seems like this problem was directly alleviated by Go and the speedup wasn't because Go is closer to the metal, but ultimately had a different method of handling many async network calls.
So the take away is Go is better at handling massive amounts of open outgoing network calls.
In Node.js you want to restrict this to prevent the server from choking. If they did that I bet performance would be very similar.
Another nit: The article also implies that the height of the flame graph would indicate a performance issue, which I argue is inaccurate: Since the height of a flame graph measures stack depth, I argue it also represents abstraction. So long as most of those stacks are are equal-width, then the abstraction is low overhead (and probably has value to justify it). Generally with a flame graph you want to be making comparisons within the graph, comparing the time spent in each call (so the relative widths); not commenting on its shape in-general (though it does always make for a pretty blog post).
Yes it is, Go is truly concurrent, NodeJS is not. Go has green threads in userland, NodeJS does not. A Go program can actually perform multiple operations at the same time since Go coroutines are implemented at the language level, NodeJS cannot. Obviously Javascript is more expressive but Go programs are easier to read and maintain. With Javascript anything goes.
Go also has "multi-process" concurrency. Go is both concurrent and parallel, Go coroutines will spawn on every processor available by default. NodeJS needs an external tool like Redis to enable "multi-process" communication. Go does not.
Go makes it more frictionless to write parallel code than in Node, but its certainly not impossible to write good parallel code in Node.
I wouldn't blame you nor the article author for not understanding the distinction - most people hear "Node is single threaded" and just accept that at face value. Coming from Ruby or Python, where the fork-process-to-be-parallel scheme is common, I am familiar with this variety of parallelism.
I've heard this "The Node.js event loop gets saturated" argument so many times - It's the #1 argument used in all articles which promote switching from Node.js to Go.
The real reason why the event loop is saturated is because your Node.js process is doing too much work! The solution is to split your process into two (use the Cluster module or a load balancer - It's super easy).
With goroutines and the like, all Go actually does is delay the inevitable point when you have to split your main process into two... It doesn't solve the problem, it just delays it...
I don't know how Go handles the situation where a process gets too many requests... Maybe it just drops requests (instead of adding latency like Node.js)? I don't know... But once a program maxes out on total available CPU, bad things will happen! It doesn't matter what language/engine is being used; whether requests will start lagging + timing out or that they will start to get rejected outright; programs cannot transcend CPU constraints.
I don't mind people switching to Go from Node.js because "they feel like it", but I don't think it's fair to write an article about it claiming that Node.js is unfit for technical reasons.
There is no meat behind this argument at all and I'm just tired of seeing it resurface again and again and again on HN. It's starting to look like a propaganda campaign.
Yes, but Go's concurrency model is far easier to reason about. "I _want_ to do this thing", and the runtime handles the _how_ and the doing. No need to draw up some IPC messaging protocol to handle trivial tasks safely. It's a language feature (ugh sounds so jingoistic). The cluster module is nothing like goroutines in concept or in practice.
You're making some mighty big assertions there, all the while disregarding the obvious weakness of a _single_threaded_ event-loop. I'm a total Node.JS fanboy since 0.2, but as was mentioned elsewhere in the comments, every language is making trade-offs: substituting shiny new concerns for it's own achilles heel.
To mention a big one, Go is mostly statically and strongly typed, in stark contrast to Javascript. Also it has significantly fewer misdesigns (such as the equality operator being weird). Are you claiming that such features don't matter for the effectivity of the platform?
I can appreciate that not everyone likes JavaScript - it's all a matter of personal preference I guess - but saying that Go is a better language because of the syntax is like saying bananas taste better because of their color.
They are two very different languages with each their strengths and weaknesses. Does JavaScript have a few design flaws? Absolutely! More so than many other languages? Quite possibly! Does it make me write worse software if I'm aware of those quirks? Most certainly not.
No need to bring up syntax. It's a superficial feature and has nothing to do with the actually important features I listed.
> Does it make me write worse software if I'm aware of those quirks? Most certainly not.
Is there some limit where good language features stop mattering? I guess we would obviously agree that creating web software with x86 assembler wouldn't be a good idea -- even if we could imagine a comparable library/framework situation.
No but the difference between static typing and dynamic typing actually does. Go compiler performs a lot of optimizations that no Javascript engine can.
If you are talking about the callback queue getting too full and causing delays you may have a point, but the objections that you have raised are quite petty.
Are you certain that that is true, or do you think it might just be that you're so used to the dynamic woes that you nearly ignore them?
That designing requires thinking, and thinking takes your attention away from whatever your goals are, whether you are used to it or not.
Strong type systems come with their own things that require extra thinking, but I think it is less than what is required to write robust dynamically typed code.
Many projects don't need truly robust code. Then, the dynamic language may still win.
Well, that's your bias. As I said in practice I don't see a problem. And also, as I said, there are static analysers, and you really HAVE to have strong type you can always use typescript.
But you seem to imply that you implement strong type checking into all(?) your functions yourself and think that is a better solution than having your platform do it for you automatically and systematically. Or is it about the freedom of not doing it?
Go is much better suited for large codebases than Node.js though, so when it becomes bloated again, it'll be easier to fix problems with/optimize the existing code and rewrites can be avoided.
Long flame graphs in node vs. golang is probably 50% refactor. The other 50% is the async calling pattern in node that leads to deep stack frames.
The overloaded event queue? If that is true the not the queue is overloaded but there are too many entries that are ready to run and at least one takes a lot of time running to completion. So there could be system level problems (flood incoming data) or a code problem. There could also be a low level i/o problem in node showing up under high load. It would have been interesting to learn the true root cause.
Considering the time already spent and the results they got and the cost of investment of switching to golang I would say the decision is rationale. The whole point of micro-services is to maintain the tooling freedom. You are trading off the need of expensive support contracts with the bounded risk of re-engineering a small service.
Maybe, but why not try Go on a small piece and see how it works.
> I can't believe the CTO of digg bought this...
You make it seem like the CTO was somehow being conned. Like Go was a bad choice. Maybe it was as simple as wanting to see what another language offered and this was a good time to try it out.
> Why can't developers just admit that they don't know how to scale a service?! It's not the language that's the problem, it's you; it's always you!
Oddly enough, this is a language feature built into Go from the get-Go. The language is designed such that a mid level programmer can ramp up and write performant Go code in a short amount of time without needing years (or even months) of experience, theory, classes, third party libraries, tooling, etc. It's a bold experiment and it may not succeed. Has it for Digg?
I was also thinking that Octo sounds way too much like a monolith. I wouldn't let retries tax the same instance.
I bet they could use something like RabbitMQ to sort work distribution and retries between instances without relying on nested callbacks.
Are there Rails-like frameworks for Go? Why I try to Google that, I find lots of stories about people switching from Rails to Go, but not a lot of frameworks discussions.
Here's a good list of the major frameworks for Go. https://github.com/avelino/awesome-go#web-frameworks
Also, the complete Awesome Go list is a wonderful resource for browsing some of the best Go libraries out there.
I switched from ruby (and rails) to golang this year as my main language.
If you're thinking "I'm hesitating between ruby on rails and golang", you should probably go with ruby on rails (no pun intended).
Golang is more low level, and it expects you prefer conceptual simplicity over abstraction [EDIT: this is why you won't see prominent golang rails-like framework]. This also means you're supposed to already know how to architecture a web application (if that's what you want to do) and to know the techs behind it.
You mention you started to learn RoR, but do you have previous and consequent experience with web development? If not, definitely use RoR, it will provide clear guidelines about how to build a web app.
I think I'm going to keep plugging away at Rails. If I ever reach a point where scale is becoming an issue, then I've got a great problem to have.
The definitive answer is no. It's no because given Go's type system it is impossible to create a framework like Rails while staying type safe. One could create something that involves a lot of reflection tricks but you'd loose type safety. The alternative would be code generation + DSL but why use Go at first place if you end up relying on a complex DSL that needs to be compiled to Go first ? it doesn't make sense.
The best you'll get with Go are Sinatra or Flask clones. But Rails? no way.
If you want the Rails experience use Rails. Go is more verbose and more rigid than Ruby anyway.
Did the author tune the size of the libuv threadpool or did they leave it at the default of 4 threads?
Also, the response times given both for Node and their new system in Go are really not that great. The results for Go are an anti-climax considering it was a rewrite. They could probably undercut both times with a better design, regardless of language.
Node is a great control plane, and you can always use C++ as your data plane.
200ms of latency is quite normal for S3, not designed with that in mind.
Step 2 - Use it until you find its achilles heel.
Step 3 - Learn a new technology which doesn't have that achilles heel.
Step 4 - Goto Step 2.
To break this cycle, you need to appreciate that every language and library has its strengths and weaknesses. Learning what they are, and tailoring your architecture appropriately is better than jumping from one place to another every time you hit a new problem.
Some languages make this workflow more obvious, and some people get stuck in the general way of thinking about problems. I think this is more of an engineering problem and thinking out of the box and ultimately his direction in looking for resources & help within the general community.
I think when you run into problems like these, you need to try to contact the community at large to ask about scaling, and ways scaling is generally done with these types of tasks. Thinking you're the only one who ever experiences trouble scaling is very juvenile and expresses inexperience & ultimately ego. Yes, he solved a problem, but he felt he had to uproot himself in the code already there to do it.
I guess he solved his problem, but he had a quicker path very possibly, he just didn't know it existed.
Whether the author knows it or not.
Hence, this does not describe a good argument for switching to Go. It describes a good argument for better education, training and mentoring of developers so that they become programmers rather than "random assemblers of bits and pieces".