I have used Node.js in production for about 5 years now and I must agree with the sentiment that JavaScript is "Easy to learn, impossible to master". Yes, error handling is a little confusing at first but it stems from JavaScript's asynchronous nature which is naturally complex for the linear mind.
My personal sentiment is to just use Promises, like everywhere. ES7 async/await will really help with this too.
> Coming from other languages such as Python, Ruby or PHP you’d expect throwing and catching errors, or even returning an error from a function would be a straightforward way of handling errors. Not so with Node. Instead, you get to pass your errors around in your callbacks (or promises) - thats right, no throwing of exceptions.
Promises let you throw errors normally. They will propagate up the call stack in a similar manner. With bluebird, you will also get full stack traces in development mode and the performance penalty for that isn't too bad.
> The last thing that I found frustrating was the lack of standards. Everyone seems to have their own idea of how the above points should be handled. Callbacks? Promises? Error handling? Build scripts?
Promises are in ES6 (i don't think it gets more standard than that) and have well defined semantics, including error handling, shared between libraries: https://promisesaplus.com/
I know that Bluebird's promisifyAll might seem like a bit of a hack, but just try it out. It works surprisingly well, and its really painstaikingly tuned for near-zero performance loss. It will probably be both less painful to do and more performant than any manual attempt to wrap a callback based API into a promise one.
Kind of related: One thing that has bit me a few times with promises is accidentally missing a return in the promise body. As such whatever called the promise just sits there forever, waiting for something that will never come. Is there a way in Bluebird to have it throw an exception or similar if the function executes without returning anything?
I sort of don't understand your problem. If you forget to return a value won't the thing consuming the Promise just get undefined as the value (and perhaps throws an exception expecting the value to be something other than undefined)?
At the time the solution was "use domains", but I think domains are deprecated now. It was painful enough that I have stuck with Rails since then. I'm glad to hear that Promises are an improvement!
Domains have been deprecated since at least 0.10. As of yet there's no replacement for them and all node apps should be using them. There's no other way to catch "But ... but ... that can't happen!" type errors.
There is no replacement because they're a fundamentally broken idea. They require the following to happen, in that order:
* V8 needs to optimize try-finally
* Node core needs to add try-finally at every single place where callbacks are invoked and make sure all state and resource cleanup is properly done to support domains
* Popular libraries need to also add try-finally handlers for the above.
As to why this is a problem in node and not so much in other languages, its because with node callbacks, the call stack goes both ways. In other languages, libraries mostly call their dependencies' code. In node's CPS style, you call the library but the library also calls your closure code. The semantics for the 2nd part aren't well defined in node - the loose law basically says: I wont call you twice, I'll try not to call you synchronously, and you wont throw (and if you do the behavior is undefined).
With promises there is a contract and its enforced by the promise implementation. Since Promises actually have error semantics, you can build resource management strategies on top of them. http://promise-nuggets.github.io/articles/21-context-manager... - and consequently there is no reason to crash your server on errors.
My recommendation is to just use songbird (which exposes the forthcoming promise API from core, built on bluebird), async/await and the async `trycatch` library so you don't have to worry about a 3rd party package's choice of asynchrony. It also comes with optional long stack traces.
in my experience, async/await is a great way to layer indirection and obfuscation over what is still callback hell. it may look better in the editor, but it's hell to debug.
EDIT: Oops, I misread the parent. You're right, async/await isn't easy to debug, but it's getting better, and it's a tradeoff. For now, I recommend debugging the compiled code (using generators) without source maps to avoid some of the quirks of source maps. In my experience, the tradeoff is worth it for new node.js developers because it offloads the asynchrony contract (e.g., calling a callback OAOO) to the language.
When I initially encountered event -based processing (libevent in C), those callbacks were indeed difficult to wrap my mind around. But I learned to structure the code in the editor, keeping everything together which made it manageable.
I 've worked with node for a couple of months now, and I find promises to be more confusing, because it obfuscates the callback in my mind, and makes it look like ordinary function calls.
The advantage of promises is that you can return them as a type, you can't do that with a callback. If you fetch something from a database , your data access object can return a promise and let the client code deal with the result. Promises are composable by nature, callbacks are not.
async/await is tiny amount of sugar over generators and promises (virtually the only thing that changes is `yield` -> `await` and `function*` -> `async function`), not callbacks.
I've spent a lot of time writing Javascript on the front-end in the last year using both React and React Native. I've found the React ecosystem to be a sane, productive and enjoyable development environment.
Interested in sharing more model logic between our front- and backends, I also investigated writing some new backend features using Node (we're currently developing with Rails). But after days of research and playing around with the available options, I came to a conclusion similar to Gavin's -- any reasonably complex backend requires you to either roll your own everything, or try to cobble together literally hundreds of tiny dependencies that weren't built to go together, and then somehow keep track of all of their regressions and breaking changes.
Node's fantastic performance and unique ability to share logic between client and server are enticing, but I just don't trust the community and "best practices" around it enough to bet the farm on it for now.
The same problems you cite for node.js are the reasons why a lot of devs love node.js.
Its much easier to do your own research and find the best module to solve a particular problem you are having then to shoehorn into some larger monolithic framework.
Also its a lot more fundamental then that - Node.js has prolly the fastest iteration cycle for any platform out there since its so easy to create your own module - it leads to some sort of Cambrian explosion of innovation and experimentation.
EDIT:
Also OP seems to think callback hell and async programming is bad. The important thing is those things are problems for python/.... too !
Its just that python doesn't have a good programming model to even begin to address those concerns.
JavaScript at-least tries to say - "hey this is a problem we need to deal with - concurrency is a issues we all face "
So when devs complain about callback hell - its just that they have never tried to use python to do async in a neat way.
> python doesn't have a good programming model to even begin to address those concerns
This isn't even close to true. Anything JavaScript has to express logic in the face of asynchrony, Python has too. There are a half-dozen asynchronous web servers written in Python.
There's not as much of a culture of writing APIs that way in Python because it's generally a terrible way to program, and threads/OS processes are good enough for basically everything except HTTP servers with absurd numbers of concurrent connections.
> There's not as much of a culture of writing APIs that way in Python because it's generally a terrible way to program
I would love to see evidence for you making that statement. Almost every programmer would put out their fav programming language as the 'right' way to program.
> everything except HTTP servers with absurd numbers of concurrent connection
Once you introduce async operations in your code - you need to follow the execution path through. http request can be async - but then what if the http request results you doing a db lookup or some form of file handling ? you need to make the whole thing event driven.
It's not like we didn't have cooperative multitasking for 50 years. Having threads/processes and a scheduler is easier and safer, full stop. Potentially long-running portions of the program don't need to be arbitrarily chopped up to yield control back to the server, because they are pre-empted. Your system is no longer at the mercy of the worst code within it.
Both nginx and Apache's event MPM handle HTTP connections with events while the app backends are still using preemptive threads for running the HTTP handler code, so it's clearly not the case that "you need to make the whole thing event driven." You just need programmers who don't think, "well since the browser doesn't expose threads to JavaScript programmers, clearly they are useless."
That Cambrian explosion is fun when you are programming, but if you are writing a product, you will want to make sure you don't pick a dinosaurus or trilobite to build it on.
These types of articles make me laugh. Typically a dev with many many years experience with one language, learned all it's quirks, standards, etc decides to try Node.js because it's the "new hot fun toy", and expect it to work like their old language, and realize that is not how it works, doesn't know where to find what and fails real hard to realize that JavaScript in general is in a huge influx of updating at this moment, which by nature propagates to Node.js.
The end result is they get frustrated and go back to their old language.
Yeah, so some js devs might need to stop overselling javascript to everyone, pretending it is the one language that people are waiting for years...Just saying.
You should try to surround yourself with developers who don't have such attitudes. I am a fan of JavaScript, I love Node.js and I will often times suggest it to newbies. But I don't pretend it's the be all/end all of languages. Just like any other language it has its strengths and weaknesses. It's up to you to decide if it's the right choice for you.
You should try to surround yourself with developers who don't have such attitudes.
The trouble is that while you can do this for yourself to some extent, you can't necessarily do it for the people you have to work with. I've worked on a project where one guy came in to do a simple database back-end for an embedded system and decided Node and its ecosystem were appropriate. He seemed to have some experience with those tools, so the rest of the team went along with his decision. A year later, there was still little useful code actually running on the required platforms. One of the other guys got fed up and wrote the basic case using C and SQLite in less than a day. That guy did have some relevant experience and had been questioning the use of Node for that project from early on, but because the Node hype was so high-profile, management hadn't known which developer to believe. This is why it's useful to have articles like the one we're discussing, where people who have actually tried a tool in real life for a significant period of time share their experiences, even if those experiences weren't quite what they'd hoped for at the start.
The difficulty of getting Node itself up and running on the various platforms involved was a significant part of the problem. These weren't Linux boxes built around Xeons. It turns out that as soon as you go outside of the mainstream, the tools and documentation for building and running Node are awful compared to a lot of other languages.
The lack of standardisation and stability and the relatively poor dependency handling within the Node ecosystem were also major factors. Way too much time was spent just trying to make sure different machines really did have identical packages installed, figuring out how to pin everything down to make reproducible builds, and so on. Again, if you're happy just to shrinkwrap, npm install on your local machine, and wait if the repository is unavailable, that's one thing. However, as soon as you start talking about preserving exact configurations within 100% local build environments or building images to install on different machines, all kinds of problems crop up that have been well addressed by now within a lot of more mature language ecosystems but not so far with Node.
Another problem was that Node and its ecosystem are just so heavy overall, and this one is nothing to do with the variety of platforms in use. It's just a big runtime, and a lot of overhead because of the way libraries and dependencies are handled. This would be an issue for anyone operating with limited resources, whether it's trying to run something on a Raspberry Pi or trying to keep costs down by using the smallest possible instances of cloud-hosted virtual servers.
There did also seem to be an element of wanting to use the shiny new toys, and of stubbornly sticking with those toys for far longer than should probably have been allowed, so in that sense there were issues with the developer as well.
But there was also a willingness among the wider team to trust that developer, at least to begin with, because with Node's high profile and large ecosystem, no-one expected it to be as immature as it turned out to be as soon as you left the mainstream. For comparison, porting code written in traditional systems programming languages like C and C++ was done routinely, but even among the dynamic/scripting family, the more established server-side languages have caused far fewer problems than Node.
> The tools and documentation for building and running Node are awful compared to a lot of other languages.
Can you be more specific?
> It's just a big runtime, and a lot of overhead because of the way libraries and dependencies are handled.
Compared to what exactly? I don't see any mainstream dynamic language except Lua doing better. Ruby? Python?
If you don't need native modules (which is actually the case in a surprising number of situations) there is no build process, except making a tarball of node_modules and the project. If you do need native modules, the situation is slightly worse than C or C++, but not significantly: you generally need to build on a machine with the same architecture (Setting up cross-compilation with C isn't very easy either)
> A lot of overhead because of the way libraries and dependencies are handled.
This has some truth to it, but in my experience its overblown. Being just a little bit careful with your dependencies goes a long way.
On the other hand, if you want to run node on something like this: https://www.olimex.com/Products/OLinuXino/iMX233/iMX233-OLin... that will definitely be a problem. Thats about the lowest specced hardware that can (barely) run it, but 64MB RAM just isn't enough, really.
Sorry, it wasn't my part of the project so I can't offer much detail. I know they never got as far as a Node build that passed the full test suite on some platforms. Many of the problems seemed to be around how floating point is handled on different types of CPU.
Compared to what exactly? I don't see any mainstream dynamic language except Lua doing better. Ruby? Python?
The total footprint for Node and supporting packages was taking up a few GB more than the equivalents for some of those other languages you mentioned. If you're working on a system where your non-volatile storage is something like a microSD card or flash chip, "a few GB" can be a huge difference.
If you do need native modules, the situation is slightly worse than C or C++, but not significantly: you generally need to build on a machine with the same architecture (Setting up cross-compilation with C isn't very easy either)
Sorry, our experience has been very different on both counts. Building on a machine with the same architecture isn't necessarily a reasonable option if you're targeting platforms that aren't general purpose computers, and at best probably means jumping through a lot of hoops and figuring out a lot of the details for yourself. In contrast, cross-compiling a language like C is straightforward and reliable, the tools for doing it are mature and well documented, and many developers who have worked in embedded systems are familiar with the techniques.
No. The person doing the Node version was trying to use some of the NoSQL tools, and seemed to be spending much of their time documenting and implementing things you get as standard with a SQL database. The person who wrote the C based alternative just used SQLite as a demonstration of how much easier and more robust everything could be using a different database instead.
One of the striking ironies of the project was that although in principle a dynamic language like Node doesn't need a compilation step, just the JS source, while C code has to be compiled for every target platform, the reality was that getting the C code running was dramatically easier. A well-made C library like SQLite could be installed, cross-compiled and running on all the required platforms within literally a few minutes, while Node and several major packages were so flaky that the equivalent level of real world functionality was never achieved even after several months of work. Again just for comparison, some of those other dynamic languages you mentioned were in between but much closer to the shorter end of the spectrum. The Node ecosystem, in our experience, proved to be an exceptionally bad choice for this sort of work.
I'm afraid that based on what you said, it only made it clearer that this wasn't an issue with the ecosystem but with that particular developer. I don't really see how you can extrapolate that this was the node ecosystem's fault, when you have a developer that is trying to implement SQL on top of NoSQL and pull in a gigabyte of dependencies for an embedded platform. If a C++ developer was trying to implement the app on top of MongoDB's source code, that says more about the developer then about C++ and its ecosystem.
I might concede that the node ecosystem does have a disproportionate number of people with this mindset though :)
By the way, cross-compilation is also an option, although I haven't personally tried it:
A plus would be that only native modules have to be compiled, and only once. After that its possible to set up downloading them as binary blobs.
You got me curious about the application, by the way. If you let me know what it does, it might be fun to try and implement a prototype in node (with sane dependencies and easy to deploy on a RPi)
Just to be clear in case I've given the wrong idea, my point here is not really to criticise Node in general. I'm just saying, with the benefit of hindsight of course, that it was not the ideal choice for that particular job. In particular, because of the hype around Node, I think the people making the decisions (including the developer who was working on it) honestly didn't expect that, and perhaps weren't sure how to proceed when the problems I mentioned started mounting up.
(Edit: That said, the difficulties with size, portability and dependency management in the Node ecosystem that were exposed by this particular project are more general flaws with that Node ecosystem and not specific to this particular developer.)
You got me curious about the application, by the way. If you let me know what it does, it might be fun to try and implement a prototype in node (with sane dependencies and easy to deploy on a RPi)
It really wasn't anything very complicated by database standards, just storing some settings in a central place. Those settings were a bit more complicated and inter-related in this case than you could conveniently store using something as simple as a plain text file, so a lightweight database was the next logical step. As mentioned before, it was the kind of thing where you could implement the basic infrastructure required within a few hours using more suitable tools.
Okay. I've had vastly different experience with node on ARM. Much more reasonable dependencies size then what you report (by about 2 orders of magnitude) and excellent performance from the JIT. Combined with the ability to first write dynamic code then switch to static types, as well as the memory safety, its been a lot more pleasant than working with C.
So honestly, I really can't relate to most of what you said here :)
Please keep in mind that the Raspberry Pi was only an example. We're talking about embedded here, so there are all kinds of weird and wonderful CPUs and storage systems and I/O components being used in various devices that need to run the code I'm talking about. Certainly not all of them were ARM chips. I don't know which chips caused the big problems with porting Node, nor which specific packages, so it's entirely possible that whatever you've worked on was closer to the mainstream and supported just fine.
This "I love [some language]" thing seems to be quite common in JS circles. Maybe that is part of the problem. Being in love is not good foundation for making objective decisions.
It is one of my most cherished professional goals to avoid ever working with people who think like this. The message this sends to junior devs is so backwards and wrong-headed that it defies discourse.
But I see your point as being precisely backwards: it's the JS crew who constantly evangelize for their way of doing things, and that way of doing things is completely inappropriate for anyone who isn't already expert.
"Choose your own tools" is advice for experts, and literally nobody else. It smacks of the cowboy attitude, and that attitude is wildly unhelpful to almost anybody doing professional work in software. Frameworks and coding standards exist for a reason: not to be unreasonable strictures, but to provide guidance and sanity in a staggeringly complex field of endeavor.
Many new devs and junior devs are flocking to the JavaScript ecosystem, and it is one of the most troubled and chaotic ecosystems in all of software right now. Anyone who is not a complete JS badass is simply not going to find Node to be even a decent choice for learning best practices pertaining to the larger world of application development. And I feel I'm stating that politely.
So: yet another JavaScript Pro tossing out the "choose your tools like a Pro" advice-morsel is part of the problem, as I see it. I don't want to work with people who toss the kids into the deep end and hope a few can learn to swim real quick. I think people deserve a helping hand and a reasonable set of expectations.
The JavaScript ecosystem has become self-parodying. Anyone who is oblivious to that fact is inherently NOT a trustworthy witness. That is not to say that the whole thing is rotten and worthless, but it IS messy as heck, and refusing to acknowledge that is a sign that one is in denial about some pretty blatant facts.
> JavaScript in general is in a huge influx of updating at this moment
Isn't that the case basically all the time? Which is a perfectly suitable reason to avoid it at all costs except for the bare minimum required for front-end..
No, ES5 to ES2015 is a HUGE leap. ES2015 to ES2016 is a very minor jump as it only adds 2 new things to the spec.
> Which is a perfectly suitable reason to avoid it at all costs except for the bare minimum required for front-end..
And honestly, that is not my job to tell you not to do. If you want to avoid Node.js go ahead. I feel Java is something I should avoid at all costs. Doesn't make Java any worse or better. Same with Node.js.
You seem to enjoy switching tools and methodologies and see it as part of embracing Node.js.
Others don't have that luxury and just can't afford that.
And if your software needs to be around for more than 5 years, you want to keep your costs low.
How long would it take you to find a bug given a stack trace from a Node API with close to no clue which part part of your app registered that callback that isn't called because of the error?
Chances are, you don't even measure the time spent in code archeology as I like to call it.
The time probably is not significant right now as you kinda know your way around your code base...yet.
> You seem to enjoy switching tools and methodologies and see it as part of embracing Node.js.
I've been using the same tools for the last 7 months outside of switching from Angular 1.5 to React because I'm not a fan of Angular 2.0.
> Others don't have that luxury and just can't afford that.
Then don't? What is this sense of needing to change everything because something else comes out, that developers have?
If your current toolset works, why change it? Why rewrite a monolithic app in another fancy language or tool just because someone wrote a blog post boasting about that new tool/language?
Author actually wants to use Python. Used Node.JS regardless, for whatever reason.. It did not work the way Python works. Author is frustrated. Complains that JavaScript is not Python.
Small company ops guy here - a bit over a year ago, I used to joke with my devs, asking them "So, what's this month's recommended way of installing node?". More recently we've been experiencing dependency hell, which is exacerbated by our small team not having enough time to upgrade to the latest LTS release. Node is definitely a language that you have to manage. It doesn't sit in the background and let you get on with writing stuff.
Is it the right tool for X or Y? I can't say, I'm not a dev. But it does require a lot of hand-holding and keeping current with the zeitgeist.
sorry if that came off harsh, but that is actually the feedback, and it's valid. whether or not javascript/node is better or worse than python, it's pretty clear that bringing a python style approach to nodejs is going to cause problems, especially with error handling and async stuff.
Really, just learn to use callbacks properly. Well, wait, actually, you should skip that and start getting used to using promises instead--a big improvement. Well, I mean, until next version is ready, and we can start using async/await...until wasm makes better thought-out languages available.
Despite the sounds of this, I do like the idea of having an experimental platform with which to gain experience using wildly different approaches to the wildly changing world of web apps. I don't take it for granted that old language concepts will turn out to be the most useful for the web platform.
So, I'm okay building a website for the PTA or ceramics club with this, and I'm very interested in the experiences of others using a wide variety of technologies and approaches, but I'm not sure Node.js would be a sensible foundation to build a business on.
A lot of this is residual effects of the (slow) evolution of JavaScript. It was thrust into the spotlight missing a lot of features and these features have only recently been fixed by the language itself.
But Node has been around since 2009. 2009 JavaScript was missing a lot of features. It was basically a runtime only advanced users should use. The Node maintainers had to make a lot of decisions that now conflict (to some degree) with fixes that have come later to the language. They chose their callback style; now we have Promises. They chose to throw in methods rather than return an error in the callback (this makes it awkward to use fs with Promises without a wrapper library). They chose to implement their version of CommonJS, now we have the .mjs issue arise.
I've been on a similar learning curve with Node over the last year, and it has certainly been a rougher incline than other languages I've used.
The whole async situation needs to settle down, it's completely unacceptable to write code with callbacks, promises, etc. This is because they are not just challenging to deal with, but intrinsically wrong in concept. I have to wait for a database query to complete, then pass the next thing to do to the callback of the query? That can't be right.
Event streams are similar to Promises in that they provide a value for something that will happen in the future. In the case of event streams it is a value for an event. In the browser this might be a user clicking a button. This event can occur many times. It might never occur. Promises are values for something that will happen in the future, but only once.
Use Promises for things that happen once, event streams for things that happen 0 to many times.
I'm not sure I'm seeing your argument against using promises over observables for single events. Observables are a superset of promises, sure, but you can map them into situations where you'd use promises and then still have the flexibility of observables and all their operators. along with this you get the consistency of using the same async paradigm everywhere.
I find you run into trouble when you start mixing them. If you use observables for 0-N events and promises for 1 time events you're always converting between the two.
Yes, I too find callbacks an obstacle to readability, among other things.
If one uses mongodb (popular on node.js), and one needs multiple queries to 'join' several documents, the cascade of callbacks is an obstacle to readability, let alone debugging.
If you're writing a cascade of callbacks then just use promises. There's no reason to be needing to cascade callbacks pretty much all libraries now support promises and the rest can be promisified.
I'm not calling it a Panacea but Promises are idiomatic Node Javascript now and there's almost no reason for mutliple levels of nested callbacks anymore.
Callbacks are ugly, but are probably the semantically simplest way to handle asynchronicity. Promises are ugly too, but are semantically the same thing as async/await. I agree that promises and callbacks are not pleasing to the eye, but they are completely logical ways to do things.
And if you're using regular callbacks, forget it: you'd have to nest do_other_things in the callback of get_username(!). It just makes things very awkward.
I understand what you mean about sync vs async calls. I won't pretend I have a better solution. I don't mean to say callbacks are illogical, just a bad way to write programs. So maybe not 'wrong in concept', I can concede that. But I think async/await can make things more readable again, i.e.:
var username = await get_username();
/* carry on... */
I find that JS often seems to tie programmers in the most extraordinary knots just to implement even quite simple logic, because of the single-threaded nature of the language.
In the programming model used by most other mainstream languages today, if you've got some work to do that interacts with some external system and might take a while, you'd probably start another thread for that task. You'd write the required logic in the usual linear fashion, and just let the thread block if and when it needs to. Modelling this using fork/join semantics and techniques to co-ordinate access to shared resources from different threads are reasonably well understood ideas.
Because there is no general support for concurrency and parallelism in JS, you only get one thread, and so in most cases you can't afford to ever block it. Consequently, you get this highly asynchronous style that feels like writing everything manually in continuation passing style, just so you can carry on with something else instead of waiting. That in turn leads to callback hell, where you start to lose cohesion and locality in your code, even though usually you're still just trying to represent a simple, linear sequence of operations.
Async/await help to bring that cohesion and locality back by writing code in a style that is closer to the natural linear behaviour it is modelling. However, even those feel a bit like papering over the cracks in some cases. Async/await kinda sorta give us some simple fork/join semantics, but as the blog post linked from the parent shows, we have a lot of promise-based details remaining underneath.
Fundamentally, the problem seems to be that JS is increasingly being used to deal with concurrent behaviours, but it lacks an execution model and language tools to describe that behaviour in a natural, systematic way as most other widely used languages can. Being strictly single-threaded avoided all the synchronisation problems in the early days, when the most you had to worry about was a couple of different browser events firing close together and it was helpful to know the handler for one would complete before anything else started happening. I'm not sure it's still a plus point now that we're trying to use JS for much more demanding concurrent systems, though.
> Modelling this using fork/join semantics and techniques to co-ordinate access to shared resources from different threads are reasonably well understood ideas.
Writing thread-safe code is anything but easy in languages that support threads.
Dealing with shared state is not always easy when you're working with multiple threads. If you can't reasonably avoid that sharing because of the nature of your problem, and if your choice of language and tools only provide tools on the level of manual locking, then I agree that writing correct, thread-safe code has its challenges.
However, there are plenty of scenarios where you don't need much if any state to be shared between threads. That includes almost every example of JS promises or async/await that I've seen this evening while reading this discussion and the examples people are linking to.
There are also plenty of more sophisticated models for co-ordinating threads that do need to interact, from message passing to software transactional memory. These are hardly obscure ideas today, and I don't think anyone could reasonably argue that for example message passing makes things complicated but async/await/promises make things simple.
I have function requestFromService and another readFromDB. On a single thread, I issue one request to a service and one to the database server, and you think it makes sense for the thread to just wait around? In javascript, the second I issue those two requests, the thread is available to now serve the next incoming request or do literally anything else while those services take as long or as little as they need.
The code itself could not be more clear as to what will happen, except for maybe the syntax Promise.join(), which was only there to illustrate how easy it is to do concurrent, asynchronous tasks.
You don't have to think twice about what to do while those requests carry out, and it's not like you had to write your code in a complex manner to take advantage of the asynchronous nature of JS, it just inherently works that way. Maybe I have gone and served 5 more requests while the db server was responding, and I didn't have to even be aware of the concurrency whatsoever.
I'm not disputing the architectural choices behind Node's async model. I'm talking about syntax. It's cool that you can process multiple promises at the same time, but most of the time I need one promise to complete before continuing. In those cases the way it inverts regular code flow is very unnatural. Like I said, I think more support/usage of 'await' will help.
We couldn't even stand 6 months let alone a year. Broken pack depts make it hard to CI, poor performance, memory leaks, huge docker base images due to deps, lots of single cpu only tasks that were too hard to scale out, an ugly language compared to ruby or Python and on top of all that how poor the package management with NPM has been. We've just ditched it and gone back to Python/Django/flask and ruby for the ops tooling.
> You use Grunt!? Everyone uses Gulp!? Wait no, use native NPM scripts!
Although couched as a criticism this is actually the community fixing itself. The evolution from Grunt > Gulp > npm scripts is movement away from needless complexity towards simplicity. Npm scripts are effectively just Bash commands that build and manage your project, which sometimes employ small, unixy tools written in Node.
This self correction was pretty quick, it happened within a few years.
> Unfortunately, there isn’t any one “standard” (like everything else in Javascript) for implementing or using Promises.
Yes there is. It's called the Promises/A+ spec, and its built into ES6.
So somebody found a project somewhere on the internet with an exceptionally complicated build process, and you use it to say npm scripts are broken? Sorry, that's absurd. Looking at that particular build process, I don't think a Makefile could have been crafted to make it much simpler or smaller. In that example, the problem lies with the complexity of what they're having to do, not the tool.
Npm scripts are really just shell scripting, which means all the real progress happens in the unixy Node tools that do the heavy lifting, where it should be. It's a future proof and scalable approach for the vast majority of projects imo.
It's an inflated example of what all npm script projects become, imo. First you just have "test", then you add "build", then you separate your "test" into one for the browser, one for Node, one for CI, then you need scripts that combine those together; then you create different "start" versions depending on environment... It blows up quickly.
> Npm scripts are really just shell scripting, which means all the real progress happens in the unixy Node tools that do the heavy lifting, where it should be.
That's fine, but you're missing critical features that Make provides; make won't even rebuild a target if no files have changed.
I completely disagree that this is what all npm scripts become
>then you separate your "test" into one for the browser, one for Node, one for CI,
For the love of God why. Don't turn every language into Java.
Plenty of tools will not rebuilt a project if no files have changed, but the dumb, quick and easy way that requires no deendendencies is simply to delete and rebuild.
The fact that this functionality is served so well by a simple dependency shows that you don't need overly complex monolithic tools to accomplish small tasks.
I completely disagree that this is what all npm scripts become
>then you separate your "test" into one for the browser, one for Node, one for CI,
For the love of God why. Don't turn every language into Java.
Plenty of tools will not rebuilt a project if no files have changed, but the dumb, quick and easy way that requires no deendendencies is simply to delete and rebuild.
The fact that this functionality is served so well by a simple dependency shows that you don't need overly complex monolithic tools to accomplish small tasks.
Admittedly, my first 2 or 3 npm scripts based build processes did start to get a bit ugly. But I'm much better at writing them now, so they stay pretty sane.
> That's fine, but you're missing critical features that Make provides; make won't even rebuild a target if no files have changed.
WebPack does this for me too. Also my ava tests don't rerun for files that haven't changed while watching.
Genuinely curious, what scenarios precisely do you find this feature of Make useful?
Also, Make isn't truly cross platform ... and since I sometimes work with Windows devs this would be a problem.
Whether the Promise/A+ spec is bare bones or not doesn't matter, the reference implementation is the ES6 native Promise, which is part of Node and now natively in browsers too. This is concretely what a promise _is_, use a different sort of promise at your peril.
I mused on similar things after getting a very small project shipped. It is based on my experience with Node, Ruby, and Node as of 4 years ago (when I first learned it).
The Netflix.com site and webapp runs on Node (and talks to a number of services written in mostly JVM based languages). While we encounter challenges just as we would with any other language -- it works for us and I would argue that it's a pretty big application.
There's always a multitude of ways to get something done, and it's up to you to decide what tool will do it best. Don't treat any one language as an end-all-be-all and you might find yourself much happier and more productive. Of course, YMMV.
I watched one of the recent Netflix engineering videos where they moved to a customized version of React and I could have sworn in that talk they actually moved away from Observables[1]. What was confusing is that the other talked released at the same time they talk about enhancing RxJS.
One of the core tenets of the Netflix culture is "Freedom & Responsibility". In this context it means that each team is free to make calls like that as they see fit. Promises, Observables, Generators, Callbacks -- it's up to your team to determine what makes the most sense.
That said, we do make heavy use of Rx on the API layer (RxJava/RxGroovy). My team isn't currently using RxJs, but I expect that to change in the future. There are a lot of Observable fans here :)
> By the way, what is Netflix's take on Promises vs RxJS?
We've been doing tons of research into the Netflix methodology and teams are highly compartmentalized (albeit communicate richly). This means that when you ask "What do you do at Netflix?" you should really be asking "What does team X do at Netflix?" or "Does any team at Netflix do X?"
For instance, unless things have changed or I have the story wrong, JVM is used for the streaming because that's the tool which that team chose. The other teams merely communicate with it over an API (regardless of client language).
I think the 'big application' bit is what challenges many people. From what I understand, Netflix is essential a huge group of micro-services, and I 'think' this is where Node.js shines.
What are your thoughts on that? When you're developing a service at Netflix, does it feel like you're building a single large application? Does the mentality of micro-services change how you approach code and therefore the validity of the original posts arguments?
Speaking only for myself, a middle-weight .NET dev who was, not too long ago, working desperately to find my footing in all this:
1. I think the learning curve to doing good work with Node is really shallow at first, and then it quickly gets almost vertical.
2. I think that ALL of the other server-side language/framework combos (Ruby/Rails, Python/Django, C#/.NET, PHP/Symfony, etc.) make it significantly easier for a "merely-good" dev to make really good Web applications.
3. I think the kinds of devs that work at, say, Netflix are often the kinds of devs that can do amazing things with almost any dev stack, but that leads directly to my last point:
4. I think that most junior devs are making a tactical mistake with their growth, and with their careers, when they choose Node for their first forays into Web app development. The relative chaos and churn, and questionable suitability of Node for many-maybe-most use cases, make Node more trouble than it is worth for someone still struggling to make sense of the basics.
> 2. I think that ALL of the other server-side language/framework combos (Ruby/Rails, Python/Django, C#/.NET, PHP/Symfony, etc.) make it significantly easier for a "merely-good" dev to make really good Web applications.
All of those frameworks are threaded/synchronous - they are maybe 10 times easier to use and potentially 100x slower (less concurrent) than Node. The ruby equivalent isn't rails, its eventmachine. Moving from django to twisted is probably just as hard as django to Node. 9/10s of the python ecosystem assume synchronous code so you end up with threads, processes, and callbacks.
If they would switch to more efficient languages, they could probably run on half of their current servers.
But there's the huge cost of rewriting something you already have.
Will that affect your total cost significantly, if you can run on less servers.
I.e. it's too late to switch now.
There needs to be a significant business reason to switch.
"Stuff-that-kinda-works but needs more hardware thrown at it" is probably not a good reason.
Elixir/Phoenix, Go/, Rust/ ... should all outperform node.js easily, plus they're actually parallel and not just concurrent AND are much more solid languages to begin with.
But aren't all Node.js deploys fronted by a reverse proxy (like nginx) to multiple node processes? That gives them actual parallelism in practise, right?
Or is it remotely common to just have a single node process and never use more than one core?
splitting the load through multiple node processes is fine for simple usecases and in general you're fine if you do mostly IO stuff.
But there are usecases that have to crunch numbers or do other CPU intensive algorithms that can be parallelized on the algorithm level. This is where node.js just falls apart. Additionally, if an algorithm requires quite much memory (multiple gigs), node.js performs very poorly.
Most companies that do actual CPU intensive work have services written in other languages to handle this, and "we're using node!" mostly means that they just serve results or do other IO-Tasks with node. But not the actual work.
> All of those frameworks are threaded/synchronous
"Threaded" and "synchronous" are orthogonal concepts that don't deserve being lumped together with a slash. They can have significantly different effects on performance depending on the scenario. C#/.NET isn't explicitly threaded (or single-threaded) or synchronous, and it's right in the middle of your list. It has more in common with Java than with the other three. Keep in mind that there are node packages for threads, too.
Node evangelists often don't seem to get that non-blocking behavior isn't a Node exclusive. You can do asynchronous non-blocking code with C#. You can do an event loop and play node all day long with any number of threads you want. It's not that opinionated about how you run your code. There is a lot of old code that is synchronous. All languages have significant amounts of old code hanging around that drags us down. But things like the TPL, async/await and IOCP can make C# code perform significantly faster than Node. And it's in the middle of a serious public overhaul.
Java and the JVM can do this as well. Java itself might be a bit behind on language features, but it tends to outperform the CLR (also uses more memory). And libraries like Netty and Vert.x are seriously impressive. You can do incredibly amazing things with the JVM, and it's been one of the leaders in performance among GC-ed environments for a long time. You've also got the option to use languages like Scala and Clojure on the JVM.
I'll give you that Node applications might outperform Ruby and Python apps when running on the default environments. But those also can run on separate runtimes which can significantly boost performance. And we're ignoring less common server languages like Erlang which are quite suitable for certain situations.
You should also note that Javascript (and node) performance will be a bit questionable for a while, anyway, as the recent ES6 support has introduced a lot of performance wrinkles (https://kpdecker.github.io/six-speed/) that will take a while to iron out. ES6 adds some really attractive features, so expect node performance to drop (albeit perhaps only temporarily) as your package dependencies start adopting ES6-isms.
Just as there are hundreds of people jumping on the Node train, there are hundreds jumping off. Javascript tends to prototype rather quickly, which has huge drawing power. The constantly shifting ecosystem and the weak tooling, among other things, can affect its staying power. Some people are leaving due to code maintainability, others are leaving because of performance concerns, others are leaving because of ecosystem things like left-pad. There's a lot of variables involved in choosing a language, Performance is only one of them and no languages is a silver bullet.
We can pick on other languages all day, but there's always something to make fun of about our own language/ecosystem. So let's let people pick the tools they want and maybe critically evangelize a lot less. If they pick something ill-suited to their requirements, they'll just have to switch later. It's a pain, but it's not the end of the world.
I haven't found the overhead of threaded frameworks to be anywhere close to that -- in practice it can be close to even, maybe 2x slower.
And a big advantage of threaded is that it is (obviously) easier to scale to multiple CPUs, and I don't have to worry about one connection taking more than a handful of milliseconds and jamming up my server.
Node is fine, but I feel many of the speed benefits come from V8 vs cpython, rather than threaded vs async
Surprisingly (to me at least) scaling single threaded code to multiple CPUs is actually very easy: just spawn the node process N times (using cluster, naught, pm2 or similar project) and that's it.
They are probably 10x faster. You'd be surprised that Node is actually pretty slow by itself and only selling point is async which for the most part is not even used e.g. callbacks inside callbacks.
And if you leverage parallelism of those languages they become 100x faster since they can do multi-threaded processing unlike Node.
How big is the Netflix webapp, though? There's what, 5 or 6 main content "views", then around 20 user-editable forms? Plus the help site. Am I viewing this incorrectly? It doesn't just doesn't feel like Netflix is a large app from the UI perspective - maybe that's just good design.
Hope this doesn't come off as "I could do this in a weekend", just questioning how big the actual webapp is.
I believe you would be surprised were you to look under the hood.
There are a number of different "UI areas" in Netflix.com -- homepage, titles, signup (multiple flavors of this, depending on how you're signing up), onboarding, member browsing/viewing experiences, account management, numerous partner integrations, etc... not to mention all the work that goes into per-user customization, AB testing, and internationalization.
No offense taken on the question. IMO, if you look at the site and view it as simple then we've done our job correctly. :)
If I am not wrong, Netflix folks make heavy use of 'stream based programming' using RxJs. This greatly simplifies a lot of the standard problems faced in JS - like callback hell and error handling, if one adopts it consistently across the stack. Not many folks have discovered this as yet.
For web applications, I've been very happy with the up-and-coming Phoenix[0], a framework for the Elixir language.
Very well-designed and thought-out, fast, productive. Leans functional and immutable rather than object-oriented and mutable. It's kind of like Rails but without most of the problems.
I've found Elixir to be a delightful language with very palatable syntax compared to ruby. It has been a really enjoyable transition, with some vague reminders of the parts I really like about JavaScript.
Most of the problems described in the articles can be solved by simply using promises. Error handling is centralized in your chain and any function can throw and just like Python or Ruby you can catch anywhere you want. As for consistency between callbacks, promises and generators, just pick one. We switched from callbacks to promises and it was a great move. Error handling greatly improved and the code looks a lot nicer. No more callbacks hell. Ever.
We switched our backend from Python to Node almost 2 years ago now and it was a great decision for us. If you handle a code base that deals with a lot of async requests Node is definitely a top contender.
I think by reading the comments here and the article, its apparent that Node.js just isn't as mature yet. If you know what you're doing, it can be great, but for the noobs the right way to do things is not easily apparent. Couple this with the huge choice in libraries and frameworks, makes Node.js harder for now. I think the base is good though and given more time it will become more easy to wield. This is typical of any new tool. And yes Node.js has been around the block for a while, but it is still newish compared to Python, Ruby, PHP, etc. So you're going to pay a new adopter tax still.
Now all of a sudden, having types and some standards to gather around doesn't sound like a bad idea anymore ;)
I agree with one of the commenters: Lessons already learned by older engineers (who went through similar woes with other languages/tools) are being re-learned again and again.
The software industry is in a sorry state.
Unless you are a very disciplined team with a very strong sense of writing modular code, don't use Node.js for any larger project.
And even then, the single-most useful function in an IDE 'Show Call Hierarchy' will never be available when using a dynamically typed language.
That is not an issue for smaller projects. However, long before you even get close to the the million lines of code project size, your tools will fail you.
Your debugging/refactoring times will explode and adding a new feature will seem unsurmountable.
Instead, let's just re-write everything from scratch because the cool hipster that wrote your backend a year ago has left for greener pastures...
I won't even try to guess the amount of technical debt produced with Node.js and the likes each day in the bay area.
And, yes, I just used Node.js to write a Slack-bot. It was fun, took me two hours and got me up and running quickly.
That's the beauty of it. Just be aware of the dangers.
If a particular library doesn't have one, you can write it yourself. In that case theres no need to model the entire library - you can only model the subset of functions / methods that you use.
With TS do I have to have a spec for every third party code or can I import something and use it as-is, but not get the benefits of TS on that part of the code?
Lessons already learned by older engineers (who went through similar woes with other languages/tools) are being re-learned again and again.
If only that were true! I think the JS (and more generally web development) ecosystem today is more a case of those who do not learn from history being doomed to repeat it. The thing is, if you actually are Google or Facebook or Microsoft or Mozilla or Apple, you can throw huge amounts of resources at problems and produce sufficiently useful solutions regardless, and then you can do the same again a few months later if you need to. These are the kinds of organisations that set the tone for the whole community because of their sheer size and influence. However, their goals are not necessarily aligned with the rest of the web development community, nor will their approach necessarily work well for someone else.
I think you're both saying something similar, but there's a "why" in there to connect:
> Younger engineers are ignoring lessons learned
Why ignoring them? Beyond the 'youth' factor directly, the signaling from the larger companies mentioned (google, etc) is that those lessons don't necessarily matter, and... look, google did XYZ, you can too (ignoring that they threw potentially 3-5x as many people at problem XYZ than you even have, much less justify).
Maybe neither of you meant that, but that was the connection I just saw between your two comments.
Yes, that's basically what I meant. I think some of the highly visible projects from tech giants today are succeeding despite their approach to software development, not necessarily because of it, but a lot of the new developers coming into the industry lack the experience to realise that and think of these projects as examples of how things should be done.
Every time I see some young developer posting on HN about how the code we write today only has to work for a year or maybe two at most, or how 10,000 lines of code is a large program, or how some high profile company's web site that is still fundamentally just forms and tables and an API for talking to a database is a complicated UI, I weep a little for the future of our industry and our lack of ambition. Of course things like performance and stability and standardisation and portability don't matter very much if all you ever do is hop from one small throwaway project to the next every few months.
I think the culture around HN is unfortunate in this respect, because from a commercial point of view, it is an alluring idea to build some sort of trivial MVP, try and get crazy amounts of funding, and then if (and only if) you succeed, to throw it all out and start over. As a business strategy for building the next WhatsApp or Instagram, that's rational and perhaps quite effective. But it also contributes to the everything-is-expendable mindset that plagues our industry, and since the overwhelming majority of software development projects don't have the luxury of starting over every five minutes, I think it breeds a lack of respect for professional programming and the skills required to build good software without the benefit of an effectively infinite funding source from benevolent investors (or from the goose that keeps laying golden eggs elsewhere in the organisation).
That might be OK for Facebook or Google, but I wonder how many other startups that fail could still have been very successful, albeit not in unicorn territory, if their developers had paid more attention to some of these issues. I wonder how much time humanity collectively wastes today just because software quality and reliability aren't taken seriously by too much of the industry.
maybe i haven't read this thread closely enough ... the parent to this post mentions "lessons of the past" and its parent mentions typed languages, so uh, what lessons are being discussed here? certainty not "the past teaches that typed languages are best," b/c lisp
I'm mostly lamenting the overall lack of progress in creating reliable, bug-free software.
You would never let a software engineer build a house, would you?
For some reason, the discipline of architecting and building houses is far more advanced than building software.
Why is that?
We keep inventing new programming languages to tackle various aspects of making software better (safe concurrency, preventing memory leaks, etc. etc.),
but we keep introducing 5 bugs per Function Point and the rate of finding and removing bugs has stayed pretty much the same the last 20 years.
There needs to be a fundamental change in how we create software.
For what it's worth, I don't think Lisp is a very convincing example of how useful a language with a limited type system can be, because to a first approximation no-one uses it.
But more generally, the modern web development world is only just waking up to the idea that if you want to build software sensibly beyond a very small scale, you need tools for modularity and composition. Separation of concerns is important to preserve developer sanity and keep maintenance manageable. It is helpful to have well-defined, stable interfaces, whatever the current underlying implementation. Having an expressive language is good, but having an expressive language built on sound foundations and with consistent patterns and clean semantics is much better. Standards for protocols and formats are important. Performance matters, in many different ways. Quality matters. Portability and standards compliance matter. Longevity and stability and backward compatibility matter. Having a few good tools that work well and offer substantial benefits is much more valuable than having a million tiny things that change or become unsupported within weeks. Frameworks can be useful for getting going quickly, but you lose flexibility down the line if your requirements evolve beyond what the framework was designed to support and that can hurt. Libraries offer more flexibility, but you often incur overheads converting to and from their conventions and those overheads need to be reasonable if the library is going to be useful. Reuse is good but not always the best choice.
These and many other lessons in real world software development were brought to you by the words "developers older than", the number 30, and experience of programming outside the JS bubble. :-)
late reply, but something occurred to me to say more as a (hopefully funny) witticism than an actual talking point:
if we approximate usage in terms of a poly sum w/ coeff derivatives wrt time (Taylor series is the word, I think) then one might estimate lisp to be a very much used language
However, having witnessed (different shapes of) the wheel being reinvented over the past 2 decades, my take is slightly different: each language/environment has its strengths. If you have the luxury of having a senior team that's decently versed in several environments, then you can have your cake and eat it too: for each tool, micro-service, component... use the right language for that. Then tie it all together on the network. Since the system as a whole is likely to be a network product anyways, embrace it to your advantage.
Joe Armstrong wrote a good post[1] about connecting programs a few months ago, that's quite relevant.
Node.js is a great environment for many microservices. Got a task to do which is 80% done in some npm module? Make a node.js service around it. Same idea for Perl & CPAN, C libs, Java tools, etc.
I've worked in three million+ loc codebases, in PHP, Python and Java. I don't share your opinion that you need static types in these circumstances. You need discipline, modularity, and most importantly you need to have been blessed with gardeners and maintainers throughout the life of a project and not just after a mess has already taken hold.
Did you read what I wrote?
I already said that you need a disciplined team.
Good luck keeping that team together for years to come.
Not sure if you are disputing the fact that keeping code around is a challenge, or not.
The main problem with most typed languages is basically that types are nominal by default.
What I want is free-form records and structural types by default, nominal types only when necessary. I get more than half of that with TypeScript (it does lack nominal types but there are ways to get around that), and honestly its the half I prefer having.
Most serious statically-typed languages have some sort of "anything" type where you can check the actual value type at runtime using some sort of reflection. For example, the direct Golang equivalent of a Javascript object is a map[string]interface{}.
A lot of the Node issues seems to be due to the fact that it's a very new language. Sure JS has been around for a long time but the problem domain was small and very different from what Node is trying to achieve. Couple that with some issues other languages don't have (JS's standard library is quite thin) and I'm not surprised at these issues. I felt bits of this playing with Clojure and Go when they were new.
I imagine a lot of these issues will settle down in the next few years as one or two sets of conventions become de facto standards.
> And even then, the single-most useful function in an IDE
> 'Show Call Hierarchy' will never be available when
> using a dynamically typed language.
True when "never" = "40 years ago". Smalltalk IDEs have had features for displaying call hierarchies statically and dynamically since essentially forever. And of course, in a dynamic environment the IDE vs. program distinction doesn't matter, so if you want to really know what's going on, just insert a self halt. and check in the debugger.
Smalltalk also introduced automatic refactoring, another canard of the "impossible to do in a dynamic language" variety. Really weird when "impossible in" = "invented there".
Yes, we can talk about how static type information can make some of this easier, but it also makes other parts harder (and the RB guys did talk about this, so the information is out there).
Please: if you want to diss JavaScript, diss JavaScript, don't say "dynamic languages". If you want to diss dynamic languages (please do!), inform yourself first.
I think the switch to an async back-end can be more initial work than many expect. It may take some time to feel as productive, but promises become powerful and became a game changer for me over my previous work with callbacks. Error handling also becomes manageable.
What I really enjoy is jumping into new community and getting to work with tools that have built. Choosing the right ones can make or break an experience. I personally enjoyed working with Express and Bookshelf.js/Knex.
I appreciate the authors perspective, but I also don't think this should deter anyone from trying out Node. I personally have no overwhelming preference to using a Python or Javascript stack.
After ruby orms, Bookshelf felt very, very underdeveloped. Anything I tried to do beyond "hello world" only brought me pain, especially dealing with associations, but honestly just about everything. I guess I've been spoiled, but getting anything done in express/bookshelf combo seemed like a chore.
247 comments
[ 4.0 ms ] story [ 329 ms ] threadMy personal sentiment is to just use Promises, like everywhere. ES7 async/await will really help with this too.
Promises let you throw errors normally. They will propagate up the call stack in a similar manner. With bluebird, you will also get full stack traces in development mode and the performance penalty for that isn't too bad.
> The last thing that I found frustrating was the lack of standards. Everyone seems to have their own idea of how the above points should be handled. Callbacks? Promises? Error handling? Build scripts?
Promises are in ES6 (i don't think it gets more standard than that) and have well defined semantics, including error handling, shared between libraries: https://promisesaplus.com/
I know that Bluebird's promisifyAll might seem like a bit of a hack, but just try it out. It works surprisingly well, and its really painstaikingly tuned for near-zero performance loss. It will probably be both less painful to do and more performant than any manual attempt to wrap a callback based API into a promise one.
http://bluebirdjs.com/docs/warning-explanations.html#warning...
If it doesn't do that for your case then its likely that it needs some fixing - you should probably open an issue and we'll try to reproduce it...
Also, thenlint might work for you: https://www.npmjs.com/package/thenlint
https://github.com/pjungwir/node-errors-talk
At the time the solution was "use domains", but I think domains are deprecated now. It was painful enough that I have stuck with Rails since then. I'm glad to hear that Promises are an improvement!
* V8 needs to optimize try-finally
* Node core needs to add try-finally at every single place where callbacks are invoked and make sure all state and resource cleanup is properly done to support domains
* Popular libraries need to also add try-finally handlers for the above.
As to why this is a problem in node and not so much in other languages, its because with node callbacks, the call stack goes both ways. In other languages, libraries mostly call their dependencies' code. In node's CPS style, you call the library but the library also calls your closure code. The semantics for the 2nd part aren't well defined in node - the loose law basically says: I wont call you twice, I'll try not to call you synchronously, and you wont throw (and if you do the behavior is undefined).
With promises there is a contract and its enforced by the promise implementation. Since Promises actually have error semantics, you can build resource management strategies on top of them. http://promise-nuggets.github.io/articles/21-context-manager... - and consequently there is no reason to crash your server on errors.
Perhaps that is the reason that people have become more conscious of making _drastic_ changes -- rust and go are examples.
My recommendation is to just use songbird (which exposes the forthcoming promise API from core, built on bluebird), async/await and the async `trycatch` library so you don't have to worry about a 3rd party package's choice of asynchrony. It also comes with optional long stack traces.
> Songbird adds promise to Object.prototype
Eek
in my experience, async/await is a great way to layer indirection and obfuscation over what is still callback hell. it may look better in the editor, but it's hell to debug.
EDIT: Oops, I misread the parent. You're right, async/await isn't easy to debug, but it's getting better, and it's a tradeoff. For now, I recommend debugging the compiled code (using generators) without source maps to avoid some of the quirks of source maps. In my experience, the tradeoff is worth it for new node.js developers because it offloads the asynchrony contract (e.g., calling a callback OAOO) to the language.
Here's a good article on promises that helped me, especially since I had seen a lot of misuse in JS land before I started using them:
https://pouchdb.com/2015/05/18/we-have-a-problem-with-promis...
Interested in sharing more model logic between our front- and backends, I also investigated writing some new backend features using Node (we're currently developing with Rails). But after days of research and playing around with the available options, I came to a conclusion similar to Gavin's -- any reasonably complex backend requires you to either roll your own everything, or try to cobble together literally hundreds of tiny dependencies that weren't built to go together, and then somehow keep track of all of their regressions and breaking changes.
Node's fantastic performance and unique ability to share logic between client and server are enticing, but I just don't trust the community and "best practices" around it enough to bet the farm on it for now.
Its much easier to do your own research and find the best module to solve a particular problem you are having then to shoehorn into some larger monolithic framework.
Also its a lot more fundamental then that - Node.js has prolly the fastest iteration cycle for any platform out there since its so easy to create your own module - it leads to some sort of Cambrian explosion of innovation and experimentation.
EDIT:
Also OP seems to think callback hell and async programming is bad. The important thing is those things are problems for python/.... too !
Its just that python doesn't have a good programming model to even begin to address those concerns.
JavaScript at-least tries to say - "hey this is a problem we need to deal with - concurrency is a issues we all face "
So when devs complain about callback hell - its just that they have never tried to use python to do async in a neat way.
This isn't even close to true. Anything JavaScript has to express logic in the face of asynchrony, Python has too. There are a half-dozen asynchronous web servers written in Python.
There's not as much of a culture of writing APIs that way in Python because it's generally a terrible way to program, and threads/OS processes are good enough for basically everything except HTTP servers with absurd numbers of concurrent connections.
I would love to see evidence for you making that statement. Almost every programmer would put out their fav programming language as the 'right' way to program.
> everything except HTTP servers with absurd numbers of concurrent connection
Once you introduce async operations in your code - you need to follow the execution path through. http request can be async - but then what if the http request results you doing a db lookup or some form of file handling ? you need to make the whole thing event driven.
Both nginx and Apache's event MPM handle HTTP connections with events while the app backends are still using preemptive threads for running the HTTP handler code, so it's clearly not the case that "you need to make the whole thing event driven." You just need programmers who don't think, "well since the browser doesn't expose threads to JavaScript programmers, clearly they are useless."
I feel so lucky to have avoided the whole disgusting web stack.
The end result is they get frustrated and go back to their old language.
The trouble is that while you can do this for yourself to some extent, you can't necessarily do it for the people you have to work with. I've worked on a project where one guy came in to do a simple database back-end for an embedded system and decided Node and its ecosystem were appropriate. He seemed to have some experience with those tools, so the rest of the team went along with his decision. A year later, there was still little useful code actually running on the required platforms. One of the other guys got fed up and wrote the basic case using C and SQLite in less than a day. That guy did have some relevant experience and had been questioning the use of Node for that project from early on, but because the Node hype was so high-profile, management hadn't known which developer to believe. This is why it's useful to have articles like the one we're discussing, where people who have actually tried a tool in real life for a significant period of time share their experiences, even if those experiences weren't quite what they'd hoped for at the start.
The difficulty of getting Node itself up and running on the various platforms involved was a significant part of the problem. These weren't Linux boxes built around Xeons. It turns out that as soon as you go outside of the mainstream, the tools and documentation for building and running Node are awful compared to a lot of other languages.
The lack of standardisation and stability and the relatively poor dependency handling within the Node ecosystem were also major factors. Way too much time was spent just trying to make sure different machines really did have identical packages installed, figuring out how to pin everything down to make reproducible builds, and so on. Again, if you're happy just to shrinkwrap, npm install on your local machine, and wait if the repository is unavailable, that's one thing. However, as soon as you start talking about preserving exact configurations within 100% local build environments or building images to install on different machines, all kinds of problems crop up that have been well addressed by now within a lot of more mature language ecosystems but not so far with Node.
Another problem was that Node and its ecosystem are just so heavy overall, and this one is nothing to do with the variety of platforms in use. It's just a big runtime, and a lot of overhead because of the way libraries and dependencies are handled. This would be an issue for anyone operating with limited resources, whether it's trying to run something on a Raspberry Pi or trying to keep costs down by using the smallest possible instances of cloud-hosted virtual servers.
There did also seem to be an element of wanting to use the shiny new toys, and of stubbornly sticking with those toys for far longer than should probably have been allowed, so in that sense there were issues with the developer as well.
But there was also a willingness among the wider team to trust that developer, at least to begin with, because with Node's high profile and large ecosystem, no-one expected it to be as immature as it turned out to be as soon as you left the mainstream. For comparison, porting code written in traditional systems programming languages like C and C++ was done routinely, but even among the dynamic/scripting family, the more established server-side languages have caused far fewer problems than Node.
Can you be more specific?
> It's just a big runtime, and a lot of overhead because of the way libraries and dependencies are handled.
Compared to what exactly? I don't see any mainstream dynamic language except Lua doing better. Ruby? Python?
If you don't need native modules (which is actually the case in a surprising number of situations) there is no build process, except making a tarball of node_modules and the project. If you do need native modules, the situation is slightly worse than C or C++, but not significantly: you generally need to build on a machine with the same architecture (Setting up cross-compilation with C isn't very easy either)
> A lot of overhead because of the way libraries and dependencies are handled.
This has some truth to it, but in my experience its overblown. Being just a little bit careful with your dependencies goes a long way.
The Raspberry Pi 2 is a speed monster for node. Plenty of RAM. Here is a basic 2012 benchmark on the Pi 1: http://guidogarcia.net/blog/2012/09/13/node-js-on-my-raspber...
On the other hand, if you want to run node on something like this: https://www.olimex.com/Products/OLinuXino/iMX233/iMX233-OLin... that will definitely be a problem. Thats about the lowest specced hardware that can (barely) run it, but 64MB RAM just isn't enough, really.
P.S. Based on what you said, I have a hunch that most of your troubles came from this module: https://github.com/mapbox/node-sqlite3 - is that right?
Sorry, it wasn't my part of the project so I can't offer much detail. I know they never got as far as a Node build that passed the full test suite on some platforms. Many of the problems seemed to be around how floating point is handled on different types of CPU.
Compared to what exactly? I don't see any mainstream dynamic language except Lua doing better. Ruby? Python?
The total footprint for Node and supporting packages was taking up a few GB more than the equivalents for some of those other languages you mentioned. If you're working on a system where your non-volatile storage is something like a microSD card or flash chip, "a few GB" can be a huge difference.
If you do need native modules, the situation is slightly worse than C or C++, but not significantly: you generally need to build on a machine with the same architecture (Setting up cross-compilation with C isn't very easy either)
Sorry, our experience has been very different on both counts. Building on a machine with the same architecture isn't necessarily a reasonable option if you're targeting platforms that aren't general purpose computers, and at best probably means jumping through a lot of hoops and figuring out a lot of the details for yourself. In contrast, cross-compiling a language like C is straightforward and reliable, the tools for doing it are mature and well documented, and many developers who have worked in embedded systems are familiar with the techniques.
P.S. Based on what you said, I have a hunch that most of your troubles came from this module: https://github.com/mapbox/node-sqlite3 - is that right?
No. The person doing the Node version was trying to use some of the NoSQL tools, and seemed to be spending much of their time documenting and implementing things you get as standard with a SQL database. The person who wrote the C based alternative just used SQLite as a demonstration of how much easier and more robust everything could be using a different database instead.
One of the striking ironies of the project was that although in principle a dynamic language like Node doesn't need a compilation step, just the JS source, while C code has to be compiled for every target platform, the reality was that getting the C code running was dramatically easier. A well-made C library like SQLite could be installed, cross-compiled and running on all the required platforms within literally a few minutes, while Node and several major packages were so flaky that the equivalent level of real world functionality was never achieved even after several months of work. Again just for comparison, some of those other dynamic languages you mentioned were in between but much closer to the shorter end of the spectrum. The Node ecosystem, in our experience, proved to be an exceptionally bad choice for this sort of work.
I might concede that the node ecosystem does have a disproportionate number of people with this mindset though :)
By the way, cross-compilation is also an option, although I haven't personally tried it:
https://github.com/mapbox/node-sqlite3/issues/249
A plus would be that only native modules have to be compiled, and only once. After that its possible to set up downloading them as binary blobs.
You got me curious about the application, by the way. If you let me know what it does, it might be fun to try and implement a prototype in node (with sane dependencies and easy to deploy on a RPi)
(Edit: That said, the difficulties with size, portability and dependency management in the Node ecosystem that were exposed by this particular project are more general flaws with that Node ecosystem and not specific to this particular developer.)
You got me curious about the application, by the way. If you let me know what it does, it might be fun to try and implement a prototype in node (with sane dependencies and easy to deploy on a RPi)
It really wasn't anything very complicated by database standards, just storing some settings in a central place. Those settings were a bit more complicated and inter-related in this case than you could conveniently store using something as simple as a plain text file, so a lightweight database was the next logical step. As mentioned before, it was the kind of thing where you could implement the basic infrastructure required within a few hours using more suitable tools.
So honestly, I really can't relate to most of what you said here :)
"Choose your own tools" is advice for experts, and literally nobody else. It smacks of the cowboy attitude, and that attitude is wildly unhelpful to almost anybody doing professional work in software. Frameworks and coding standards exist for a reason: not to be unreasonable strictures, but to provide guidance and sanity in a staggeringly complex field of endeavor.
Many new devs and junior devs are flocking to the JavaScript ecosystem, and it is one of the most troubled and chaotic ecosystems in all of software right now. Anyone who is not a complete JS badass is simply not going to find Node to be even a decent choice for learning best practices pertaining to the larger world of application development. And I feel I'm stating that politely.
So: yet another JavaScript Pro tossing out the "choose your tools like a Pro" advice-morsel is part of the problem, as I see it. I don't want to work with people who toss the kids into the deep end and hope a few can learn to swim real quick. I think people deserve a helping hand and a reasonable set of expectations.
The JavaScript ecosystem has become self-parodying. Anyone who is oblivious to that fact is inherently NOT a trustworthy witness. That is not to say that the whole thing is rotten and worthless, but it IS messy as heck, and refusing to acknowledge that is a sign that one is in denial about some pretty blatant facts.
Isn't that the case basically all the time? Which is a perfectly suitable reason to avoid it at all costs except for the bare minimum required for front-end..
No, ES5 to ES2015 is a HUGE leap. ES2015 to ES2016 is a very minor jump as it only adds 2 new things to the spec.
> Which is a perfectly suitable reason to avoid it at all costs except for the bare minimum required for front-end..
And honestly, that is not my job to tell you not to do. If you want to avoid Node.js go ahead. I feel Java is something I should avoid at all costs. Doesn't make Java any worse or better. Same with Node.js.
Others don't have that luxury and just can't afford that.
And if your software needs to be around for more than 5 years, you want to keep your costs low.
How long would it take you to find a bug given a stack trace from a Node API with close to no clue which part part of your app registered that callback that isn't called because of the error?
Chances are, you don't even measure the time spent in code archeology as I like to call it. The time probably is not significant right now as you kinda know your way around your code base...yet.
I've been using the same tools for the last 7 months outside of switching from Angular 1.5 to React because I'm not a fan of Angular 2.0.
> Others don't have that luxury and just can't afford that.
Then don't? What is this sense of needing to change everything because something else comes out, that developers have?
If your current toolset works, why change it? Why rewrite a monolithic app in another fancy language or tool just because someone wrote a blog post boasting about that new tool/language?
Author actually wants to use Python. Used Node.JS regardless, for whatever reason.. It did not work the way Python works. Author is frustrated. Complains that JavaScript is not Python.
It's like saying
"JS is stupid because people use braces and whitespace to denote flow control"
Adopt the paradigm and see how well the technology fits it
Is it the right tool for X or Y? I can't say, I'm not a dev. But it does require a lot of hand-holding and keeping current with the zeitgeist.
Despite the sounds of this, I do like the idea of having an experimental platform with which to gain experience using wildly different approaches to the wildly changing world of web apps. I don't take it for granted that old language concepts will turn out to be the most useful for the web platform.
So, I'm okay building a website for the PTA or ceramics club with this, and I'm very interested in the experiences of others using a wide variety of technologies and approaches, but I'm not sure Node.js would be a sensible foundation to build a business on.
But Node has been around since 2009. 2009 JavaScript was missing a lot of features. It was basically a runtime only advanced users should use. The Node maintainers had to make a lot of decisions that now conflict (to some degree) with fixes that have come later to the language. They chose their callback style; now we have Promises. They chose to throw in methods rather than return an error in the callback (this makes it awkward to use fs with Promises without a wrapper library). They chose to implement their version of CommonJS, now we have the .mjs issue arise.
The whole async situation needs to settle down, it's completely unacceptable to write code with callbacks, promises, etc. This is because they are not just challenging to deal with, but intrinsically wrong in concept. I have to wait for a database query to complete, then pass the next thing to do to the callback of the query? That can't be right.
Interesting article I found: "Async/Await: The Hero JavaScript Deserved" https://www.twilio.com/blog/2015/10/asyncawait-the-hero-java...
Use Promises for things that happen once, event streams for things that happen 0 to many times.
I find you run into trouble when you start mixing them. If you use observables for 0-N events and promises for 1 time events you're always converting between the two.
If one uses mongodb (popular on node.js), and one needs multiple queries to 'join' several documents, the cascade of callbacks is an obstacle to readability, let alone debugging.
I'm not calling it a Panacea but Promises are idiomatic Node Javascript now and there's almost no reason for mutliple levels of nested callbacks anymore.
I understand what you mean about sync vs async calls. I won't pretend I have a better solution. I don't mean to say callbacks are illogical, just a bad way to write programs. So maybe not 'wrong in concept', I can concede that. But I think async/await can make things more readable again, i.e.:
In the programming model used by most other mainstream languages today, if you've got some work to do that interacts with some external system and might take a while, you'd probably start another thread for that task. You'd write the required logic in the usual linear fashion, and just let the thread block if and when it needs to. Modelling this using fork/join semantics and techniques to co-ordinate access to shared resources from different threads are reasonably well understood ideas.
Because there is no general support for concurrency and parallelism in JS, you only get one thread, and so in most cases you can't afford to ever block it. Consequently, you get this highly asynchronous style that feels like writing everything manually in continuation passing style, just so you can carry on with something else instead of waiting. That in turn leads to callback hell, where you start to lose cohesion and locality in your code, even though usually you're still just trying to represent a simple, linear sequence of operations.
Async/await help to bring that cohesion and locality back by writing code in a style that is closer to the natural linear behaviour it is modelling. However, even those feel a bit like papering over the cracks in some cases. Async/await kinda sorta give us some simple fork/join semantics, but as the blog post linked from the parent shows, we have a lot of promise-based details remaining underneath.
Fundamentally, the problem seems to be that JS is increasingly being used to deal with concurrent behaviours, but it lacks an execution model and language tools to describe that behaviour in a natural, systematic way as most other widely used languages can. Being strictly single-threaded avoided all the synchronisation problems in the early days, when the most you had to worry about was a couple of different browser events firing close together and it was helpful to know the handler for one would complete before anything else started happening. I'm not sure it's still a plus point now that we're trying to use JS for much more demanding concurrent systems, though.
Writing thread-safe code is anything but easy in languages that support threads.
Dealing with shared state is not always easy when you're working with multiple threads. If you can't reasonably avoid that sharing because of the nature of your problem, and if your choice of language and tools only provide tools on the level of manual locking, then I agree that writing correct, thread-safe code has its challenges.
However, there are plenty of scenarios where you don't need much if any state to be shared between threads. That includes almost every example of JS promises or async/await that I've seen this evening while reading this discussion and the examples people are linking to.
There are also plenty of more sophisticated models for co-ordinating threads that do need to interact, from message passing to software transactional memory. These are hardly obscure ideas today, and I don't think anyone could reasonably argue that for example message passing makes things complicated but async/await/promises make things simple.
Look at the following code: https://paste.ee/r/LJnhg
I have function requestFromService and another readFromDB. On a single thread, I issue one request to a service and one to the database server, and you think it makes sense for the thread to just wait around? In javascript, the second I issue those two requests, the thread is available to now serve the next incoming request or do literally anything else while those services take as long or as little as they need.
The code itself could not be more clear as to what will happen, except for maybe the syntax Promise.join(), which was only there to illustrate how easy it is to do concurrent, asynchronous tasks.
You don't have to think twice about what to do while those requests carry out, and it's not like you had to write your code in a complex manner to take advantage of the asynchronous nature of JS, it just inherently works that way. Maybe I have gone and served 5 more requests while the db server was responding, and I didn't have to even be aware of the concurrency whatsoever.
Although couched as a criticism this is actually the community fixing itself. The evolution from Grunt > Gulp > npm scripts is movement away from needless complexity towards simplicity. Npm scripts are effectively just Bash commands that build and manage your project, which sometimes employ small, unixy tools written in Node.
This self correction was pretty quick, it happened within a few years.
> Unfortunately, there isn’t any one “standard” (like everything else in Javascript) for implementing or using Promises.
Yes there is. It's called the Promises/A+ spec, and its built into ES6.
https://twitter.com/sindresorhus/status/724259780676575232?l...
https://github.com/ReactiveX/rxjs/blob/a3ec89605a24a6f54e577...
Already people are coming up with new "solutions" to this problem that looks more like Grunt. It's a repetitive circle. Personally I just use Make.
Npm scripts are really just shell scripting, which means all the real progress happens in the unixy Node tools that do the heavy lifting, where it should be. It's a future proof and scalable approach for the vast majority of projects imo.
> Npm scripts are really just shell scripting, which means all the real progress happens in the unixy Node tools that do the heavy lifting, where it should be.
That's fine, but you're missing critical features that Make provides; make won't even rebuild a target if no files have changed.
>then you separate your "test" into one for the browser, one for Node, one for CI,
For the love of God why. Don't turn every language into Java
>then you separate your "test" into one for the browser, one for Node, one for CI,
For the love of God why. Don't turn every language into Java
>then you separate your "test" into one for the browser, one for Node, one for CI,
For the love of God why. Don't turn every language into Java.
Plenty of tools will not rebuilt a project if no files have changed, but the dumb, quick and easy way that requires no deendendencies is simply to delete and rebuild.
The fact that this functionality is served so well by a simple dependency shows that you don't need overly complex monolithic tools to accomplish small tasks.
>then you separate your "test" into one for the browser, one for Node, one for CI,
For the love of God why. Don't turn every language into Java.
Plenty of tools will not rebuilt a project if no files have changed, but the dumb, quick and easy way that requires no deendendencies is simply to delete and rebuild.
The fact that this functionality is served so well by a simple dependency shows that you don't need overly complex monolithic tools to accomplish small tasks.
> That's fine, but you're missing critical features that Make provides; make won't even rebuild a target if no files have changed.
WebPack does this for me too. Also my ava tests don't rerun for files that haven't changed while watching.
Genuinely curious, what scenarios precisely do you find this feature of Make useful?
Also, Make isn't truly cross platform ... and since I sometimes work with Windows devs this would be a problem.
And the Promise/A+ spec is so bare-bones, it is laughable. And the actual issue is that not every module is embracing it yet.
Sure, it will all be fixed. For freeeee :)
https://medium.com/@yoooodaaaa/reflections-on-node-698abecce...
There's always a multitude of ways to get something done, and it's up to you to decide what tool will do it best. Don't treat any one language as an end-all-be-all and you might find yourself much happier and more productive. Of course, YMMV.
Well said.
By the way, what is Netflix's take on Promises vs RxJS?
[1]: https://youtu.be/5sETJs2_jwo?t=5m32s
That said, we do make heavy use of Rx on the API layer (RxJava/RxGroovy). My team isn't currently using RxJs, but I expect that to change in the future. There are a lot of Observable fans here :)
We've been doing tons of research into the Netflix methodology and teams are highly compartmentalized (albeit communicate richly). This means that when you ask "What do you do at Netflix?" you should really be asking "What does team X do at Netflix?" or "Does any team at Netflix do X?"
For instance, unless things have changed or I have the story wrong, JVM is used for the streaming because that's the tool which that team chose. The other teams merely communicate with it over an API (regardless of client language).
What are your thoughts on that? When you're developing a service at Netflix, does it feel like you're building a single large application? Does the mentality of micro-services change how you approach code and therefore the validity of the original posts arguments?
1. I think the learning curve to doing good work with Node is really shallow at first, and then it quickly gets almost vertical.
2. I think that ALL of the other server-side language/framework combos (Ruby/Rails, Python/Django, C#/.NET, PHP/Symfony, etc.) make it significantly easier for a "merely-good" dev to make really good Web applications.
3. I think the kinds of devs that work at, say, Netflix are often the kinds of devs that can do amazing things with almost any dev stack, but that leads directly to my last point:
4. I think that most junior devs are making a tactical mistake with their growth, and with their careers, when they choose Node for their first forays into Web app development. The relative chaos and churn, and questionable suitability of Node for many-maybe-most use cases, make Node more trouble than it is worth for someone still struggling to make sense of the basics.
All of those frameworks are threaded/synchronous - they are maybe 10 times easier to use and potentially 100x slower (less concurrent) than Node. The ruby equivalent isn't rails, its eventmachine. Moving from django to twisted is probably just as hard as django to Node. 9/10s of the python ecosystem assume synchronous code so you end up with threads, processes, and callbacks.
If they would switch to more efficient languages, they could probably run on half of their current servers.
But there's the huge cost of rewriting something you already have. Will that affect your total cost significantly, if you can run on less servers.
I.e. it's too late to switch now. There needs to be a significant business reason to switch. "Stuff-that-kinda-works but needs more hardware thrown at it" is probably not a good reason.
Or is it remotely common to just have a single node process and never use more than one core?
But there are usecases that have to crunch numbers or do other CPU intensive algorithms that can be parallelized on the algorithm level. This is where node.js just falls apart. Additionally, if an algorithm requires quite much memory (multiple gigs), node.js performs very poorly.
Most companies that do actual CPU intensive work have services written in other languages to handle this, and "we're using node!" mostly means that they just serve results or do other IO-Tasks with node. But not the actual work.
You're getting downvoted because some of those frameworks had generators (some specifically built for concurrency) before the concept of ES6.
"Threaded" and "synchronous" are orthogonal concepts that don't deserve being lumped together with a slash. They can have significantly different effects on performance depending on the scenario. C#/.NET isn't explicitly threaded (or single-threaded) or synchronous, and it's right in the middle of your list. It has more in common with Java than with the other three. Keep in mind that there are node packages for threads, too.
Node evangelists often don't seem to get that non-blocking behavior isn't a Node exclusive. You can do asynchronous non-blocking code with C#. You can do an event loop and play node all day long with any number of threads you want. It's not that opinionated about how you run your code. There is a lot of old code that is synchronous. All languages have significant amounts of old code hanging around that drags us down. But things like the TPL, async/await and IOCP can make C# code perform significantly faster than Node. And it's in the middle of a serious public overhaul.
Java and the JVM can do this as well. Java itself might be a bit behind on language features, but it tends to outperform the CLR (also uses more memory). And libraries like Netty and Vert.x are seriously impressive. You can do incredibly amazing things with the JVM, and it's been one of the leaders in performance among GC-ed environments for a long time. You've also got the option to use languages like Scala and Clojure on the JVM.
I'll give you that Node applications might outperform Ruby and Python apps when running on the default environments. But those also can run on separate runtimes which can significantly boost performance. And we're ignoring less common server languages like Erlang which are quite suitable for certain situations.
You should also note that Javascript (and node) performance will be a bit questionable for a while, anyway, as the recent ES6 support has introduced a lot of performance wrinkles (https://kpdecker.github.io/six-speed/) that will take a while to iron out. ES6 adds some really attractive features, so expect node performance to drop (albeit perhaps only temporarily) as your package dependencies start adopting ES6-isms.
Just as there are hundreds of people jumping on the Node train, there are hundreds jumping off. Javascript tends to prototype rather quickly, which has huge drawing power. The constantly shifting ecosystem and the weak tooling, among other things, can affect its staying power. Some people are leaving due to code maintainability, others are leaving because of performance concerns, others are leaving because of ecosystem things like left-pad. There's a lot of variables involved in choosing a language, Performance is only one of them and no languages is a silver bullet.
We can pick on other languages all day, but there's always something to make fun of about our own language/ecosystem. So let's let people pick the tools they want and maybe critically evangelize a lot less. If they pick something ill-suited to their requirements, they'll just have to switch later. It's a pain, but it's not the end of the world.
And a big advantage of threaded is that it is (obviously) easier to scale to multiple CPUs, and I don't have to worry about one connection taking more than a handful of milliseconds and jamming up my server.
Node is fine, but I feel many of the speed benefits come from V8 vs cpython, rather than threaded vs async
And if you leverage parallelism of those languages they become 100x faster since they can do multi-threaded processing unlike Node.
Hope this doesn't come off as "I could do this in a weekend", just questioning how big the actual webapp is.
There are a number of different "UI areas" in Netflix.com -- homepage, titles, signup (multiple flavors of this, depending on how you're signing up), onboarding, member browsing/viewing experiences, account management, numerous partner integrations, etc... not to mention all the work that goes into per-user customization, AB testing, and internationalization.
No offense taken on the question. IMO, if you look at the site and view it as simple then we've done our job correctly. :)
Very well-designed and thought-out, fast, productive. Leans functional and immutable rather than object-oriented and mutable. It's kind of like Rails but without most of the problems.
[0] http://www.phoenixframework.org/
We switched our backend from Python to Node almost 2 years ago now and it was a great decision for us. If you handle a code base that deals with a lot of async requests Node is definitely a top contender.
Callback hell and error handling are no longer issues in node if you just embrace generators or async/await.
We use Koa in production and serve billions of requests just fine.
If I had to go back to Express I'd say no.
I agree with one of the commenters: Lessons already learned by older engineers (who went through similar woes with other languages/tools) are being re-learned again and again.
The software industry is in a sorry state.
Unless you are a very disciplined team with a very strong sense of writing modular code, don't use Node.js for any larger project. And even then, the single-most useful function in an IDE 'Show Call Hierarchy' will never be available when using a dynamically typed language.
That is not an issue for smaller projects. However, long before you even get close to the the million lines of code project size, your tools will fail you. Your debugging/refactoring times will explode and adding a new feature will seem unsurmountable.
Instead, let's just re-write everything from scratch because the cool hipster that wrote your backend a year ago has left for greener pastures...
I won't even try to guess the amount of technical debt produced with Node.js and the likes each day in the bay area.
And, yes, I just used Node.js to write a Slack-bot. It was fun, took me two hours and got me up and running quickly. That's the beauty of it. Just be aware of the dangers.
If a particular library doesn't have one, you can write it yourself. In that case theres no need to model the entire library - you can only model the subset of functions / methods that you use.
If only that were true! I think the JS (and more generally web development) ecosystem today is more a case of those who do not learn from history being doomed to repeat it. The thing is, if you actually are Google or Facebook or Microsoft or Mozilla or Apple, you can throw huge amounts of resources at problems and produce sufficiently useful solutions regardless, and then you can do the same again a few months later if you need to. These are the kinds of organisations that set the tone for the whole community because of their sheer size and influence. However, their goals are not necessarily aligned with the rest of the web development community, nor will their approach necessarily work well for someone else.
> Younger engineers are ignoring lessons learned
Why ignoring them? Beyond the 'youth' factor directly, the signaling from the larger companies mentioned (google, etc) is that those lessons don't necessarily matter, and... look, google did XYZ, you can too (ignoring that they threw potentially 3-5x as many people at problem XYZ than you even have, much less justify).
Maybe neither of you meant that, but that was the connection I just saw between your two comments.
Every time I see some young developer posting on HN about how the code we write today only has to work for a year or maybe two at most, or how 10,000 lines of code is a large program, or how some high profile company's web site that is still fundamentally just forms and tables and an API for talking to a database is a complicated UI, I weep a little for the future of our industry and our lack of ambition. Of course things like performance and stability and standardisation and portability don't matter very much if all you ever do is hop from one small throwaway project to the next every few months.
I think the culture around HN is unfortunate in this respect, because from a commercial point of view, it is an alluring idea to build some sort of trivial MVP, try and get crazy amounts of funding, and then if (and only if) you succeed, to throw it all out and start over. As a business strategy for building the next WhatsApp or Instagram, that's rational and perhaps quite effective. But it also contributes to the everything-is-expendable mindset that plagues our industry, and since the overwhelming majority of software development projects don't have the luxury of starting over every five minutes, I think it breeds a lack of respect for professional programming and the skills required to build good software without the benefit of an effectively infinite funding source from benevolent investors (or from the goose that keeps laying golden eggs elsewhere in the organisation).
That might be OK for Facebook or Google, but I wonder how many other startups that fail could still have been very successful, albeit not in unicorn territory, if their developers had paid more attention to some of these issues. I wonder how much time humanity collectively wastes today just because software quality and reliability aren't taken seriously by too much of the industry.
You would never let a software engineer build a house, would you? For some reason, the discipline of architecting and building houses is far more advanced than building software.
Why is that? We keep inventing new programming languages to tackle various aspects of making software better (safe concurrency, preventing memory leaks, etc. etc.), but we keep introducing 5 bugs per Function Point and the rate of finding and removing bugs has stayed pretty much the same the last 20 years.
There needs to be a fundamental change in how we create software.
But more generally, the modern web development world is only just waking up to the idea that if you want to build software sensibly beyond a very small scale, you need tools for modularity and composition. Separation of concerns is important to preserve developer sanity and keep maintenance manageable. It is helpful to have well-defined, stable interfaces, whatever the current underlying implementation. Having an expressive language is good, but having an expressive language built on sound foundations and with consistent patterns and clean semantics is much better. Standards for protocols and formats are important. Performance matters, in many different ways. Quality matters. Portability and standards compliance matter. Longevity and stability and backward compatibility matter. Having a few good tools that work well and offer substantial benefits is much more valuable than having a million tiny things that change or become unsupported within weeks. Frameworks can be useful for getting going quickly, but you lose flexibility down the line if your requirements evolve beyond what the framework was designed to support and that can hurt. Libraries offer more flexibility, but you often incur overheads converting to and from their conventions and those overheads need to be reasonable if the library is going to be useful. Reuse is good but not always the best choice.
These and many other lessons in real world software development were brought to you by the words "developers older than", the number 30, and experience of programming outside the JS bubble. :-)
late reply, but something occurred to me to say more as a (hopefully funny) witticism than an actual talking point:
if we approximate usage in terms of a poly sum w/ coeff derivatives wrt time (Taylor series is the word, I think) then one might estimate lisp to be a very much used language
However, having witnessed (different shapes of) the wheel being reinvented over the past 2 decades, my take is slightly different: each language/environment has its strengths. If you have the luxury of having a senior team that's decently versed in several environments, then you can have your cake and eat it too: for each tool, micro-service, component... use the right language for that. Then tie it all together on the network. Since the system as a whole is likely to be a network product anyways, embrace it to your advantage.
Joe Armstrong wrote a good post[1] about connecting programs a few months ago, that's quite relevant.
Node.js is a great environment for many microservices. Got a task to do which is 80% done in some npm module? Make a node.js service around it. Same idea for Perl & CPAN, C libs, Java tools, etc.
[1] http://joearms.github.io/2016/01/28/A-Badass-Way-To-Connect-...
What I want is free-form records and structural types by default, nominal types only when necessary. I get more than half of that with TypeScript (it does lack nominal types but there are ways to get around that), and honestly its the half I prefer having.
And I hate to say it, but Go doesn't qualify as a serious statically typed language. No language without generics does.
I imagine a lot of these issues will settle down in the next few years as one or two sets of conventions become de facto standards.
> 'Show Call Hierarchy' will never be available when
> using a dynamically typed language.
True when "never" = "40 years ago". Smalltalk IDEs have had features for displaying call hierarchies statically and dynamically since essentially forever. And of course, in a dynamic environment the IDE vs. program distinction doesn't matter, so if you want to really know what's going on, just insert a self halt. and check in the debugger.
Smalltalk also introduced automatic refactoring, another canard of the "impossible to do in a dynamic language" variety. Really weird when "impossible in" = "invented there".
Yes, we can talk about how static type information can make some of this easier, but it also makes other parts harder (and the RB guys did talk about this, so the information is out there).
Please: if you want to diss JavaScript, diss JavaScript, don't say "dynamic languages". If you want to diss dynamic languages (please do!), inform yourself first.
I've started with Allegro Common Lisp, so i kinda know a thing or two about that.
The success of Smalltalk then speaks for itself?
What I really enjoy is jumping into new community and getting to work with tools that have built. Choosing the right ones can make or break an experience. I personally enjoyed working with Express and Bookshelf.js/Knex.
I appreciate the authors perspective, but I also don't think this should deter anyone from trying out Node. I personally have no overwhelming preference to using a Python or Javascript stack.
After ruby orms, Bookshelf felt very, very underdeveloped. Anything I tried to do beyond "hello world" only brought me pain, especially dealing with associations, but honestly just about everything. I guess I've been spoiled, but getting anything done in express/bookshelf combo seemed like a chore.