Fixing the callback spaghetti in node.js (github.com)
node.js is a fantastic platform, but it suffers from your code quickly turning into callback spaghetti and a mess of functions.
I modified the V8 parser to support the await keyword that allows C# to handle this quite elegantly.
108 comments
[ 3.4 ms ] story [ 218 ms ] threadUsing await - is the program flow suspended until the corresponding function returns , or does await keyword act more like a 'pause and continue' mechanism.
I've been using LuaJIT embedded in Nginx (LuaNginxModule). Lua supports coroutines, so a function can just yield. Here's a brief example:
With code like the above I can easily handle into the thousands of concurrent connections per second on the lowest end Linode VPS node available, with barely any load on the box -- and I'm told it should be able to handle 40k+ connections per second, if I were to do any tuning. Oh, and I have only 512Mb of RAM, which it doesn't even get close to under load. And the longest request took less than 500ms at high load.I've been using OpenResty [1] which has the Lua module and a bunch of others all configured together. Works great, and I can't complain about the performance.
Someday I'm sure I'll hand the maintenance of this off, and then I might regret not using one of the "popular" frameworks. But the code is SO straightforward using this stack -- and what I'm using it for is so simple -- I think not.
[1] http://openresty.org/
I had several failed private projects doing the same on C, Lua, and Haskell. Haskell is pretty ideal but I’m not smart enough to hack the GHC. Lua is less ideal but a lovely language - I did not like that it already had a lot of libraries written with blocking code. No matter what I did, someone was going to load an existing blocking Lua library in and ruin it. C has similar problems as Lua and is not as widely accessible. There is room for a Node.js-like libc at some point – I would like to work on that some day.
V8 came out around the same time I was working on these things and I had a rather sudden epiphany that JavaScript was actually the perfect language for what I wanted: single threaded, without preconceived notions of “server-side” I/O, without existing libraries.
http://bostinnovation.com/2011/01/31/node-js-interview-4-que...
I don't think node.js is good enough because you have to deal with the issue being discussed here. In Haskell, as you said, you just write normal code with no worries about mutable state messing with your thread.
https://developer.mozilla.org/en/JavaScript/Guide/Iterators_...
It's the same situation when using callbacks.
I don't have a lot of experience in this area. I'm just reporting back what I remember hearing.
That is true for a single threaded reactor, but in practice I haven't found it to be an especially useful property because naturally that doesn't include external state (i.e. the database).
Getting atomicity right in evented-code has in fact often been a hairier issue for me than doing the same with co-routines or threads, because when you're not allowed to block ever then you quickly find yourself in a situation where you need a retry-mechanism.
Such codebases then tend to quickly converge towards the actor-pattern (tied together by queues) which, ironically, could be had much easier by starting out with co-routines in first place.
So except for things you should EXPECT to change (like the state of a database you're querying) between calls, the "world" you (as a programmer) should care about stays perfectly consistent.
Unless I'm not understanding what you're asking -- is there some situation that I haven't encountered where the state of something that isn't a local variable matters?
Of course people like to point at the overhead of native threads and assume coroutines have similar overhead, which is total bunk. Ironically, event sources in node use a stack-like tracking element which brings back a similar sort of overhead you see in coroutines in Lua, for example.
I doubt we'll see node take another shot at coroutines but that's okay. Node will do fine without coroutines but it will come at the cost of making certain types of code a little less natural (same as eventide code in threads becomes unnatural).
Wondering, is the OpenResty solution otherwise single-threaded-with-an-eventloop much like node?
From the nginx wiki:
> Unlike Apache's mod_lua and Lighttpd's mod_magnet, Lua code written atop this module can be 100% non-blocking on network traffic as long as you use the ngx.location.capture or ngx.location.capture_multi interfaces to let the nginx core do all your requests to mysql, postgresql, memcached, redis, upstream http web services, and etc etc etc (see HttpDrizzleModule, ngx_postgres, HttpMemcModule, HttpRedis2Module and HttpProxyModule modules for details).
Is that talk-via-nginx-commands thing cumbersome in practice?
Slides and performance comparison here: http://slideshare.net/olegp/server-side-javascript-going-all...
I'm about to do an updated version of this talk at #rejectjs in Berlin, so will add audio and update slides in a couple of hours.
We'd like to think that the C# guys were looking our way when they came up with async/await, but there's no proof. :3
http://tomasp.net/blog/csharp-fsharp-async-intro.aspx
http://news.ycombinator.com/item?id=2999260
http://bvanderveen.com/a/owin-buffering-async/
http://devtalk.net/csharp/async-await-and-c-vnext/
http://msmvps.com/blogs/jon_skeet/archive/2010/10/29/initial...
(read comment)
Anyway, it shouldn't be too difficult to build the compiler yourself and set up a basic emacs/vim + mono environment for linux.
For example, F#'s let! binding works with any monad not just async.
If you're going to innovate, then design a language that compiles down to JS that provides the innovation. CoffeeScript.
If you want to take that a step further, look at ClojureScript. Want delimited continuations? Fine. All w/o requiring you to fork Node.js or CoffeeScript.
This is not going to catch most errors occurring in an asynchronous APIs:
Most errors will occur asynchronously, thus the convention of "err" being the first argument to asynchronous API callbacks in Node.Unless this supports that convention, your code should actually look like:
...or something similar. It's better than the alternative, but not great.This certainly could support the "err" first convention, but APIs that don't use that convention wouldn't work correctly.
Here's a concrete example. Normally in Node you do something like this:
The await version would look like this: Ideally it would look like this:http://github.com/crabdude/trycatch
Using only that, I rarely go over 80 character column limit that I impose on myself. There is absolutely zero callback spaghetti whether it's 2 or 25 functions deep in the chain.
Tbh, callback spaghetti only happens to newer async programmers in the same way that a newer programmer will write arrow code with if/else statements.
It's simply not a problem that needs to be addressed other than educating people who are new to node.js with some example tutorials that use an async helper library.
Now, let's say c() changes and needs to call someAsyncFunction() and provide a callback. Which means c itself needs to take a callback. Which means b needs to provide a callback, so b needs to also take a callback, so a needs to provide a callback. And so forth.
Callbacks are infectious - once anybody in the call stack needs one, everybody needs one even if they just pass it on down the stack. Unless you don't need to do anything with return values, but that's fairly rare.
It looks pretty decent in CoffeeScript:
At least that's my experience with structuring asynchronous event-driven programs (without coroutines).
Without it small build/utility scripts turn into macaroni cheese unless you resort back to Python/Ruby/Bash - more languages, harder to maintain.
(disclaimer: I am the guy who worked on the now-defunct `defer` support in Coffeescript, and am now working with the onilabs folk on the stratifiedJS runtime)
Remember that not everyone is as awesomely brilliant as you are.
Writing a user interface that can respond to user input, whilst also being able to handle and respond to a long running data access or computational requests is a concurrent problem.
Having a single threaded model with callbacks, like the JavaScript browser model is one of the less complex ways to handle this.
I agree that a less complex model is appropriate for some developers/applications. But to call yourself a "skilled-professional GUI-developer", you need to get awesomely brilliant enough to handle this.
this.io.sockets.on('connection', this.onConnection.bind(this));
"Programs can be automatically transformed from direct style to CPS." [1]
Do the math.
[1] http://en.wikipedia.org/wiki/Continuation-passing_style
http://github.com/crabdude/trycatch