HN Explorer

Wait.for: Sequential programming for Node.js (beta) (github.com)

52 points by lucio ↗ HN
Simple, straightforward abstraction over Fibers.

By using wait.for, you can call any nodejs standard async function in sequential/Sync mode, waiting for result data, without blocking node's event loop (thanks to fibers)

I'm looking for real-world node apps, to test wait.for

36 comments

[ 5.5 ms ] story [ 90.6 ms ] thread
casual_slacker
Cool idea. You should mention upfront that you're using Fibers under the hood, since it has important performance implications. Also, your Wait.for function only takes functions right now; but if you check for a .then function you can accept promises too.
lucio
Thanks. I've put a link to node-fibers, 2nd line of readme. ¿are Fiber's performance problems solvable? ¿what if fibers were integrated in node core?
AdrianRossouw
fibers will never be integrated into node core.

for what it's worth, I also refuse to use or depend on any code that makes use of node-fibers. I know i'm far from the only person who feels that way too.

EGreg
I don't get it,

    console.log(wait.for(fs.readfile,'/etc/passwd'));
won't that execute console.log immediately, whereas fs.readfile will return the data later? "I don't think it means what you think it means" http://www.youtube.com/watch?v=G2y8Sx4B2Sk
lucio
wait.for, waits for fs.readfile to callback, and returns "data" from callback(err,data)

wait.for(fs.readfile,'/etc/passwd') == fs.readfileSync('/etc/passwd')

wait.for allows you to call any async function as if it were a sync function, without blocking node's event loop. It works inside a fiber. Check the examples.

unnuun
Right, but how the heck is this achieved? Could you show a line or two of example pseudo code that demonstrates the core of this functionality?
tg3
It uses Fibers (https://github.com/laverdet/node-fibers). I don't know much about it, but you have to wrap all your code in a call to `Fiber()`, which is written in C++.
lucio
Fiber() is under the hood, and I hope ES6 generators -when stable- will provide similar tools, so wait.for could be based on ES6 generators.

You don't have to wrap all your code in a call to `Fiber()`

full classic node-server sample:

    var http=require('http'), wait=require('wait.for');

    http.createServer(function(req, res) {
      wait.launchFiber(handleRequest,req, res); //run in a Fiber, keep node spinning
    }).listen(8080);


   //in a fiber
   function handleRequest(req, res) {
     try {
       res.writeHead(200, {'Content-Type': 'text/html'});
       res.end(
       "<h1>Demo page</h1>" 
       + "<p>"+ wait.for(fs.readFile,'/etc/somefile') +"</p>"
       + "<p>"+ wait.for(fs.readFile,'/etc/otherfile') +"</p>");
     }
     catch(err) {
      res.end('error '+e.message); 
     }
   }
tg3
in your example below, you do have to wrap it in a call to `Fiber()`, although you've disguised it as `launchFiber`, which you pass `handleRequest` to (I would call that wrapping `handleRequest`.

That is probably a worthwhile tradeoff, but the first time I read through it, that mechanic wasn't obvious to me.

lucio
It's not 'disguised', it's abstracted. launchFiber does what it's name implies: launchs a fiber. How it is achieved 'internally', it's irrelevant as long the abstraction is not leaky.

Today launchFiber is calling the fn inside a Fiber(), tomorrow could be launching the fiber by means of a generator.

The idea was to create a very simple abstraction to avoid callback hell, adding low or no complexity, and keeping the flexibility to use all the existing async fns with a callback or just "wait" for the results.

lucio
-pseudo code-

  function wait.for ( asyncFn, params ) {

    //call the async fn, returns immediately
    asyncFn(params, function(err,data){
               //here the data is ready
               thisFiber.err = err;
               thisFiber.data = data;
               thisFiber.resume(); //resume execution (after yield())
    });

    //we're in a fiber, so we can yield() to node's event loop 
    thisFiber.yield(); //pause here, give back control to node's event loop
    
    // the next line will be executed when thisFiber.resume() is called
    if (thisFiber.err) throw thisFiber.err; // throw into the caller of wait.for
    return thisFiber.data; //ok
  }
anko
fibers are lightweight threads, employing a sort of non-preemptive multitasking paradigm. it basically spawns a 'thread' that will be resumed only when the callback has returned, which returns the data passed to the callback.

the code is here: https://github.com/luciotato/waitfor/blob/master/waitfor.js, but it does require fibers.

cypher543
The code is open source and only 72 lines long (with comments and whitespace).
qznc
A function executes once all its arguments are evaluated. console.log executes once its single argument wait.for(...) is evaluated. Hence: No, it will not execute immediately.
chrisohara
Why not use harmony generators?

Generators are provided by V8, don't require an external fibers extension (i.e. hack), and will be available in the next node stable via `node --harmony-generators`.

Here's an example for those interested: https://github.com/visionmedia/co

lucio
node-fibers is under the hood, and works on the actual node stable. I hope ES6 generators -when stable- will provide similar tools, so wait.for will be based on ES6 generators.
tspike
Is this different from async.series? If so, in what way?
lucio
- syntax

- wait.for it's simpler

- wait.for uses try/catch

I don't know async.series deeply, but, ¿how do you use results from the first function on the second or third?

wait.for:

  try{
    data = wait.for(function1);
    result = wait.for(function2, data);    
  }
  catch(err){
    // error handling
  };
async.series:

  async.series([
    function1,
    function2
    ],
    // optional callback
    function(err, results){
      // results is now equal to ['one', 'two']
    });
async has a lot of functionality. You can use wait.for and async at the same time. They supply different functionality.

Example:

  results = wait.for(async.map,['file1','file2','file3'], fs.stat);
htilford
> ¿how do you use results from the first function on the second or third?

waterfall

  async.waterfall([
    function (cb) {
      . . .
      cb(stuff);
    },
    function (stuff, cb) {
      . . .
zongitsrinzler
Correct me if I'm wrong but isn't a big part of async programming that it forces you to think about error handling. Working as a Java developer I have seen very lousy error handling mostly because try/catch lets you wrap everything and kill errors easily. Most of the node code I have seen handles errors much better thanks to it's async nature.
gruseom
If you do it well, you can get very precise error handling, because errors are detected in the context where they first occur. However, to do it well you have to put a try/catch around every asynchronous operation, which makes nested callbacks even harder to read than they already are. Consequently people write code that doesn't handle all possible errors, and when one falls between the cracks, it gets caught only by the top-level default handler (assuming you've declared one), at which point the context is lost.

The other problem that comes up is the difficulty of getting a backtrace beyond the current callback. It can be hard to troubleshoot an error in a callback when you can't look back at how the async call got started. In such cases what you really want is a list of backtraces that capture all the relevant history. I seem to recall that people have come up with solutions for this, but can't remember what they are. Maybe someone else can comment?

zongitsrinzler
I think this is a good point, sometimes you don't need too precise error handling, then try/catch is very good.
tg3
Most Node.js I've seen doesn't make extensive use of Javascript's built-in exception handling, and instead relies on the convention of the first parameter of a callback being an Error object if one occurred (so no try/catch, which doesn't catch async errors anyway).

The stack trace is a pain for those operations, and any error that occurs as part of "normal" javascript operation (e.g. ReferenceError) has to be handled like a normal exception, which makes the style inconsistent and therefore a bit harder to manage.

I think that the latest or upcoming version of Node is presenting a solution to this in the form of domains, but I'm not too familiar with it.

gruseom
The first-parameter convention only works for application errors and errors from libraries that also follow the convention. Everything else, including the JS runtime, throws exceptions whether we like it or not. So either you have to make sure that your code never triggers any of those—which is a pretty tall order—or put try/catches around everything, even if your own code doesn't throw any exceptions. That's more than a bit harder to manage—if you do it correctly, there are so many try-catch blocks that the rest of the code almost gets drowned out by them. In my observation, most people deal with this by sacrificing a certain amount of error safety in order to make the code more tractable.

It seems like the kind of problem that is better dealt with by a compiler.

aaronyo
Promises help with this quite a lot. http://promises-aplus.github.io/promises-spec/ What 3.2.6.2 means is that any thrown exception is caught and turned into a rejected promise which will bubble up (across async boundaries) until it gets to a rejection handler.

I haven't had too much experience with Domains, but I believe they are generally used for courser grain error handling (because it's a bit verbose to attach them all over the place). E.g., to catch most errors (it's no guarantee what 3p libs do to your context) generated in fulfillment of a particular web request.

gruseom
Can promises guarantee that condition with only library code? or do you still have to have try/catch in the application code, and if so where?

Also, do any of the promises libraries solve the backtrace problem, and if so how do they do it?

tg3
Unfortunately putting a try/catch around a 3p lib doesn't actually capture exceptions generated by those libs (unless they occur before any async operations, which I would say is the exception rather than the rule).

You're right that you have to manage exceptions, which is difficult. The biggest problem I've had with Node is the runtime exceptions that crash the entire process because they go uncaught. To deal with that, I have the server automatically restart on failure - far from an ideal scenario of error recovery (and what I think you're referring to when you talk about sacrificing error safety).

olegp
If all you want to do is make existing async methods synchronous, it's easier to modify the function prototoype like this: https://github.com/olegp/mongo-sync/blob/master/lib/mongo-sy... or just use Futures that come with fibers.

However, in practice you want a higher level of abstraction, since multiple async calls typically end up mapping to one sync call. This applies to streams in particular where you need to wait until they become readable.

I've written Common Node (https://github.com/olegp/common-node) to do just this and made it CommonJS and RingoJS compatible, which means the code written for Common Node runs on both V8 and the JVM. We've been using it in production at https://starthq.com for a while with great success. Due to some optimizations we put in place, Common Node actually performs faster than Node in some benchmarks.

ilaksh
Please also take a look at ToffeeScript which is a brilliant and amazingly relatively unknown solution. https://github.com/jiangmiao/toffee-script
SeanDav
Not dissing your comment, but I am not a believer in any "compile to" language because ultimately when things go wrong you are forced to deal with the underlying language in any case. Rather become an expert in the underlying language and learn to deal with its idiosyncrasies.
ilaksh
I'm not suggesting that people shouldn't learn JavaScript. Anyway all languages compile are abstractions of some lower level (usually multiple levels).
FooBarWidget
> What if... Fibers and WaitFor were part of node core?

> then you can deprecate almost half the functions at: http://nodejs.org/api/fs.html (clue: the Sync versions)

Not quite. I use the Sync versions not to avoid callbacks, but to ensure that nothing else happens while I'm doing the operation. For example, take a look at the Database.reload function from this project (a NoSQL database written in Node): https://github.com/phusion/zangetsu/blob/master/lib/zangetsu.... Reloading the database contents involves traversing multiple directories and reading multiple files. While reloading the database contents, I do not want the database to be modified. With asynchrony, the amount of different states that I have to be aware of rises so quickly that it's much, MUCH easier to use sync calls and tell the user "reloading is not concurrent, deal with it; at least it's correct". Wait.for does not solve this problem.

It seems people believe that you can't run into concurrency problems or race conditions because Node is single threaded. This is false. I even had to write my own locks (https://github.com/phusion/zangetsu/blob/master/lib/zangetsu...) because Node doesn't provide one.

ibizaman
Wouldn't it be easier to just not use Node.js then ? :p