MapRejuice - Distributed Client Side Computing (Node Knockout Entry) (nodeknockout.com)

Thanks! We're pretty happy with what we've got, but there's still a lot more planned.

You're right about the potential to reach more; this is actually where this idea spawned from. Always thought that *@home had too many barriers (having to download/install/etc). This is fairly automatic, and takes almost no integration effort on anyone's part other than one line of HTML. ;D

Yeah, we're still musing on this, it's certainly a known point. Besides the obvious method of offering an "opt out" extension (like Google does with their ad/tracking stuff), there's the more ethical pieces to consider - e.g, loading on Mobile, where people may be paying for data consumption... we shouldn't be taking their data rates high (or potentially using up their battery in the background) for arbitrary reasons.

As of right now it doesn't even fire on mobile devices for this reason.

tl;dr Good point, we're aware, and there'll be ways to deal with it. Must sleep now.

What exactly do you mean with "stealing your CPU cycles"?

We've been discussing whether or not to make the computations transparent to the visitor of the sites. The workers could expose an API to site-owners to start/pause/stop/disable the computations. Would that help?

While watching YouTube video - my CPU load from browser is about 15-20%, that means YouTube player "steals" up to 400 million CPU cycles each second!..

There's nothing to worry about.

Exactly! this was one of the thoughts we had. Can you imagine someone like farmville implementing it? Lots of potential revenue.

Would also be great to donate cycles to educational or non-profit teams.

Here's something I tried running (index generation):

    map = function (docID, text) {
      var results = [];
      var words = text.split(' ');
      for(var index in words) {
        results.push({key: words[index], value: docID});
      }
    
      this.done = true;
      this.results = results;
    }
    
    reduce = function (word, docIDs) {
      self.postMessage("starting reduce for distributed relevance-sorted index generation");
      var counts = {};
      var results = [];
      for(var index in docIDs) {
        var docID = docIDs[index];
        if(docID in counts) {
          counts[docID]++;
        } else {
          counts[docID] = 1;
          results.push(docID);
        }
      }
    
      results.sort(function(a, b) {return counts[b] - counts[a];})
      
      this.done = true;
      this.results = results;
    }
    
    data = {1: "the quick brown fox jumps over the lazy dog while the slow yellow fox stumbles around the yard", 2: "on the ning nang nong where the cows go bong and the monkeys all say boo", 3: "lions and tigers and bears oh my , i suppose next we'll see monkeys and maybe a fox"}

Didn't seem to work, but maybe I did something wrong.

A job is one run of map or reduce on one key.

In the case of word count, during the map phase, that means that 300 input files are analyzed per minute. During the reduce phase that indeed means that only 300 unique word frequencies per minute are computed.

This is a purely academic consideration, though. Currently we don't have any running jobs; the clients are idling. When idling, the clients request a new job every 10 seconds, which is longer than a typical reduce run would take. We're also throttling the load on the server from the workers (workers only spawn 50% of the time on the include script) because we don't want to ruin our hackety-hack server. Latency between the workers and servers and the server and our free couchone instance make up large portions of the running time of smaller jobs.

This was just an experiment. Once (and if) we figure out the problems I mentioned earlier we can start building to scale :)