I looked pretty hard at systems to do parallel batch jobs.
One thing I looked at was to store intermediate results in Amazon S3 and control the execution of job steps with SQS.
Although the system couldn't do everything Hadoop does, it worked for a wide range of problems and had the desirable quality that it could transparently parallelize multiple processing phases when it was possible.
I went with Hadoop, however, because I can run a Hadoop cluster in the office or in the cloud, and that's a big advantage.
1 comment
[ 2.3 ms ] story [ 26.7 ms ] threadOne thing I looked at was to store intermediate results in Amazon S3 and control the execution of job steps with SQS.
Although the system couldn't do everything Hadoop does, it worked for a wide range of problems and had the desirable quality that it could transparently parallelize multiple processing phases when it was possible.
I went with Hadoop, however, because I can run a Hadoop cluster in the office or in the cloud, and that's a big advantage.