Looks great. I was looking into creating something similar in terms of processing, some years ago, but never took it very far. Your work sounds very interesting and I can think of some interesting uses - I have access to some fascinating data sets.
Really great to know, thanks! Do you think you would also be able to help us maintaining and enhancing it? Could you share more ideas regarding the usage of the platform?
Please, could you share the uses you thoughts for it? This is what we are most interested in. We did the platform for ourselves, and it seems powerful now, we would like to know what could interest others...
We have developed kind of the same concept but agnostic to the underlying runtime.
by agnostic, I mean that we can take advantage of Hadoop (via cascading) for our full indexation processes or Storm for the incremental indexation processes.
Another difference is the way we handle the data, we chose to use tuples (as in cascading and storm) to transfer the data between the different components.
This is interesting, indeed.
As your platform is agnostic to the runtime, wouldn't it be easy to use ours instead of hadoop or storm? Or you don't see any benefit? About the tuples, you still could use them, if I understood it well...
Scheduling is done by any external tool like cron, a java app running quartz or control-m. However, the processor job knows the last time it was run and know which data it should process to update the time. You might also choose to run the job overriding the last time it ran, to reprocess data it you want.
9 comments
[ 2.8 ms ] story [ 34.5 ms ] threadWe are strongly considering opening the source of our DMP (Data Management Platform), if it proves to be interesting to other developers / companies.
Please, feel free to send questions, comments and critics... We will try to reply them all.
Regards, Gabriel.
We have developed kind of the same concept but agnostic to the underlying runtime. by agnostic, I mean that we can take advantage of Hadoop (via cascading) for our full indexation processes or Storm for the incremental indexation processes. Another difference is the way we handle the data, we chose to use tuples (as in cascading and storm) to transfer the data between the different components.