We've been trying out goldengate to get streaming replication, but it has proven rather unreliable. Stops replicating if you sneeze in its general vicinity. I wonder whether the alternatives like shareplex and tungsten are more reliable.
I feel like I've recommended it enough times I'm turning into a shill but Pentaho is an open-source and commercially supported ETL tool that will natively do what you want, or call sqoop when you discover that's kinda slow. :-) And no I definitely don't work there.
Airflow is very similar to Luigi; we've been using in in production to schedule all of our workflows for ~4 months now and it's worked out really well for us.
Thats a great idea - but could you elaborate on the scheduling of jobs on Luigi - it does not have a scheduler like AirFlow - how do you schedule Luigi tasks ?
Have a look at Kafka Connect (http://docs.confluent.io/2.0.0/connect). The JDBC Connector will poll for database changes changes and push them to a Kafka topic. Means you should see all the changes in the database rather than a snapshot say once a day
Thumbtack is great for lazy consumers. But word on the street is they contribute to too much price pressure on the market, therefore drive the overall quality of services down. Good work Thumbtack! You have figured out a convenient way to sacrifice long term value in favor of short term profit.
18 comments
[ 1.7 ms ] story [ 55.8 ms ] threadhttp://www.striim.com/download-striim/
You can define dependencies between jobs based on output file which allows you to re-run only part of your pipeline
Airflow is very similar to Luigi; we've been using in in production to schedule all of our workflows for ~4 months now and it's worked out really well for us.