HN Explorer

17 comments

[ 6.2 ms ] story [ 51.3 ms ] thread
hobbyist
So, what is the key innovation that changes hadoop from batch processing to real time. Is it similar to google's dremel?
TallGuyShort
This doesn't change Hadoop from batch processing to real time - it's a new query engine that uses the same data sources/formats as Hadoop and the same interfaces as Hive. So yeah, similar idea to Google Dremel / F1.
monstrado
You're right that the data is still stored in HDFS, and you can access the data that was originally accessed in Hive. However, because of the way Impala works, it does in fact provide a real-time query interface for data that is sitting in Hadoop (HDFS). The back-end is written in C++ and accesses/processes data much differently than MapReduce which allows for some pretty awesome performance numbers :-)
ryanpers
It isn't like F1 at all... F1 is a multi-datacenter, transactional datastore and SQL, that google uses to replace mysql.

This aint nothing like that.

monstrado
Correct, the technology is based on Google Dremel.
jbellis
My reading was that columnar storage was a critical piece of Dremel. "Accessing the same data as Hive" doesn't sound like Impala is doing that. Is that optional then or is this just a query engine?
monstrado
I can't say for sure, but I believe Doug Cutting's Trevni format is already in the beta but it's a slow version. Impala is initially focusing on architecture, I'm sure columnar format will have attention eventually.

https://github.com/cutting/trevni

jgrahamc
This looks very interesting. I've got multiple PB of log files that I'd like to be able to do real-time and batch processing on. Log files are being generated at a small number of Gbps.

Anyone else got that much data and have a good solution?

untitledwiz
I don't think anyone has managed to get interactive speeds (~seconds to max couple of minutes) at the PB scale.
benbjohnson
What kind of analysis are you trying to do on that dataset? Log files can be verbose as well. Do you need all the data in the logs or can you cut down the data in each row to only what you need?
nphase
Interesting, I wonder how this will be different than Apache Drill, championed by Ted Dunning (MapR)
rmnoon
They're both Dremel clones, but this one seems to be a lot more complete (as Drill is just getting started).