In chapter 2, you describe writing the master dataset to a distributed filesystem (presumably HDFS). If you create a new file with a new record for each update to the dataset, wouldn't this result in lots of small files…
In chapter 2, you describe writing the master dataset to a distributed filesystem (presumably HDFS). If you create a new file with a new record for each update to the dataset, wouldn't this result in lots of small files…