I'm amazed at the fact that more people don't push mostly static or rarely updated data to local sqlite instances or flat files.
In my line of work, the goverment distributes a daily file with all those tax ids that can emit tax deductible invoices. Since it has every tax id along with additional information, the file is quite large (30Gb of XML split into 4 files because of the format). This file has to be downloaded and used daily by all our front end servers.
Every Engineer immediately proposed throwing the data on the Galera cluster that we're currently running. They didn't see the problem of slowing down the main database every day with a huge update cycle (you could diff it previously but it's still a large update). We'd also be querying a network storage for basically static data, which I thought idiotic.
I proposed having a simple daily task in a job server that downloads the data, parses it and writes it to a sqlite database. It then pushes this file to all of our frontend servers. The frontend servers simply reference that file. It's a lot faster, especially since the whole thing can fit in memory when its compressed binary data (drops down to about 12G), and you simplify your reads.
I completely agree with the statement made by the author too:
"Aside from the quirk of sharing SQLite across containers, it delivers extremely well on one core idea: the thematic shift of complexity from the data plane read path to the control plane write path."
1 comment
[ 4.8 ms ] story [ 12.5 ms ] threadIn my line of work, the goverment distributes a daily file with all those tax ids that can emit tax deductible invoices. Since it has every tax id along with additional information, the file is quite large (30Gb of XML split into 4 files because of the format). This file has to be downloaded and used daily by all our front end servers.
Every Engineer immediately proposed throwing the data on the Galera cluster that we're currently running. They didn't see the problem of slowing down the main database every day with a huge update cycle (you could diff it previously but it's still a large update). We'd also be querying a network storage for basically static data, which I thought idiotic.
I proposed having a simple daily task in a job server that downloads the data, parses it and writes it to a sqlite database. It then pushes this file to all of our frontend servers. The frontend servers simply reference that file. It's a lot faster, especially since the whole thing can fit in memory when its compressed binary data (drops down to about 12G), and you simplify your reads.
I completely agree with the statement made by the author too:
"Aside from the quirk of sharing SQLite across containers, it delivers extremely well on one core idea: the thematic shift of complexity from the data plane read path to the control plane write path."