HN Explorer

Ask HN: Best way to store integer array/list per key

1 points by sangupta ↗ HN
What is the best data-store or technique if we were to store millions of integer lists identified by a unique key. The following functions are intended on such lists:

1. Read for a given key 2. Add items to the end of the list against a key 3. Read portions of this list, from-to index 4. Rarely, remove items from the list

The data size is estimated to be roughly 100-million keys, with average of 2-3 thousand integer-list per key.

The key itself is an integer.

8 comments

[ 2.7 ms ] story [ 26.8 ms ] thread
[deleted]
(comment deleted)
sangupta
The lists are expected to contain around 2-3 thousand integers. The maximum may lie in the range of 10-11 thousand. As the lists are expected to mostly grow, the numbers would increase. But, the older elements can be purged to another storage mechanism as they might not be very heavily accessed.
xd
Sorry I meant to say what size of integer you require, 8,16,32,64bit?
sangupta
Sorry - my bad.

32-bit for now. I don't anticipate them going to 64-bit so soon.

xd
Just working on something, will be right with you.
sangupta
No problem - I will wait for your response. Thanks in advance.
xd
https://gist.github.com/AndrewRose/8805629

Considering you are storing only millions of rows I figured a mysql implementation using blobs should do the trick .. but it all depends on the load you expect .. worth playing around with the idea I think.

sangupta
Thanks for the code and helping with this. I have posted my queries to the gist itself.