Ask HN: What is the great (or better) option to replicate storage?
I (or we're) am running a Kubernetes Cluster.
It's total of 11 worker nodes. each node have a 2 HDDs and mostly 2 of NVMe Storage. on Networking side, just 1 10Gbps Ethernet port is attached to each nodes.
However, it's pretty hard to determine which distributed storage solution (or way) is fit to our situation.
We've tried a various ways to achieve.
- Longhorn: Great. Simple to Setup. Sometimes it goes to Read-Only State. (because changed block couldn't synced to other replicated storage, especially Superblock)
- OpenEBS (copied version of Longhorn iSCSI version): Same as Longhorn.
- Ceph: Great. No issues at all. But really stressful when HDDs are jammed with S.M.A.R.T. Error.
How do you manage a storage cluster like ours?
Someone suggests to "don't replicate unless you need to" or "use NFS to Shard disks" but IDK it's the best way to keep the data on failure.
Thank you in advance.
Lee.
3 comments
[ 2.9 ms ] story [ 17.4 ms ] thread- Databases. (Usually PostgreSQL, seems like it's pretty easy to setup to replicate between PostgreSQL instances.)
- Qdrant, or milvus. (seems like they're support replication between instances, we've *accidentally* running a replicated DB on Single Ceph Cluster)
- Some user generated contents. (usually from Jupyter Environment)
> Sounds like you want to do your own storage infra?
We don't actually have a "storage infra". it's just a bunch (I really don't think 2 HDD is a part of "Bunch") of disks attached to each machines (or node).