Show HN: We scaled Git to support 1 TB repos (xethub.com)
Unlike Git LFS, we don’t just store the files. We use content-defined chunking and Merkle Trees to dedupe against everything in history. This allows small changes in large files to be stored compactly. Read more here: https://xethub.com/assets/docs/how-xet-deduplication-works
Today, XetHub works for 1 TB repositories, and we plan to scale to 100 TB in the next year. Our implementation is in Rust (client & cache + storage) and our web application is written in Go. XetHub includes a GitHub-like web interface that provides automatic CSV summaries and allows custom visualizations using Vega. Even at 1 TB, we know downloading an entire repository is painful, so we built git-xet mount - which, in seconds, provides a user-mode filesystem view over the repo.
XetHub is available today (Linux & Mac today, Windows coming soon) and we would love your feedback!
Read more here:
146 comments
[ 2.1 ms ] story [ 215 ms ] threadI see far less reasons to version data, in fact, I find reasons against versioning data and storing them in diffs.
People in ML ops use git because they aren't very sophisticated with programming professionally and they have git available to them and they haven't run into the consequences of using it to store large binary blobs, namely that it becomes impossible to live with eventually and wastes a huge amount of time and space.
ML didn't invent the need for large artifacts that can't be versioned in source control but must be versioned with it, but they don't know that because they are new to professional programming and aren't familiar with how it's done.
It's not perfect, and still feels like a bit of a hack compared to something like p4 for the context I uses LFS in (game dev), but it works, and doesn't require expensive custom licenses when teams grow beyond an arbitrary number like 3 or 5.
As an example, a Unity game repo reduced in size by 41% using our block-level deduplication vs Git LFS. Raw repo was 48.9GB, Git LFS was 48.2GB, and with XetHub was 28.7GB.
Why do you think using a Git-based solution is a hack compared to p4? What part of the p4 workflow feels more natural to you?
Some of these have workarounds and hacks for more experienced users. I'm not about to run around teaching people the intricacies of arcane git incantations, while p4 functions, by default, how you'd want to. The programming side is better on git though, yeah.
We're working on perforce-style locking on XetHub, and I believe git already supports things like only cloning the latest version of files. Cloning the full repo without "smudging" (pulling in binary file contents) is already possible, and cloning while smudging a subset is on our roadmap. We're definitely on a path to making git UX for dealing with large binary files as easy as perforce, and there are lots of advantages to keeping a git-based workflow for teams that already work with git.
Mlops people are very aware of tools that are more suited for the job... even too aware in fact. The entire field is full of tools, databases, etc to the point where it's hard make sense of it. So your comment is a bit weird to me
This is versioning
Well unless fraud is the goal.
https://www.dolthub.com/blog/2022-07-11-dolt-case-studies/
The wheels of data versioning just get reinvented over and over and over again, with all sorts of slightly different tools. Most of the job of "boring CRUD app development" is data version management and some of the "joy" is how every database you ever encounter is often its own little snowflake with respect to how it versions its data.
There have been times I've pined for being able to just store it all in git and reduce things to a single paradigm. That said, I'd never actually want to teach business analysts or accountants how to use git (and would probably spend nearly as much time building custom CRUD apps against git as against any other sort of database). There are times though where I have thought for backend work "if I could just checkout the database at the right git tag instead needing to write this five table join SQL statement with these eighteen differently named timestamp fields that need to be sorted in four different ways…".
Reasons to version data are plenty and most of the data versioning in the world is ad hoc and/or operationally incompatible/inconsistent across systems. (Ever had to ETL SharePoint lists and its CVC-based versioning with a timestamp based data table? Such "fun".) I don't think git is necessarily the savior here, though there remains some appeal in "I can use the same systems I use for code" two birds with one stone. Relatedly, content-addressed storage and/or merkle trees are a growing tool for Enterprise and do look a lot like a git repository and sometimes you also have the feeling like if you are already using git why build your own merkle tree store when git gives you a swiss army knife tool kit on top of that merkle tree store.
As someone who'd love to put their data into a git like system, this sounds pretty interesting. Aside from not offering a tier for someone like me who would maybe have a couple of repositories of size O(250GB) it's unclear how e.g. bandwidth would work & whether other people could simply mount and clone the full repo if desired for free etc.
In general, we are thinking about usage-based pricing (which would include bandwidth and storage) - what are your thoughts for that?
Also, where would you be mounting your repos from? We have local caching options that can greatly reduce the overall bandwidth needed to support data center workloads.
First, it's all open-source, so I can take it and run it. Second, you provide a hosted service, and by virtue of being the author, you're the default SaaS host. You charge a premium over AWS fees for self-hosting, which works out to:
1. Enough to sustain you.
2. Less than the cost of doing dev-ops myself (AWS fees + engineer).
3. A small premium over potential cut-rate competitors.
You offer value-added premium services too. Whether that's economically viable, I don't know.
1. I'm unlikely to adopt something proprietary for this sort of use. Lock-in is bad, but it's especially with a startup which can disappear tomorrow or pivot who is holding my key data. Open-source means if you disappear, I'm alive. I don't trust you, and open-source mostly means I don't need to.
2. With open-source, pricing which is more than the cost of AWS + engineer makes no sense. I'd rather host myself. However, the labor costs means that AWS + engineer gives a lot of potential profit margin for you. I'd much rather not run servers myself.
3. A cut-rate competitor will have similar per-customer cost structure as you, but you'll have somewhat higher fixed costs. For me, paying a little bit more for the reliability of going with the most competent vendor is an obvious choice (which you would be, by virtue of having written it). I wouldn't consider a cut-rate vendor unless the savings was very significant.
4. Not in my current job, but in past jobs, I'd gladly pay for service and support on top of that. A lot of things are cheaper for you to do (or explain) as the author / expert, than for my guys to figure out themselves.
For this to work requires a certain economy-of-scale. That requires deep VC pockets to get to profitability, or a good beachhead (e.g. a single big customer). There are many single big customers, but I have no idea how you'd build that connection. Most are in oddball industries, and not companies you'd think of.
For example, a few years back, I interacted with a major military contractor who specializes in manufacturing, and has no competence in technology. They did many billions in business, and had just paid a few million for a semi-incompetent tech consulting firm as an acqui-hire to try to build basic tech expertise. Outsourcing everything to you would have been a far better decision for them (and for many companies like them) if they could find you and vet you, and vice-versa (probably with a strategic investment as well).
They were very good at what they did, but what they did was very much not tech or software.
Generally usage based pricing sounds fair. In the end for cases like mine where it's "read rarely, but should be available publicly long term" it would need to compute with pricing offered by the big cloud providers.
I'm about to leave my academic career and I'm thinking about how to make sure all my detector data will be available to other researchers in my field in the future. Aside from the obvious candidate https://zenodo.org it's an annoying problem as usually most universities I'm familiar with only archive data internally, which is hard to access for researchers from different institutions. As I don't want to rely on a single place to have that data available I'm looking for an additional alternative (that I'm willing to pay for out of my own pocket, it just shouldn't be a financial burden).
In particular while still taking data a couple of years ago I would have loved being able to commit each daily data taking in the same way as I commit code. That way having things timestamped, backed up and all possible notes that came up that day associated straight in the commit message would have been very nice.
Regarding mounting I don't have any specific needs there anymore. Just thinking about how other researchers would be able to clone the repo to access the data.
[1] When you run `git annex add` it hashes the file and moves the original file to a `.git/annex/data` folder under the hash/content addressable file system, like git. Then it replaces the original file with a symlink to this hashed file path. The file is marked as read only, so any command in any language which tries to write to it will error (you can always `git annex unlock` so you can write to it). If you have duplicated files, they easily point to the same hashed location. As long as you git push normally and back up the `.git/annex/data` you're totally version controlled, and you can share the subset of files as needed
Imagine you have a 500MB file (lastmonth.csv) where every day 1MB is changed.
With file-based deduplication every day 500MB will be uploaded, and all clones of the repo will need to download 500MB.
With block-based deduplication, only around the 1MB that changed is uploaded and downloaded.
Block-based dedup can be done either with fixed block sizes or variable block sizes. For a database with fixed page sizes, a fixed block size matching the page size is most efficient. For a database with variable page sizes, a variable block size will work better, assuming there the dedup "chunking" algorithm is fine-grained enough to detect the database page size. For example, if the db used a 4-6K variable page size and the dedup algo used a 1M variable block size, it could not save a single modified db page but would save more like 20 db pages surrounding the modified page.
Your column vs row question depends on how the db stores data, whether key fields are changed, etc. The main dedup efficiency criteria are whether the changes are physically clustered together in the file or whether they are dispersed throughout the file, and how fine-grained the dedup block detection algorithm is.
I have a couple ~1TB repositories I've had the misfortune of working with using perforce in the past.
Do you have a repo you could try us out with?
We have tried a couple Unity projects (41% smaller due to republication) but not much from Unreal projects yet.
I keep expecting someone to come along and dethrone it but as far as I can tell it hasn't been done yet. The combination of specific filetree views, drop-in proxies, UI-forward and checkout based workflow that works well with unmergeable binary assets still left Git LFS and other solutions in the dust.
+1 on testing this against a moderate size gamedev repo, that usually has some of the harder constraints where code + assets can be coupled and the art portion of a sync can easily top a couple hundred GB.
I actually wrote a script which I'm happy to share, that makes this much easier, and even lets you mount your bup repo over .git/annex/objects for direct access.
[1]: https://git-annex.branchable.com/walkthrough/using_bup/
[2]: https://github.com/bup/bup
It's always a tradeoff. Deduplication is a CPU-heavy process, and if it's done inline, it is also memory-heavy, so you're basically trading CPU and memory for storage space. It heavily depends on the use-case (and the particular FS / deduplication implementation) whether it's worth it or not
[1]: https://btrfs.wiki.kernel.org/index.php/Deduplication
[2]: https://docs.oracle.com/cd/E36784_01/html/E39134/fsdedup-1.h...
I am not a user of git annex but I do know that it works perfectly with an rsync.net account as a target:
https://git-annex.branchable.com/forum/making_good_use_of_my...
... which means that you could do a dumb mirror of your repo(s) - perhaps just using rsync - and then let the ZFS snapshots handle the versioning/rotation which would give you the benefits of block level diffs.
One additional benefit, beyond more efficient block level diffs, is that the ZFS snapshots are immutable/readonly as opposed to your 'git' or 'git annex' produced versions which could be destroyed by Mallory ...
Can you explain this a bit? I don't know anything about ZFS, but it sounds as though it creates snapshots based on block level differences? Maybe a git-annex backend could be written to take advantage of that -- I don't know.
When you have checked something out and fetched it, then it consumes space on disk, but that is true with git-lfs, and most other tools like it. It does NOT consume any space in any git object files.
I regularly use a git-annex repo that contains about 60G of files, which I can use with github or any git host, and uses about 6G in its annex, and 1M in the actual git repo itself. I chain git-annex to an internal .bup repo, so I can keep track of the location, and benefit from dedup.
I honestly have not found anything that comes close to the versatility of git-annex.
[1]: https://news.ycombinator.com/item?id=33976418
[2]: https://git-annex.branchable.com/special_remotes/
I know that they're well within their rights to do this as they only ever offered subscription licensing for Semantic Merge, but that doesn't make it suck less to lose access.
Can you sync to another machine without Xethub ?
How about cleaning up old files?
- Automatically includes all files >256KB in size
- By default data is de-duplicated 16KB chunks instead of whole files (with the ability to customize this per file type).
- Has a "mount" command to allow read-only browse without downloading
When launching on HN it would be better if the team was a bit more transparent with the internals. I get that "we made a better GitLFS" doesn't market as well. But you can couple that with a credible vision and story about how you are a better and where you are headed next. Instead this is mostly closer to market speak of "trust our magic solution to solve your problem".
(excerpt from the OP post:
> Unlike Git LFS, we don’t just store the files. We use content-defined chunking and Merkle Trees to dedupe against everything in history. This allows small changes in large files to be stored compactly. Read more here: https://xethub.com/assets/docs/how-xet-deduplication-works)
Also, why can't Git show me an accurate progress-bar while fetching?
As for why git can't show you an accurate progress bar while fetching (specifically when using an extension like git-lfs or git-xet), this has to do with the way git extensions work -- each file gets "cleaned" by the extension through a Unix pipe, and the protocol for that is too simple to reflect progress information back to the user. In git-xet, we do write a percent-complete to stdout so you get some more info (but a real progress bar would be nice).
That goes double for products where paying for "enterprise" is only to get SAML, which at least in my experience causes me to go shopping for an entirely different product because I view it as extortion
I don't see an issue with charging more for SSO though as I said some of the prices are egregious.
Sometimes even individual art project files can be many gigabytes each. I saw a .psd that was 30gb because of the embedded hi-res reference images.
You can throw pretty much anything in there, in one place and things like locking, partial-checkout, etc. Which gets artists to use it
Something like git-lfs is the appropriate solution. You need a little bit of centralization.
All that's really needed is a way to mark individual files as lazily fetched from a remote only when needed. LFS is a hacky substandard way to emulate that behaviour. It should be built in to Git.
The git userspace would need to be able to easily:
1. Not grab all files
2. Got grab the whole version history
... and that's more-or-less it. At that point, it'd do great with large files.
If you're interested in something you can self-host... I work on Pachyderm (https://github.com/pachyderm/pachyderm), which doesn't have a Git-like interface, but also implements data versioning. Our approach de-duplicates between files (even very small files), and our storage algorithm doesn't create objects proportional to O(n) directory nesting depth as Xet appears to. (Xet is very much like Git in that respect.)
The data versioning system enables us to run pipelines based on changes to your data; the pipelines declare what files they read, and that allows us to schedule processing jobs that only reprocess new or changed data, while still giving you a full view of what "would" have happened if all the data had been reprocessed. This, to me, is the key advantage of data versioning; you can save hundreds of thousands of dollars on compute. Being able to undo an oopsie is just icing on the cake.
Xet's system for mounting a remote repo as a filesystem is a good idea. We do that too :)
How about cleaning up old versions?
...isn't that just parsing git diff --name-only A..B tho ? "Process only files that changed since last commit" is extremely simple problem to solve.
Dolt hasn't come up here yet, probably because we're focused on OLTP use cases, not MLOps, but we do have some customers using Dolt as the backing store for their training data.
https://github.com/dolthub/dolt
Dolt also scales to the 1TB range and offers you full SQL query capabilities on your data and diffs.
Kart (https://kartproject.org) is built on git to provide data version control for geospatial vector & tabular data. Per-row (feature & attribute) version control and the ability to collaborate with a team of people is sorely missing from those workflows. It's focused on geographic use-cases, but you can work with 'plain old tables' too, with MySQL, PostgreSQL and MSSQL working copies (you don't have to pick - you can push and pull between them).
Is the Merkle true used because it brings something else than deduplication, like chunks integrity verification or something like that?
Why do you need 1Tb for repos? What do you store inside, besides code and some images?
I personally would love to be able to store datasets next to code for regression testing, easier deployment, easier dev workstation spin up, etc.
Once you get to that amount of images it would be much easy to manage it with some files storage solution.
Or I'm missing something important?
Which is a huge hassle, and a lot of work I’d rather not do.
My current photogrammetry dataset is well over 1TB, and it isn’t a lot for the industry by any stretch of the imagination.
In fact, the only thing that considers it ‘a lot’ and is hard to work with is git.