HN Explorer

Ask HN: How do I support multiple file versioning (any file type) using diffs?

1 points by babyent ↗ HN
Hi Friends.

I'm trying to do multi-version support for my users so they can upload the same file (same name) and have a version history.

For example, user uploads "Site Details Oct-2024.docx". They make some changes, upload the file again. Instead of giving them an option to Keep/Replace, I'd like for them to be able to have a version history that they can view/revert/etc. Less clutter, better organization, and most importantly it will make it easier for them to see what actually changed.

Right now I'm doing the naive way - uploading that updated file and storing the distinct files. The downside is that this will Nx my storage. If they have a 2mb file with 5 revisions, my current approach will use up 10mb.

I have no idea how this stuff works, so I did some googling and also asked chatGPT.

I found something called xdelta and bsdiff. Has anyone used this before? Seems like neither are actively maintained.

If you know any other way to do this also please let me know.

Text files are pretty straightforward, but again for formats like PDF docx etc. I am not sure how to do this.

I'd be happy if there are node/go/python bindings (in that order of preference), but no worries if not I can figure it out.

Thank you!

10 comments

[ 3.5 ms ] story [ 38.4 ms ] thread
examango
What are you thinking? It's the business that matters, not the technical details. The program runs as long as it runs. What do you care how it runs. Expand the business first, and when you're making money, hire a programmer who can work out the technical details.
babyent
Totally, which is why I'm not working on it right now. That is why I asked so I could gather information :-)

However, I would prefer to give users a better experience and set up towards having diffs.

In addition, from test users behavior, it seems like some folks have 20+ mb PDF files that they make revisions on and upload. If they do 4-5 revisions on these files it blows out storage. Of course I can just pass these costs on, but if there's a way to do it more efficiently I'd rather do that.

Storage is unlimited for enterprise customers, but again I'm just wondering how to make this process better.

Like I said, text isn't a problem. It is these PDF and other formats where I'm wondering if there's some binary diff solution that already exists.

Anyway, thanks for taking the time to reply.

pestatije
are you sure you dont want to use a versioning system?
babyent
Thank you. Can you please elaborate? I think your question will help me think this through again.

I came to the file versioning conclusion after speaking to a few folks in this space who I want to help.

The common issues that kept coming up are around organization and accountability. In existing systems like Finder/File Explorer (and G Drive, One Drive, other cloud storage) we are prompted to Keep/Replace files. In some existing SaaS these folks use, it has neither versioning nor keep/replace prompts. It just uploads the same file name again and confuses users.

When users upload a new version of a file, with some new changes, people often rename arbitrarily, or accidentally replace, and it is hard to enforce process.

Using versioning will help them stay organized (tidier workspace), show them what changes and who made them, and finally help them trace back if need be.

These are files dealing with designs, permits, drafts, etc. Mostly PDFs, but sometimes doc or excel files as well.

pestatije
right, i am talking about versioning systems like git...text files diffing comes included by default and pdfs, docs, etc might have a way to be dealt with as well
solardev
Are you using a cloud service or trying to host your own server?

If you're on AWS, for example, there's built-in versioning for S3 buckets: https://docs.aws.amazon.com/AmazonS3/latest/userguide/versio.... GCP and Azure have similar versioning systems for their storage.

To save disk space, the term you probably want is "data deduplication" (https://en.wikipedia.org/wiki/Data_deduplication), and there are various algorithms, filesystems, etc. that can help with that. However, it's somewhat slow and risky, because you have to recombine a final file from fragments spread across different versions. And if a fragment gets corrupted, it could affect not just one version but several or even all.

Disk space is generally cheap these days, and it might not be worth sacrificing performance and integrity just to save a few cents per user.

Can you just give them a disk space limit and charge them for additional space if they need it?

babyent
S3 versioning doesn’t do diffs. It stores distinct files and uses internal version number.

I’m specifically curious about binary diff between files :sob: for text files it’s easy and not a problem. But I am wondering for other files.

I will definitely charge them for disk space, but wondering about diff.

Thanks!

zahlman
>I'm trying to do multi-version support for my users so they can upload the same file (same name) and have a version history.

>I have no idea how this stuff works

>I'd be happy if there are node/go/python bindings (in that order of preference)

When you write code in those languages, over time you accumulate multiple versions of it in much the same way, yes? I assume you use a tool such as Git, Mercurial (hg), SVN etc. to manage this.

As you say, text files (like your source code) are pretty straightforward. But these tools can generally be adapted to handle binary files as well - see for example https://stackoverflow.com/questions/3601672 and https://stackoverflow.com/questions/9478023.

xdelta and bsdiff are implementations of actual diffing algorithms - they compare two similar versions of a file and create a representation of what changed.