The fact that copying doesn't copy seems dangerous. Like what if I wanted to copy for the purpose of modifying the file while retaining the original. A trivial example of this might be I have a meme template and I want to write text in it while still keeping a blank copy of the template.
There's a place for alias file pointers, but lying to the user and pretending like an alias is a copy is bound to lead to unintended and confusing results
Copy-on-write means that it performs copy only when you make the first change (and only copies part that changes, rest is used from the original file), until then copying is free.
I love the model of it being free to scan and see if you'd get any benefit, then paying for the actual results. I, too, am a packrat, ran it, and got 7GB to reclaim. Not quite worth the squeeze for me, but I appreciate it existing!
He’s talked about it on the podcast he was on. So many users would buy this, run it once, then save a few gigs and be done. So a subscription didn’t make a ton of sense.
After all how many perfect duplicate files do you probably create a month accidentally?
There’s a subscription or buy forever option for people who think that would actually be quite useful to them. But for a ton of people a one time IAP that gives them a limited amount of time to use the program really does make a lot of sense.
And you can always rerun it for free to see if you have enough stuff worth paying for again.
it’s very refreshing compared to those “free trials” you have to remember to cancel (pro tip: use virtual credit cards which you can lock for those so if you forget to cancel the charges are blocked)
however has anyone been able to find out from the website how much the license actually costs?
am I really that old that I remember this being the default for most of the software about 10 years ago? Are people already that used to the subscription trap that they think this is a new model ?
I grew up with shareware in the 90s that often adopted a similar model (though having to send $10 in the mail and wait a couple weeks for a code or a disk to come back was a bit of a grind!) but yes, it's refreshing in the current era where developers will even attempt to charge $10 a week for a basic coloring in app on the iPad..
For me the value in a dedup app like this isn't as much the space savings, since I just don't generate huge amounts, but it's the lack of duplicated files, some of which or all in aggregate may be large. There are some weird scenarios where this occurs, usually due to having to reconcile a hard drive recovery with another location for the files, or a messy download directory with an organized destination.
For example, I discovered my time machine backup kicked out the oldest versions of files I didn't know it had a record of and thought I'd long since lost, but it destroyed the names of the directories and obfuscated the contents somewhat. Thousands of numerically named directories, some of which have files I may want to hang onto, but don't know whether I already have them or not, or where they are since it's completely unstructured. Likewise, many of them may just have one of the same build system text file I can obvs toss away.
In earlier episodes of ATP when they were musing on possible names, one listener suggested the frankly amazing "Dupe Nukem". I get that this is a potential IP problem, which is why John didn't use it, but surely Duke Nukem is not a zealously-defended brand in 2025. I think interest in that particular name has been stone dead for a while now.
It's a genius name, but Gearbox owns Duke Nukem. They're not exactly dormant. Duke Nukem as a franchise made over a billion in revenue. In 2023, Zen released a licensed Duke Nukem pinball table, so there is at least some ongoing interest in the franchise.
Reminds me of Avira's Luke Filewalker - I wonder if they needed any special agreement with Lucasfilm/Disney. I couldn't find any info on it, and their website doesn't mention Star Wars at all.
< You can also create [APFS (copy on write) clones] in Terminal using the command `cp -c oldfilename newfilename` where the c option requires cloning rather than a regular copy.
`fclones dedupe` uses the same command[1]:
if cfg!(target_os = "macos") {
result.push(format!("cp -c {target} {link}"));
brew install fclones
cd ~
fclones group . | fclones dedupe
I've used fclones before in the default mode (create hard links) but this is the first time I've run it at the top level of my home folder, in dedupe mode (i.e. using APFS clones). Fingers crossed it didn't wreck anything.
Nice, also compression at file system level can save a lot of space and with current CPU speeds is completely transparent. It is feature from HFS+ that is still works in APFS, but is not officially supported anymore, what is wrong with you Apple ?
This tool to enable compression is free and open source
Also note about APFS vs HFS+, if you use HDD e.g. as backup media for Time Machine, HFS+ is must have over APFS as it is optimised only for SSD (random access).
Not so smart Time Machine setup utility forcefully re-creates APFS on a HDD media, so you have to manually create HFS+ volume (e.g. with Disk Utily) and then use terminal command to add this volume as TM destination
As a web dev, it’s been fun listening to Accidental Tech Podcast where Siracusa has been talking (or ranting) about the ins and outs of developing modern mac apps in Swift and SwiftUI.
The part where he said making a large table in HTML and rendering it with a web view was orders of magnitude faster than using the SwiftUI native platform controls made me bash my head against my desk a couple times. What are we doing here, Apple.
Hacker News loves to hate Electron apps. In my experience ChatGPT on Mac (which I assume is fully native) is nearly impossible to use because I have a lot of large chats in my history but the website works much better and faster. ChatGPT website packed in Electron would've been much better. In fact, I am using a Chrome "PWA App" for ChatGPT now instead of the native app.
Someone more experienced that me could probably comment on this more, but theoretically is it possible for Electron production builds to become more efficient by having a much longer build process and stripping out all the unnecessary parts of Chromium?
> In my experience ChatGPT on Mac (which I assume is fully native)
If we are to believe ChatGPT itself: "The ChatGPT macOS desktop app is built using Electron, which means it is primarily written in JavaScript, HTML, and CSS"
I wish there were modern benchmarks against browser engines. A long time ago native apps were much faster at rendering UI than the browser, but that may performance rewrites ago, so I wonder how browsers perform now.
Shoutout to iced, my favorite GUI toolkit, which isn't even in 1.0 yet but can do that with ease and faster than anything I've ever seen: https://github.com/iced-rs/iced
It's easy to write a quick and clean UI toolkit, but it's when you add all the stuff for localization (like support for RTL languages - which also means swapping over where icons are) and accessibility (all the screen reader support) is where you really get bogged down and start wanting to add all these abstractions that slow things down.
SwiftUI is a joke when it comes to performance. Even Marco's Overcast stutters when displaying a table of a dozen rows (of equal height).
That being said, it's not quite an apples to apples comparison, because SwiftUI or UIKit can work with basically an infinite number of rows, whereas HTML will eventually get to a point where it won't load.
Yes. Hyperspace is finding the identical files and then replacing all but one copy with a reflink copy using the filesystem's reflink functionality.
When you asked about the filesystem, I assumed you were asking about which filesystem feature was being used, since hyperspace itself is not provided by the filesystem.
Someone else mentioned[0] fclones, which can do this task of finding and replacing duplicates with reflinks on more than just macOS, if you were looking for a userspace tool.
Hyperspace uses built in APFS features, it just applies them to existing files.
You only get CoW on APFS if you copy a file with certain APIs or tools.
If you have a program that does it manually, you copied a duplicate to somewhere on your desk from some other source, or your files already existed on the file system when you converted to APFS because you’ve been carrying them for a long time then you’d have duplicates.
APFS doesn’t look for duplicates at any point. It just keeps track of those that it knows are duplicates because of copy operations.
Yes, Linux has a systemcall to do this for any filesystem with reflink support (and it is safe and atomic). You need a "driver" program to identify duplicates but there are a handful out there. I've used https://github.com/markfasheh/duperemove and was very pleased with how it worked.
Also models that various AI libraries and plugins love to autodownload into custom locations. Python folks definitely need to learn caching, symlinks, asking a user where to store data, or at least logging where they actually do it.
Edit: Let’s be real here, Tim Cook is keeping the lights on. He isn’t a product guy. They lack leadership and vision at present and are committed to a foolish release cycle based on calendar year not quality of product. These wouldn’t have come to pass had Steve lived till today. Yes its opinion but I doubt its an unpopular one.
So I’ll say it again succinctly, to answer “what happened?”
A ~20 y.o. account with perhaps hundreds of devices in history across different continents and countries along with family sharing.
Every time I need to purchase something via Apple, it becomes a quest. Enter password, validate card, welcome to endless login loop. Reboot. Click purchase, enter password, confirm OTP on another device, then nothing happens, purchase button is active, clicks ignored. Reboot. Click "Get", program begins downloading, wait 30s, app cannot be download, go to Settings to verify account. Sure. Account is perfectly fine in Settings. Reboot. Click "Get". Finally program installed. Click in-app purchase. Enter password again. Choose Apple Pay. Proceed with purchase. You need to verify your account. Account is fine in Settings. Reboot. Click purchase. Cannot be completed at this time. Wait couple of hours, try again. Purchase successful.
All. The. Time. For years. On almost all of the devices which I upgrade annually.
I would never dismiss such a complaint with a glib "works for me". And yet, your experience is so utterly, completely different from mine that I have to think something's busted in your account somewhere. I've had an account for about as long, with family sharing and all the rest. I never, ever, have anywhere near that level of difficulty. For me it works as documented: I click "Get", it asks for Face ID to confirm it's really me, then a few seconds later I have the app installed and ready to use.
Again, I don't think you're doing anything wrong, and I don't doubt your experience. But I really think something's fundamentally wrong somewhere, because what you're dealing with is not normal. It's not the common experience others are tolerating.
Oh yes, it's pretty clear to me that something is wrong on Apple's side specifically with my account. Obviously people are having close to zero friction with Apple's stuff.
I can't complain though because I have this account for like couple decades and loosing that account would be painful. Apple did ban my account twice on the grounds that it's a US account while I'm not physically located there. I was able to revert the ban by explaining that I've got a US legal entity (account, banking card, etc) and thus I beg to continue using it. Not taking chances for the third time, so I silently endure.
> Obviously people are having close to zero friction with Apple's stuff.
I don't believe that to be true, I have been having issues with Apple bugs for the last 7-8 years. Totally unnecessary friction due to features I do not want and do not use.
Oh, I was agreeing with you. My 2015 iMac died two weeks ago. Pretty sure it's the SSD I installed when I first got it. And while most of my files are in cloud storage, I also had a series of chained external drives running Time Machine. Guess what? I can't use any apple tools to grab any files because different file system type between that machine and my M1 MB Pro (permissions issues).
I'm going to have to clone the drive, then use terminal to chmod the /usr dirs to extract the files I want (mostly personal music production).
I immediately ordered a Mac mini, but since I didn't want a 256GB drive and 16GB RAM, I'm still waiting for it to arrive from China.
Also, the M1 MB Pro was the most expensive* and worst computer I have ever owned. I wish I had just bought an air. No tactile volume controls. As a musician that is the worst.
(*company I worked for a while in school paid $12k for a Mac 2 FX. Lol.)
In my experience, Macs use up a ridiculous amount of "System" storage for no reason that users can't delete. I've grown tired of family members asking me to help them free up storage that I can't even find. That's the major issue from what I've seen; unless this app prevents apple deliberately eating up 50%+ of the storage space of a machine, this doesn't do much for the people I know.
There's no magic around it, macOS just doesn't do a good job explaining it using the built in tools. Just use Daisy Disk or something. It's all there and can be examined.
I don't think czkawa supports deduplication via reflink so it's not exactly the same thing. fclones as linked by another user is more similar: https://news.ycombinator.com/item?id=43173713
I gave it a try on my massive folder of NodeJS projects but it only found 1GB of savings on a 8.1GB folder.
I then tried again including my user home folder (731K files, 127K folders, 2755 eligible files) to hopefully catch more savings and I only ended up at 1.3GB of savings (300MB more than just what was in the NodeJS folders.)
I tried to scan System and Library but it refused to do so because of permission issues.
I think the fact that I use pnpm for my package manager has made my disk space usage already pretty near optimal.
Oh well. Neat idea. But the current price is too high to justify this. Also I would want it as a background process that runs once a month or something.
This is basically only a win on macOS, and only because Apple charges through the nose for disk space.
Ex - On my non-apple machines, 8GB is trivial. I load them up with the astoundingly cheap NVMe drives in the multiple terabyte range (2TB for ~$100, 4TB for ~$250) and I have a cheap NAS.
So that "big win" is roughly 40 cents of hardware costs on the direct laptop hardware. Hardly worth the time and effort involved, even if the risk is zero (and I don't trust it to be zero).
If it's just "storage" and I don't need it fast (the perfect case for this type of optimization) I throw it on my NAS where it's cheaper still... Ex - it's not 40 cents saved, it's ~10.
---
At least for me, 8GB is no longer much of a win. It's a rounding error on the last LLM model I downloaded.
And I'd suggest that basically anyone who has the ability to not buy extortionately priced drives soldered onto a mainboard is not really winning much here either.
I picked up a quarter off the ground on my walk last night. That's a bigger win.
> This is basically only a win on macOS, and only because Apple charges through the nose for disk space
You do realize that this software is only available on macOS, and only works because of Apple's APFS filesystem? You're essentially complaining that medicine is only a win for people who are sick.
This is NOT a novel or new feature in filesystems... Basically any CoW file system will do it, and lots of other filesystems have hacks built on top to support this kinds of feature.
---
My point is that "people are only sick" because the company is pricing storage outrageously. Not that Apple is the only offender in this space - but man are they the most egregious.
It should be proportional to the total used space, not the space available. The previous commenter said it was a 1 GB savings from ~8 GB of used space; that's equally significant whether it happens on a 10 GB drive or a 10 TB one.
He "only" saved 30%? That's amazing. I really doubt most people are going to get anywhere near that.
When I run it on my home folder (Roughly 500GB of data) I find 124 MB of duplicated files.
At this stage I'd like it to tell me what those files are - The dupes are probably dumb ones that I can simply go delete by hand, but I can understand why he'd want people to pay up first, as by simply telling me what the dupes are he's proved the app's value :-)
Another 30% more than the 1GB saved in node modules, for 1.3GB total. Not 30% of total disk space.
For reference, from the comment they’re talking about:
> I then tried again including my user home folder (731K files, 127K folders, 2755 eligible files) to hopefully catch more savings and I only ended up at 1.3GB of savings (300MB more than just what was in the NodeJS folders.)
> He "only" saved 30%? That's amazing. I really doubt most people are going to get anywhere near that.
You misunderstood my comment. I ran it on my home folder which contains 165GB of data and it found 1.3GB is savings. That isn't significant for me to care about because I currently have 225GB free of my 512GB drive.
BTW I highly recommend the free "disk-inventory-x" utility for MacOS space management.
His comment is pretty understandable if you've done frontend work in javascript.
Node_modules is so ripe for duplicate content that some tools explicitly call out that they're disk efficient (It's literally in the tagline for PNPM "Fast, disk space efficient package manager": https://github.com/pnpm/pnpm)
So he got ok results (~13% savings) on possibly the best target content available in a user's home directory.
Then he got results so bad it's utterly not worth doing on the rest (0.10% - not 10%, literally 1/10 of a single percent).
---
Deduplication isn't super simple, isn't always obviously better, and can require other system resources in unexpected ways (ex - lots of CPU and RAM). It's a cool tech to fiddle with on a NAS, and I'm generally a fan of modern CoW filesystems (incl APFS).
But I want to be really clear - this is people picking spare change out of the couch style savings. Penny wise, pound foolish. The only people who are likely to actually save anything buying this app probably already know it, and have a large set of real options available. Everyone else is falling into the "download more ram" trap.
Absolutely, 100% backwards. The tool cannot save space from disk space that is not scanned. Your "not a big win" comment assumes that there is no space left to be reclaimed on the rest of the disk. Or that the disk is not empty, or that the rest of the disk can't be reclaimed at an even higher rate.
I have a 512GB drive in my MacBook Air M3 with 225GB free. Saving 1GB is 0.5% of my total free space, and it is definitely "below my line." It is a neat tool still in concept.
When I ran it on my home folder with 165GB of data it only found 1.3GB of savings. This isn't that significant to me and it isn't really worth paying for.
BTW I highly recommend the free "disk-inventory-x" utility for MacOS space management.
But the way you put them after each other, makes it sound like npm does de-duplication, and since pnpm tries to be a drop-in replacement for npm, so does pnpm.
So for clarification: npm doesn't do de-duplication across all your projects, and that in particular was of the more useful features that pnpm brought to the ecosystem when it first arrived.
More importantly, pnpm installs packages as symlinks, so the deduping is rather more effective. I believe it also tries to mirror the NPM folder structure and style of deduping as well, but if you have two of the same package installed anywhere on your system, pnpm will only need to download and save one copy of that package.
Even if I have both a Mac and iPhone, but happen to use my Linux computer right now, it seems like the store page (https://apps.apple.com/us/app/hyperspace-reclaim-disk-space/...) is not showing the price, probably because I'm not actively on a Apple device? Seems like a poor UX even for us Mac users.
It’s a free app because you don’t have to buy it to run it. It will tell you how much space it can save you for free. So you don’t have to waste $20 to find out it only would’ve been 2kb.
But that means the parts you actually have to buy are in app purchases, which are always hidden on the store pages.
For some reason "disk-inventory-x" will scan those folders. I used that amazing tool to prune left over Unreal Engine files and docker caches when they put them not in my home folder. The tool asks for a ton of permissions when you run it in order to do the scan though, which is a bit annoying.
It’s not obvious but the system folder is on a separate, secure volume; the Finder does some trickery to make the system volume and the data volume appear as one.
Didn't have time to try it myself, but there is an option for minimal files size to consider clearly seen on the AppStore screenshot. I suppose it was introduced to minimize comparison buffers. It is possible that node modules are sliding under this size and wasn't considered.
He spoke to this on No Longer ery Good, episode 626 of The Accidental Tech Podcast. Time stamp ~1:32:30
It tries, but there are some things it can't perfectly preserve like the last access time. Instances where it can't duplicate certain types of extended attributes or ownership permissions it will not perform the operation.
Well, the FAQ also states that people should notify if you're missing attributes, so it really sounds like it's a predefined list instead of just enumeration through everything.
No word about alternate data streams. I'll pass for now.. Although it's nice to see how much duplicates you have
Q: Does Hyperspace preserve file metadata during reclamation?
A: When Hyperspace replaces a file with a space-saving clone, it attempts to preserve all metadata associated with that file. This includes the creation date, modification date, permissions, ownership, Finder labels, Finder comments, whether or not the file name extension is visible, and even resource forks. If the attempt to preserve any of these piece of metadata fails, then the file is not replaced.
If you find some piece of file metadata that is not preserved, please let us know.
Q: How does Hyperspace handle resource forks?
A: Hyperspace considers the contents of a file’s resource fork to be part of the file’s data. Two files are considered identical only if their data and resource forks are identical to each other.
When a file is replaced by a space-saving clone during reclamation, its resource fork is preserved.
> There is no way for Hyperspace to cooperate with all other applications and macOS itself to coordinate a “safe” time for those files to be replaced, nor is there a way for Hyperspace for forcibly take exclusive control of those files.
This got me wondering why the filesystem itself doesn't run a similar kind of deduplication process in the background. Presumably, it is at a level of abstraction where it could safely manage these concerns. What could be the downsides of having this happen automatically within APFS?
Even using sha-256 or greater type of hashing, I'd still have concerns about letting a system make deletion decisions without my involvement. I've even been part of de-dupe efforts, so maybe my hesitation is just because I wrote some of the code and I know I'm not perfect in my coding or even my algo decision trees. I know that any mistake I made would not be of malice but just ignorance or other stupid mistake.
I've done the entire compare every file via hashing and then log each of the matches for humans to compare, but never has any of that ever been allowed to mv/rm/link -s anything. I feel my imposter syndrome in this regard is not a bad thing.
Now you understand why this app costs more than 2x the price of alternatives such as diskDedupe.
Any halfway-competent developer can write some code that does a SHA256 hash of all your files and uses the Apple filesystem API's to replace duplicates with shared-clones. I know swift, I could probably do it in an hour or two. Should you trust my bodgy quick script? Heck no.
The author - John Siracusa - has been a professional programmer for decades and is an exceedingly meticulous kind of person. I've been listening to the ATP podcast where they've talked about it, and the app has undergone an absolute ton of testing. Look at the guardrails on the FAQ page https://hypercritical.co/hyperspace/ for an example of some of the extra steps the app takes to keep things safe. Plus you can review all the proposed file changes before you touch anything.
You're not paying for the functionality, but rather the care and safety that goes around it. Personally, I would trust this app over just about any other on the mac.
Best course of action is to not trust John, and just wait for a year of the app out the wild, until everyone else trusts John . I have enough hard drive space in the meantime to not rush into trusting John.
Yeah, the lack of involvement was more in response to ZFS doing this not this app. I could have crossed the streams with other threads about ZFS if it's not directly in this thread
Most EULA’s would disclaim liability for data loss and suggest users keep good backups. I haven’t read a EULA for a long time, but I think most of them do so.
I can't find a specific EULA or disclaimer for the Hyperspace app, but given that the EULA's for major things like Microsoft Office basically say "we offer you no warranty or recourse no matter what this software does" I would hardly expect an indie app to offer anything like that
Dedupe seemed more interesting when storage was expensive, but nowadays it feels like the overhead you get from running dedupe, in most cases, is priced-in. At least with software like CommVault for backups, dedupe requires beefy hardware and low-latency SSDs for the database, If there is even a few extra milliseconds of latency or the server can’t handle requests fast enough, your backup throughput absolutely tanks. Depending on your data though you could see some ridiculous savings here that make it worth the trouble.
I’ve heard many horror stories of dedupe related corruption or restoration woes though, especially after a ransomware attack.
Windows Server does this for NTFS and ReFS volumes. I used it quite a bit on ReFS w/ Hyper-V VMs and it worked wonders. Cut my storage usage down by ~45% with a majority of Windows Server VMs running a mix of 2016/2019 at the time.
Yep. At a previous job we had a file server that we published Windows build output to.
There were about 1000 copies of the same pre-requisite .NET and VC++ runtimes (each build had one) and we only paid for the cost of storing it once. It was great.
It is worth pointing out though, that on Windows Server this deduplication is a background process; When new duplicate files are created, they genuinely are duplicates and take up extra space, but once in a while the background process comes along and "reclaims" them, much like the Hyperspace app here does.
Because of this (the background sweep process is expensive), it doesn't run all the time and you have to tell it which directories to scan.
If you want "real" de-duplication, where a duplicate file will never get written in the first place, then you need something like ZFS
Both ZFS and WinSvr offer "real" dedupe. One is on-write, which requires a significant amount of available memory, the other is on a defined schedule, which uses considerably less memory (300MB + 10MB/TB).
ZFS is great if you believe you'll exceed some threshold of space while writing. I don't personally plan my volumes with that in mind but rather make sure I have some amount of excess free space.
WinSvr allows you to disable dedupe if you want (don't know why you would) where as ZFS is a one-way street without exporting the data.
Both have pros and cons. I can live with the WinSvr cons while ZFS cons (memory) would be outside of my budget, or would have been at the particular time with the particular system.
On ZFS it consumes a lot of RAM. In part I think this is because ZFS does it on the block level, and has to keep track of a lot of blocks to compare against when a new one is written out. It might be easier on resources if implemented on the file level. Not sure if the implementation would be simpler or more complex.
It might also be a little unintuitive that modifying one byte of a large file would result in a lot disk activity, as the file system would need to duplicate the file again.
Files are always represented as lists of blocks or block spans within a file system. Individual blocks could in theory be partially shared between files at the complexity cost of a reference counter for each block. So changing a single byte in a copy on write file could take the same time regardless of file size because only the affected bock would have to be duplicated. I don't know at all how MacOS implements this copynon write scheme, though.
APFS is a copy on write filesystem if you use the right APIs, so it does what you describe but only for entire files.
I believe as soon as you change a single bite you get a complete copy that’s your own.
And that’s how this program works. It finds perfect duplicates and then effectively deletes and replaces them with a copy of the existing file so in the background there’s only one copy of the bits on the disk.
I suppose this means that you could find yourself unexpectedly out of disk space in unintuitive ways, if you're only trying to change one byte in a cloned file but there isn't enough space to copy its entire contents?
It doesn't work like you think.
If you change one byte of duplicated file - the only "byte" will be changed on disk (a "byte", because, technically is not a byte, but a block).
As far as I understand, it works like a reflink feature in the modern linux FSs.
If so, thats really cool, and thats also a bit better than the zfs's snapshots.
Iam newbie on macos, but it looks amazing
That’s true as long as the writing application only writes to blocks that have changed. Is a VM tool going to write blocks, or write multi-MB segments of a sparse image that can be swapped atomically? Unfortunately, once a file changes there are no APIs to check which blocks are still shared (at least there weren’t as of macOS 13).
I’m not sure if it works on a file or block level for CoW, but yes.
However APFS gives you a number of space related foot-guns if you want. You can overcommit partitions, for example.
It also means if you have 30 GB of files on disk that could take up anywhere from a few hundred K to 30 GB of actual data depending on how many dupes you have.
It’s a crazy world, but it provides some nice features.
If we only have two files, A and its duplicate B with some changes as a diff, this works pretty well. Even if the user deletes A, the OS could just apply the diff to the file on disk, unlink A, and assign B to that file.
But if we have A and two different diffs B1 and B2, then try to delete A, it gets a little murkier. Either you do the above process and recalculate the diff for B2 to make it a diff of B1; or you keep the original A floating around on disk, not linked to any file.
Similarly, if you try to modify A, you'd need to recalculate the diffs for all the duplicates. Alternatively, you could do version tracking and have the duplicate's diffs be on a specific version of A. Then every file would have a chain of diffs stretching back to the original content of the file. Complex but could be useful.
It's certainly an interesting concept but might be more trouble than it's worth.
ZFS does this by de-duplicating at the block level, not the file level. It means you can do what you want without needing to keep track of a chain of differences between files. Note that de-duplication on ZFS has had issues in the past, so there is definitely a trade-off. A newer version of de-duplication sounds interesting, but I don't have any experience with it: https://www.truenas.com/docs/references/zfsdeduplication/
VAST storage does something like this. Unlike how most storage arrays identify the same block by hash and only store it once VAST uses a content aware hash so hashes of similar blocks are also similar. They store a reference block for each unique hash and then when new data comes in and is hashed the most similar block is used to create byte level deltas against. In practice this works extremely well.
ZFS: "The main benefit of deduplication is that, where appropriate, it can greatly reduce the size of a pool and the disk count and cost. For example, if a server stores files with identical blocks, it could store thousands or even millions of copies for almost no extra disk space." (emphasis added)
APFS shares blocks so only blocks that changed are no longer shared. Since a block is the smallest atomic unit (except maybe an inode) in a FS, that’s the best level of granularity to expect.
In regards to the second point, this isn't correct for ZFS: "If several files contain the same pieces (blocks) of data or any other pool data occurs more than once in the pool, ZFS stores just one copy of it. Instead of storing many copies of a book it stores one copy and an arbitrary number of pointers to that one copy." [0]. So changing one byte of a large file will not suddenly result in writing the whole file to disk again.
It reads like that is what they meant: "modifying one byte of a large file would result in a lot disk activity, as the file system would need to duplicate the file again"
Yeah, I did not write it very clearly. On ZFS, you're right. On a file system that applied deduplication to files and not individual blocks, the file would need to be duplicated again, no matter where and what kind of change was made.
Records (which are of variable size) are already checksummed, and there were checksum-hashes which made it vanishingly unlikely that one could choose two different records with the same (optionally cryptographically strong) checksum-hash. When a newly-created record's checksum is generated, one could look into a table of existing checksum-hashes and avoid the record write if it already exists, substituting an incremented refcount for that table entry.
ZFS is essentially an object store database at one layer; the checksum-hash deduplication table is an object like any other (file, metadata, bookmarks, ...). There is one deduplication table per pool, shared among all its datasets/volumes.
On reads, one does not have to consult the dedup table.
The mechanism was fairly easy to add. And for highly-deduplicatable data that is streaming-write-once-into-quiescent-pool-and-never-modify-or-delete-what's-written-into-a-deduplicated-dataset-or-volume, it was a reasonable mechanism.
In other applications, the deduplication table would tend to grow and spread out, requiring extra seeks for practically every new write into a deduplicated dataset or volume, even if it's just to increment or decrement the refcount for a record.
Destroying a deduplicated dataset has to decrement all its refcounts (and remove entries from the table where it's the only reference), and if your table cannot all fit in ram, the additional IOPS onto spinning media hurt, often very badly. People experimenting with deduplication and who wanted to back out after running into performance issues for typical workloads sometimes determined it was much MUCH faster to destroy the entire pool and restore from backups, rather than wait for a "zfs destroy" on a set of deduplicated snapshots/datasets/volumes to complete.
I have no specialized knowledge (just a ZFS user for over a decade). I suspect the reason is that in addition to files, ZFS will also allow you to create volumes. These volumes act like block devices, so if you want to dedup them, you need to do it at the block level.
That's a ZFS online dedup limitation. I think xfs and btrfs are better prior art here since they use extent-based deduplication and can do offline dedup which means they don't have have to keep it in memory and the on-disk metadata is smaller too.
If Apple is anything like where I work, there's probably a three-year-old bug ticket in their system about it and no real mandate from upper management to allocate resources for it.
This is commonly done with compression on block storage devices. That fails, of course, if the file system is encrypting the blocks it sends down to the device.
Doing deduplication at this level is nice because you can dedupe across file systems. If you have, say, a thousand systems that all have the same OS files you can save vats of storage. Many times, the only differences will be system specific configurations like host keys and hostnames. No single filesystem could recognize this commonality.
This fails when the deduplication causes you to have fewer replicas of files with intense usage. To take the previous example, if you boot all thousand machines at the same time, you will have a prodigious I/O load on the kernel images.
> There is no way for Hyperspace to cooperate with all other applications and macOS itself to coordinate a “safe” time for those files to be replaced, nor is there a way for Hyperspace for forcibly take exclusive control of those files.
This is the standard API for deduplication on Linux (used for btrfs and XFS); you ask the OS nicely to deduplicate a given set of ranges, and it responds by locking the ranges, verifying that they are indeed identical and only then deduplicates for you (you get a field back saying how many bytes were deduplicated from each range). So there's no way a userspace program can mess up your files.
Probably because APFS runs on everything from the Apple Watch to the Mac Pro and everything in between.
You probably don’t want your phone or watch de-duping stuff.
There are tons of knobs you can tweak in macOS but Apple has always been pretty conservative when it comes to what should be default behavior for the vast majority of their users.
Certainly when you duplicate a file using the Finder or use cp -c at the command line, the Copy-on-Write functionality is being used; most users don’t need to know that.
The cost is because of the fact people won't use it regularly. The developer is offering life time unlocks, lower cost levels for shorter timeframes etc.
I have yet to see a GUI variant of deduplication software for Linux. There are plenty of command line tools, which probably can be ported to macOS, but there's no user friendly tool to just click through as far as I know.
There's value in convenience. I wouldn't pay for a yearly license (that price seems more than fair for a "version lifetime" price to me?) but seeing as this tool will probably need constant maintenance as Apple tweaks and changes APFS over time, combined with the mandatory Apple taxes for publishing software like this, it's not too awful.
The Mac App Store (and all of Apple's App Stores) doesn't enable this sort of licensing. It's exactly the sort of thing that drives a lot of developers to independent distribution.
That's why we see so many more subscription-based apps these days, application development is an ongoing process with ongoing costs, so it needs to have ongoing income. But the traditional buy-it-once app pricing doesn't enable that long-term development and support. The app store supports subscriptions though, so now we get way more subscription-based apps.
I really think Siracusa came up with a clever pricing scheme here, given his want to use the app store for distribution.
I did this with two scripts - one that produces and cached sha1 sums of files, and another that consumes the output of the first (or any of the *sum progs) and produces stats about duplicate files, with options to delete or hard-link them.
if file is not going to be modified (in the low-level sense - open("w") on the filename; as opposed to rename-and-create-new), then reflinks (what this app does) and hardlinks act somewhat identically.
For example if you have multiple node_modules, or app installs, or source photos/videos (ones you don't edit), or music archives, then hardlinks work just fine.
It is not a hard link. A clone is an independent file which is backed by the same storage. So far mostly the same as a hard link you’ll say. However if you modify a clone, it will be “uncloned” and will be modified independently of its clones.
I wrote a similar (but simpler) script which would replace a file by a hardlink if it has the same content.
My main motivation was for the packages of Python virtual envs, where I often have similar packages installed, and even if versions are different, many files would still match. Some of the packages are quite huge, e.g. Numpy, PyTorch, TensorFlow, etc. I got quite some disk space savings from this.
If you'd like. In the blog post he says he wrote the prototype in an afternoon. Hyperspace does try hard to preserve unique metadata as well as other protections.
> Like all my apps, Hyperspace is a bit difficult to explain. I’ve attempted to do so, at length, in the Hyperspace documentation. I hope it makes enough sense to enough people that it will be a useful addition to the Mac ecosystem.
Am I missing something, or isn't it a "file de-duplicator" with a nice UI/UX? Sounds pretty simple to describe, and tells you why it's useful with just two words.
No because it isn't getting rid of the duplicate, it's using a feature of APFS that allows for duplicates to exist separately but share the same internal data.
My understanding is that it is a copy-on-write clone, not a hard link. [1]
> Q: Are clone files the same thing as symbolic links or hard links?
> A: No. Symbolic links ("symlinks") and hard links are ways to make two entries in the file system that share the same data. This might sound like the same thing as the space-saving clones used by Hyperspace, but there’s one important difference. With symlinks and hard links, a change to one of the files affects all the files.
> The space-saving clones made by Hyperspace are different. Changes to one clone file do not affect other files. Cloned files should look and behave exactly the same as they did before they were converted into clones.
What kind of changes could you make to one clone that would still qualify it as a clone? If there are changes, it's no longer the same file. Even after reading the How It Works[0] link, I'm not groking how it works. Is it making some sort of delta/diff that is applied to the original file? That's not possible for every file format like large media files. I could see that being interesting for text based files, but that gets complicated for complex files.
If I understand correctly, a COW clone references the same contents (just like a hardlink) as long as all the filesystem references are pointing to identical file contents.
Once you open one of the reference handles and modify the contents, the copy-on-write process is invoked by the filesystem, and the underlying data is copied into a new, separate file with your new changes, breaking the link.
Comparing with a hardlink, there is no copy-on-write, so any changes made to the contents when editing the file opened from one reference would also show up if you open the other hardlinks to the same file contents.
With a hard link, the content of each of the two 'files' are identical in perpetuity.
With APFS Clones, the contents start off identical, but can be changed independently. If you change a small part of a file, those block(s) will need to be created, but the existing blocks will continue to be shared with the clone.
Almost, but the difference is that if you change one of hardlinked files, you change "all of them". (It's really the same file but with different paths.)
It’s not the same because clones can have separate meta data; in addition, if a cloned file changes, it stores a diff of the changes from the original.
Right, but the concept is the same, "remove duplicates" in order to save storage space. If it's using reflinks, softlinks, APFS clones or whatever is more or less an implementation detail.
I know that internally it isn't actually "removing" anything, and that it uses fancy new technology from Apple. But in order to explain the project to strangers, I think my tagline gets the point across pretty well.
> Right, but the concept is the same, "remove duplicates" in order to save storage space.
The duplicates aren't removed, though. Nothing changes from the POV of users or software that use those files, and you can continue to make changes to them independently.
De-duplication does not mean the duplicates completely disappear. If I download a deduplication utility I expect it to create some sort of soft/hard link. I definitely don’t want it to completely remove random files on the filesystem, that’s just going to wreak havoc.
But it can still wreak havoc if you use hardlinks or softlinks, because maybe there was a good reason for having a duplicate file! Imagine you have a photo “foo.jpg.” You make a copy of it “foo2.jpg” You’re planning on editing that file, but right now, it’s a duplicate. At this point you run your “deduper” that turns the second file into a hardlink. Then a few days later you go and edit the file, but wait, the original “backup” file is now modified too! You lost your original.
That’s why Copy-on-write clones are completely different than hardlinks.
Replacing duplicates with hard links would be extremely dangerous. Software which expects to be able to modify file A without modifying previously-identical file B would break.
The author of the software is a file system enthusiast (so much that in the podcast he's a part of they have a dedicated sound effect every time "filesystem" comes up), a long time blogger and macOS reviewer. So you'll have to see it in that context while documenting every bit and the technical details behind it is important to him...even if it's longer than a tag line on a landing page.
In times where documentation is often an afterthought, and technical details get hidden away from users all the time ("Ooops some error occurred") this should be celebrated.
Judging by this sub-thread, the process really is harder to explain that it appears on the surface. The basic idea is simple but the implementation requires deeper knowledge.
But why would you discuss the implementation to end-users who probably wouldn't even understand what "implementation" means? The discussions you see in the subthread is not a discussion that would appear on less-technical forums, and I wouldn't draw any broader conclusions based on HN conversations in general.
Because the implementation leaks to the user experience. The user at least needs to know whether after running the utility, the duplicate files will be gone, or whether changing one of the files will change the other.
Symbolic links, hard links, ref links are all part of the file system interface, not the implementation.
If you replace `sh:link` with `sh:clone` instead, it will.
> clone: reflink-capable filesystems only. Try to clone both files with the FIDEDUPERANGE ioctl(3p) (or BTRFS_IOC_FILE_EXTENT_SAME on older kernels). This will free up duplicate extents while preserving the metadata of both. Needs at least kernel 4.2.
Would be nice if git could make use of this on macOS.
Each worktree I usually work on is several gigs of (mostly) identical files.
Unfortunately the source files are often deep in a compressed git pack file, so you can't de-duplicate that.
(Of course, the bigger problem is the build artefacts on each branch, which are like 12G per debug/release per product, but they often diverge for boring reasons.)
Git is a really poor fit for a project like that since it's snapshot based instead of diff based... Luckily, `git lfs` exists for working around that, I'm assuming you've already investigated that for the large artifacts?
"git worktree" shares a .git folder between multiple checkouts. You'll still have multiple files in working copy, but at least the .pack files would be shared. It is great feature, very robust, I use it all the time.
There is also ".git/objects/info/alternates", accessed via "--shared"/"--reference" option of "git clone", that allows only sharing of object storage and not branches etc... but it is has caveats, and I've only used it in some special circumstances.
Git de-duplicates everything in its store (in the .git directory) already. That's how it can store thousands of commits which are snapshots of the entire repository without eating up tons of disk space. Why do you have duplicated files in the working directory, though?
Many comments here offering similar solutions based on hardlinks or symlinks.
This uses a specific feature of APFS that allows the creation of copy-on-write clones. [1] If a clone is written to, then it is copied on demand and the original file is unmodified. This is distinct from the behavior of hardlinks or symlinks.
502 comments
[ 3.1 ms ] story [ 313 ms ] threadThere's a place for alias file pointers, but lying to the user and pretending like an alias is a copy is bound to lead to unintended and confusing results
Update: whoops, missed it in your comment. Block (changed bytes) level.
I wish it were more obvious how to do it with other software. Often there's a learning curve in the way before you can see the value.
After all how many perfect duplicate files do you probably create a month accidentally?
There’s a subscription or buy forever option for people who think that would actually be quite useful to them. But for a ton of people a one time IAP that gives them a limited amount of time to use the program really does make a lot of sense.
And you can always rerun it for free to see if you have enough stuff worth paying for again.
however has anyone been able to find out from the website how much the license actually costs?
For example, I discovered my time machine backup kicked out the oldest versions of files I didn't know it had a record of and thought I'd long since lost, but it destroyed the names of the directories and obfuscated the contents somewhat. Thousands of numerically named directories, some of which have files I may want to hang onto, but don't know whether I already have them or not, or where they are since it's completely unstructured. Likewise, many of them may just have one of the same build system text file I can obvs toss away.
I probably wouldn't have risked it, either.
< You can also create [APFS (copy on write) clones] in Terminal using the command `cp -c oldfilename newfilename` where the c option requires cloning rather than a regular copy.
`fclones dedupe` uses the same command[1]:
[1] https://github.com/pkolaczk/fclones/blob/555cde08fde4e700b25...But then I ran this command and saved over 20GB:
I've used fclones before in the default mode (create hard links) but this is the first time I've run it at the top level of my home folder, in dedupe mode (i.e. using APFS clones). Fingers crossed it didn't wreck anything.This tool to enable compression is free and open source
https://github.com/RJVB/afsctool
Also note about APFS vs HFS+, if you use HDD e.g. as backup media for Time Machine, HFS+ is must have over APFS as it is optimised only for SSD (random access).
https://bombich.com/blog/2019/09/12/analysis-apfs-enumeratio...
https://larryjordan.com/blog/apfs-is-not-yet-ready-for-tradi...
Not so smart Time Machine setup utility forcefully re-creates APFS on a HDD media, so you have to manually create HFS+ volume (e.g. with Disk Utily) and then use terminal command to add this volume as TM destination
`sudo tmutil setdestination /Volumes/TM07T`
If we are to believe ChatGPT itself: "The ChatGPT macOS desktop app is built using Electron, which means it is primarily written in JavaScript, HTML, and CSS"
https://github.com/tarkah/iced_table is a third-party widget for tables, but you can roll out your own or use other alternatives too
It's in Rust, not Swift, but I think switching from the latter to the former is easier than when moving away from many other popular languages.
I'd argue there's a lot more to iced than just being a quick toolkit. the Elm Architecture really shines for GUI apps
That being said, it's not quite an apples to apples comparison, because SwiftUI or UIKit can work with basically an infinite number of rows, whereas HTML will eventually get to a point where it won't load.
Like with Hyperspace, you would need to use a tool that can identify which files are duplicates, and then convert them into reflinks.
When you asked about the filesystem, I assumed you were asking about which filesystem feature was being used, since hyperspace itself is not provided by the filesystem.
Someone else mentioned[0] fclones, which can do this task of finding and replacing duplicates with reflinks on more than just macOS, if you were looking for a userspace tool.
[0]: https://news.ycombinator.com/item?id=43173713
You only get CoW on APFS if you copy a file with certain APIs or tools.
If you have a program that does it manually, you copied a duplicate to somewhere on your desk from some other source, or your files already existed on the file system when you converted to APFS because you’ve been carrying them for a long time then you’d have duplicates.
APFS doesn’t look for duplicates at any point. It just keeps track of those that it knows are duplicates because of copy operations.
Edit: Let’s be real here, Tim Cook is keeping the lights on. He isn’t a product guy. They lack leadership and vision at present and are committed to a foolish release cycle based on calendar year not quality of product. These wouldn’t have come to pass had Steve lived till today. Yes its opinion but I doubt its an unpopular one.
So I’ll say it again succinctly, to answer “what happened?”
Steve died.
Every time I need to purchase something via Apple, it becomes a quest. Enter password, validate card, welcome to endless login loop. Reboot. Click purchase, enter password, confirm OTP on another device, then nothing happens, purchase button is active, clicks ignored. Reboot. Click "Get", program begins downloading, wait 30s, app cannot be download, go to Settings to verify account. Sure. Account is perfectly fine in Settings. Reboot. Click "Get". Finally program installed. Click in-app purchase. Enter password again. Choose Apple Pay. Proceed with purchase. You need to verify your account. Account is fine in Settings. Reboot. Click purchase. Cannot be completed at this time. Wait couple of hours, try again. Purchase successful.
All. The. Time. For years. On almost all of the devices which I upgrade annually.
Again, I don't think you're doing anything wrong, and I don't doubt your experience. But I really think something's fundamentally wrong somewhere, because what you're dealing with is not normal. It's not the common experience others are tolerating.
I can't complain though because I have this account for like couple decades and loosing that account would be painful. Apple did ban my account twice on the grounds that it's a US account while I'm not physically located there. I was able to revert the ban by explaining that I've got a US legal entity (account, banking card, etc) and thus I beg to continue using it. Not taking chances for the third time, so I silently endure.
I don't believe that to be true, I have been having issues with Apple bugs for the last 7-8 years. Totally unnecessary friction due to features I do not want and do not use.
I'm going to have to clone the drive, then use terminal to chmod the /usr dirs to extract the files I want (mostly personal music production).
I immediately ordered a Mac mini, but since I didn't want a 256GB drive and 16GB RAM, I'm still waiting for it to arrive from China.
Also, the M1 MB Pro was the most expensive* and worst computer I have ever owned. I wish I had just bought an air. No tactile volume controls. As a musician that is the worst.
(*company I worked for a while in school paid $12k for a Mac 2 FX. Lol.)
I then tried again including my user home folder (731K files, 127K folders, 2755 eligible files) to hopefully catch more savings and I only ended up at 1.3GB of savings (300MB more than just what was in the NodeJS folders.)
I tried to scan System and Library but it refused to do so because of permission issues.
I think the fact that I use pnpm for my package manager has made my disk space usage already pretty near optimal.
Oh well. Neat idea. But the current price is too high to justify this. Also I would want it as a background process that runs once a month or something.
You "only" found that 12% of the space you are using is wasted? Am I reading this right?
Ex - On my non-apple machines, 8GB is trivial. I load them up with the astoundingly cheap NVMe drives in the multiple terabyte range (2TB for ~$100, 4TB for ~$250) and I have a cheap NAS.
So that "big win" is roughly 40 cents of hardware costs on the direct laptop hardware. Hardly worth the time and effort involved, even if the risk is zero (and I don't trust it to be zero).
If it's just "storage" and I don't need it fast (the perfect case for this type of optimization) I throw it on my NAS where it's cheaper still... Ex - it's not 40 cents saved, it's ~10.
---
At least for me, 8GB is no longer much of a win. It's a rounding error on the last LLM model I downloaded.
And I'd suggest that basically anyone who has the ability to not buy extortionately priced drives soldered onto a mainboard is not really winning much here either.
I picked up a quarter off the ground on my walk last night. That's a bigger win.
You do realize that this software is only available on macOS, and only works because of Apple's APFS filesystem? You're essentially complaining that medicine is only a win for people who are sick.
There are lots of other file systems that support this kind of deduplication...
Like ZFS that the author of the software explicitly mentions in his write up https://www.truenas.com/docs/references/zfsdeduplication/
Or Btrfs ex: https://kb.synology.com/en-id/DSM/help/DSM/StorageManager/vo...
Or hell, even NTFS: https://learn.microsoft.com/en-us/windows-server/storage/dat...
This is NOT a novel or new feature in filesystems... Basically any CoW file system will do it, and lots of other filesystems have hacks built on top to support this kinds of feature.
---
My point is that "people are only sick" because the company is pricing storage outrageously. Not that Apple is the only offender in this space - but man are they the most egregious.
If you read the rest of the comment he only saved another 30% running his entire user home directory through it.
So this is not a linear trend based on space used.
When I run it on my home folder (Roughly 500GB of data) I find 124 MB of duplicated files.
At this stage I'd like it to tell me what those files are - The dupes are probably dumb ones that I can simply go delete by hand, but I can understand why he'd want people to pay up first, as by simply telling me what the dupes are he's proved the app's value :-)
For reference, from the comment they’re talking about:
> I then tried again including my user home folder (731K files, 127K folders, 2755 eligible files) to hopefully catch more savings and I only ended up at 1.3GB of savings (300MB more than just what was in the NodeJS folders.)
You misunderstood my comment. I ran it on my home folder which contains 165GB of data and it found 1.3GB is savings. That isn't significant for me to care about because I currently have 225GB free of my 512GB drive.
BTW I highly recommend the free "disk-inventory-x" utility for MacOS space management.
You wrote: but it only found 1GB of savings on a 8.1GB folder.
It’s quite a saving and that’s what everyone understood from your comment.
His comment is pretty understandable if you've done frontend work in javascript.
Node_modules is so ripe for duplicate content that some tools explicitly call out that they're disk efficient (It's literally in the tagline for PNPM "Fast, disk space efficient package manager": https://github.com/pnpm/pnpm)
So he got ok results (~13% savings) on possibly the best target content available in a user's home directory.
Then he got results so bad it's utterly not worth doing on the rest (0.10% - not 10%, literally 1/10 of a single percent).
---
Deduplication isn't super simple, isn't always obviously better, and can require other system resources in unexpected ways (ex - lots of CPU and RAM). It's a cool tech to fiddle with on a NAS, and I'm generally a fan of modern CoW filesystems (incl APFS).
But I want to be really clear - this is people picking spare change out of the couch style savings. Penny wise, pound foolish. The only people who are likely to actually save anything buying this app probably already know it, and have a large set of real options available. Everyone else is falling into the "download more ram" trap.
When I ran it on my home folder with 165GB of data it only found 1.3GB of savings. This isn't that significant to me and it isn't really worth paying for.
BTW I highly recommend the free "disk-inventory-x" utility for MacOS space management.
True
> and dedupes automatically
Also true.
But the way you put them after each other, makes it sound like npm does de-duplication, and since pnpm tries to be a drop-in replacement for npm, so does pnpm.
So for clarification: npm doesn't do de-duplication across all your projects, and that in particular was of the more useful features that pnpm brought to the ecosystem when it first arrived.
It’s a free app because you don’t have to buy it to run it. It will tell you how much space it can save you for free. So you don’t have to waste $20 to find out it only would’ve been 2kb.
But that means the parts you actually have to buy are in app purchases, which are always hidden on the store pages.
macOS has a sealed volume which is why you're seeing permission errors.
https://support.apple.com/guide/security/signed-system-volum...
In general, you don’t want to mess with that.
It tries, but there are some things it can't perfectly preserve like the last access time. Instances where it can't duplicate certain types of extended attributes or ownership permissions it will not perform the operation.
https://podcasts.apple.com/podcast/id617416468?i=10006919599...
No word about alternate data streams. I'll pass for now.. Although it's nice to see how much duplicates you have
Q: Does Hyperspace preserve file metadata during reclamation?
A: When Hyperspace replaces a file with a space-saving clone, it attempts to preserve all metadata associated with that file. This includes the creation date, modification date, permissions, ownership, Finder labels, Finder comments, whether or not the file name extension is visible, and even resource forks. If the attempt to preserve any of these piece of metadata fails, then the file is not replaced.
If you find some piece of file metadata that is not preserved, please let us know.
Q: How does Hyperspace handle resource forks?
A: Hyperspace considers the contents of a file’s resource fork to be part of the file’s data. Two files are considered identical only if their data and resource forks are identical to each other.
When a file is replaced by a space-saving clone during reclamation, its resource fork is preserved.
This got me wondering why the filesystem itself doesn't run a similar kind of deduplication process in the background. Presumably, it is at a level of abstraction where it could safely manage these concerns. What could be the downsides of having this happen automatically within APFS?
I still do not trust de-duplication software.
I've done the entire compare every file via hashing and then log each of the matches for humans to compare, but never has any of that ever been allowed to mv/rm/link -s anything. I feel my imposter syndrome in this regard is not a bad thing.
Any halfway-competent developer can write some code that does a SHA256 hash of all your files and uses the Apple filesystem API's to replace duplicates with shared-clones. I know swift, I could probably do it in an hour or two. Should you trust my bodgy quick script? Heck no.
The author - John Siracusa - has been a professional programmer for decades and is an exceedingly meticulous kind of person. I've been listening to the ATP podcast where they've talked about it, and the app has undergone an absolute ton of testing. Look at the guardrails on the FAQ page https://hypercritical.co/hyperspace/ for an example of some of the extra steps the app takes to keep things safe. Plus you can review all the proposed file changes before you touch anything.
You're not paying for the functionality, but rather the care and safety that goes around it. Personally, I would trust this app over just about any other on the mac.
You are involved. You see the list of duplicates and can review them as carefully as you'd like before hitting the button to write the changes.
I’ve heard many horror stories of dedupe related corruption or restoration woes though, especially after a ransomware attack.
I've been writing a similar thing to dedupe my photo collection and I'm so paranoid of pulling the trigger I just keep writing more tests.
I think that ZFS actually does this. https://www.truenas.com/docs/references/zfsdeduplication/
EDIT: Is this referring to the "fast" dedup feature?
There were about 1000 copies of the same pre-requisite .NET and VC++ runtimes (each build had one) and we only paid for the cost of storing it once. It was great.
It is worth pointing out though, that on Windows Server this deduplication is a background process; When new duplicate files are created, they genuinely are duplicates and take up extra space, but once in a while the background process comes along and "reclaims" them, much like the Hyperspace app here does.
Because of this (the background sweep process is expensive), it doesn't run all the time and you have to tell it which directories to scan.
If you want "real" de-duplication, where a duplicate file will never get written in the first place, then you need something like ZFS
(not really, since it's not fragmentation, but conceptually similar)
ZFS is great if you believe you'll exceed some threshold of space while writing. I don't personally plan my volumes with that in mind but rather make sure I have some amount of excess free space.
WinSvr allows you to disable dedupe if you want (don't know why you would) where as ZFS is a one-way street without exporting the data.
Both have pros and cons. I can live with the WinSvr cons while ZFS cons (memory) would be outside of my budget, or would have been at the particular time with the particular system.
It might also be a little unintuitive that modifying one byte of a large file would result in a lot disk activity, as the file system would need to duplicate the file again.
I believe as soon as you change a single bite you get a complete copy that’s your own.
And that’s how this program works. It finds perfect duplicates and then effectively deletes and replaces them with a copy of the existing file so in the background there’s only one copy of the bits on the disk.
As far as I understand, it works like a reflink feature in the modern linux FSs. If so, thats really cool, and thats also a bit better than the zfs's snapshots. Iam newbie on macos, but it looks amazing
However APFS gives you a number of space related foot-guns if you want. You can overcommit partitions, for example.
It also means if you have 30 GB of files on disk that could take up anywhere from a few hundred K to 30 GB of actual data depending on how many dupes you have.
It’s a crazy world, but it provides some nice features.
I think it stores a delta:
https://en.m.wikipedia.org/wiki/Apple_File_System#Clones
https://en.wikipedia.org/wiki/Apple_File_System?wprov=sfti1#...
If we only have two files, A and its duplicate B with some changes as a diff, this works pretty well. Even if the user deletes A, the OS could just apply the diff to the file on disk, unlink A, and assign B to that file.
But if we have A and two different diffs B1 and B2, then try to delete A, it gets a little murkier. Either you do the above process and recalculate the diff for B2 to make it a diff of B1; or you keep the original A floating around on disk, not linked to any file.
Similarly, if you try to modify A, you'd need to recalculate the diffs for all the duplicates. Alternatively, you could do version tracking and have the duplicate's diffs be on a specific version of A. Then every file would have a chain of diffs stretching back to the original content of the file. Complex but could be useful.
It's certainly an interesting concept but might be more trouble than it's worth.
https://www.vastdata.com/blog/breaking-data-reduction-trade-...
Identifying similar blocks and, maybe sub-rechunking isn’t something I’ve ever considered.
https://www.truenas.com/docs/references/zfsdeduplication/
https://www.microsoft.com/en-us/download/details.aspx?id=397...
[0] https://www.truenas.com/docs/references/zfsdeduplication/
Of course, that is true of most filesystems.
ZFS is essentially an object store database at one layer; the checksum-hash deduplication table is an object like any other (file, metadata, bookmarks, ...). There is one deduplication table per pool, shared among all its datasets/volumes.
On reads, one does not have to consult the dedup table.
The mechanism was fairly easy to add. And for highly-deduplicatable data that is streaming-write-once-into-quiescent-pool-and-never-modify-or-delete-what's-written-into-a-deduplicated-dataset-or-volume, it was a reasonable mechanism.
In other applications, the deduplication table would tend to grow and spread out, requiring extra seeks for practically every new write into a deduplicated dataset or volume, even if it's just to increment or decrement the refcount for a record.
Destroying a deduplicated dataset has to decrement all its refcounts (and remove entries from the table where it's the only reference), and if your table cannot all fit in ram, the additional IOPS onto spinning media hurt, often very badly. People experimenting with deduplication and who wanted to back out after running into performance issues for typical workloads sometimes determined it was much MUCH faster to destroy the entire pool and restore from backups, rather than wait for a "zfs destroy" on a set of deduplicated snapshots/datasets/volumes to complete.
Doing deduplication at this level is nice because you can dedupe across file systems. If you have, say, a thousand systems that all have the same OS files you can save vats of storage. Many times, the only differences will be system specific configurations like host keys and hostnames. No single filesystem could recognize this commonality.
This fails when the deduplication causes you to have fewer replicas of files with intense usage. To take the previous example, if you boot all thousand machines at the same time, you will have a prodigious I/O load on the kernel images.
> There is no way for Hyperspace to cooperate with all other applications and macOS itself to coordinate a “safe” time for those files to be replaced, nor is there a way for Hyperspace for forcibly take exclusive control of those files.
Maybe not solving the same problem.
You probably don’t want your phone or watch de-duping stuff.
There are tons of knobs you can tweak in macOS but Apple has always been pretty conservative when it comes to what should be default behavior for the vast majority of their users.
Certainly when you duplicate a file using the Finder or use cp -c at the command line, the Copy-on-Write functionality is being used; most users don’t need to know that.
However, considering Apple will never ever ever allow user replaceable storage on a laptop, this might be worth it.
There's value in convenience. I wouldn't pay for a yearly license (that price seems more than fair for a "version lifetime" price to me?) but seeing as this tool will probably need constant maintenance as Apple tweaks and changes APFS over time, combined with the mandatory Apple taxes for publishing software like this, it's not too awful.
Which really means up until the dev gets bored, which can be as short as 18 months.
I wouldn't mind something like this versioned to OS. 20$ for the current OS, and ten dollars for every significant update.
That's why we see so many more subscription-based apps these days, application development is an ongoing process with ongoing costs, so it needs to have ongoing income. But the traditional buy-it-once app pricing doesn't enable that long-term development and support. The app store supports subscriptions though, so now we get way more subscription-based apps.
I really think Siracusa came up with a clever pricing scheme here, given his want to use the app store for distribution.
For example if you have multiple node_modules, or app installs, or source photos/videos (ones you don't edit), or music archives, then hardlinks work just fine.
While it's not very user-friendly software (command line, after all), duperemove has proven quite useful for my filesystems.
My main motivation was for the packages of Python virtual envs, where I often have similar packages installed, and even if versions are different, many files would still match. Some of the packages are quite huge, e.g. Numpy, PyTorch, TensorFlow, etc. I got quite some disk space savings from this.
https://github.com/albertz/system-tools/blob/master/bin/merg...
[1] https://en.wikipedia.org/wiki/Apple_File_System#Clones
Am I missing something, or isn't it a "file de-duplicator" with a nice UI/UX? Sounds pretty simple to describe, and tells you why it's useful with just two words.
> Q: Are clone files the same thing as symbolic links or hard links?
> A: No. Symbolic links ("symlinks") and hard links are ways to make two entries in the file system that share the same data. This might sound like the same thing as the space-saving clones used by Hyperspace, but there’s one important difference. With symlinks and hard links, a change to one of the files affects all the files.
> The space-saving clones made by Hyperspace are different. Changes to one clone file do not affect other files. Cloned files should look and behave exactly the same as they did before they were converted into clones.
[1] https://hypercritical.co/hyperspace/
[0] https://hypercritical.co/hyperspace/#how-it-works
Once you open one of the reference handles and modify the contents, the copy-on-write process is invoked by the filesystem, and the underlying data is copied into a new, separate file with your new changes, breaking the link.
Comparing with a hardlink, there is no copy-on-write, so any changes made to the contents when editing the file opened from one reference would also show up if you open the other hardlinks to the same file contents.
With APFS Clones, the contents start off identical, but can be changed independently. If you change a small part of a file, those block(s) will need to be created, but the existing blocks will continue to be shared with the clone.
https://hypercritical.co/hyperspace/#how-it-works
APFS apparently allows for creating "link files" which when changed, start to diverge.
I know that internally it isn't actually "removing" anything, and that it uses fancy new technology from Apple. But in order to explain the project to strangers, I think my tagline gets the point across pretty well.
The duplicates aren't removed, though. Nothing changes from the POV of users or software that use those files, and you can continue to make changes to them independently.
That’s why Copy-on-write clones are completely different than hardlinks.
In times where documentation is often an afterthought, and technical details get hidden away from users all the time ("Ooops some error occurred") this should be celebrated.
https://arstechnica.com/author/john-siracusa/
https://github.com/pkolaczk/fclones
Symbolic links, hard links, ref links are all part of the file system interface, not the implementation.
will produce a script which, if run, will hardlink duplicates
> clone: reflink-capable filesystems only. Try to clone both files with the FIDEDUPERANGE ioctl(3p) (or BTRFS_IOC_FILE_EXTENT_SAME on older kernels). This will free up duplicate extents while preserving the metadata of both. Needs at least kernel 4.2.
Each worktree I usually work on is several gigs of (mostly) identical files.
Unfortunately the source files are often deep in a compressed git pack file, so you can't de-duplicate that.
(Of course, the bigger problem is the build artefacts on each branch, which are like 12G per debug/release per product, but they often diverge for boring reasons.)
There is also ".git/objects/info/alternates", accessed via "--shared"/"--reference" option of "git clone", that allows only sharing of object storage and not branches etc... but it is has caveats, and I've only used it in some special circumstances.
This uses a specific feature of APFS that allows the creation of copy-on-write clones. [1] If a clone is written to, then it is copied on demand and the original file is unmodified. This is distinct from the behavior of hardlinks or symlinks.
[1] https://en.wikipedia.org/wiki/Apple_File_System#Clones
Sources: https://unix.stackexchange.com/questions/631237/in-linux-whi... https://forums.veeam.com/veeam-backup-replication-f2/openzfs...