hope you're not running -o compress=lz4 , because you are going to be in for a big surprise when you try to pull this emergency lever! you may be shocked to see you don't actually get much space back!
i do wonder how many FS would actually allocate the 8GB if you, for example, opened a file, seeked to 8GB mark, and wrote a character. many file systems support "sparse files"[1]. for example on btrfs, i can run 'dd if=/dev/zero of=example.sparse count=1 seek=2000000' to make a "1GB" file that has just one byte in it. btrfs will only allocate a very small amount in this case, some meta-data to record an "extent", and a page of data.
i was expecting this article to be about a rude-and-crude overprovisioning method[2], but couldn't guess how it was going to work. SSDs notably perform much much better when they have some empty space to make shuffling data around easier. leaving a couple GB for the drive to do whatever can be a colossal performance improvement, versus a full drive, where every operation has to scrounge around to find some free space. i wasn't sure how the author was going to make an empty file that could have this effect. but that's not what was going on here.
> hope you're not running -o compress=lz4 , because you are going to be in for a big surprise when you try to pull this emergency lever! you may be shocked to see you don't actually get much space back!
This is true. If you are replicating this, copy from /dev/urandom rather than using an empty file.
Before reading this, I had presumed that sparse files did not overcommit drive space, but apparently, they do. I don't use them regularly and certainly not to "reserve disk space" but I was surprised that you can make sparse files way larger than available free space on the drive. I had assumed they were simply not initialized, but the FS still required <x> amount of free space in case a block is accessed.
The filesystems where headroom matters are var, tmp and sometimes root. I like this strategy with logfiles because nethack.log.gz.30 was approximately as important as empty space.
Keeping another 8gb on root and tmp seems extreme.
> On Linux servers it can be incredibly difficult for any process to succeed if the disk is full.
You won't feel too clever if you come to grow your LVM volume into the free space and it won't work because there's no free space on the filesystem! :)
(I don't actually know if this would fail or not - but the point is "rm spacer.img" is pretty much guaranteed not to fail).
I've used LVM for this purpose plenty of times. lvm2 at least has not prevented me from extending a full disk. lvm + reserved blocks + a small spacer file are all decent options, even better when used together.
Good for you. That's not how my memory works. If I don't use a command regularly, I don't trust myself to remember it correctly. Even if I did though, that's a multi-step process, compared to the single command needed to remove a file.
Tomato potato. If you use LVM or anything like it to reserve space then in your failure situation you have to extend the lv, partition, and fs before the space becomes available. More work than just rm'ing the file.
I think the ideal ideal is tuning the reserved blocks in your filesystem.
Even a swap file shouldn't matter, since it's still not sparse. The one exception is if you're on a system that dynamically adds and removes swap files - I believe darwin does that, and I think it might be possible to do on Linux(?) but I've not actually seen it done.
Not quite the same situation as described in the article, but it is still possible for the kernel to swap memory in and out of disk even without a swap file/partition. Memory used for storing executable binaries is allowed to be moved out of memory, as a copy of it lives on disk. This means you can still encounter memory thrashing (and thus system unresponsiveness) under low memory situations.
> Memory used for storing executable binaries is allowed to be moved out of memory, as a copy of it lives on disk.
On Linux, this is not necessarily the case, as you can change the file on disk while the executable is running. I don't know if Linux just keeps executable code in memory all the time, or if it is smart enough to detect whether a copy of executable pages still lives on disk.
I suspect the blog author did not understand this (based on the content) - as a Linode user myself, I just had a look at one of my VMs and they install with the regular 5% reserved space (ext4/Debian).
Funny because I have always tune2fs -m1 or tune2fs -m0 because the reserved space was never supposed to scale linearly with hard drive capacities and is not useful to userspace in anyway. Have never had any issues and been doing it for decades in commercial applications. In some cases, where you probably shouldn't be using ext3/4 anyways, we are talking about reclaiming TBs of reserved space.
It's important to note that mkfs doesn't care if you are formatting the root partition or a data volume partition, it will still reserve space for the kernel.
If you try to use a file system to 99% full --- and it doesn't matter whether it is a 10GB file system or a 10TB file system, you will see significant performance penalties as the file system gets badly fragmented. So that's why having a fixed percentage even for massively big disks still makes sense.
Disk space is cheap enough that even 5% of a 14TB disk is really not that much money --- and if you see bad performance, and then have to pay $$$ to switch to an SSD, maybe it would have been much cheaper to use a HDD with a larger free space reserve....
> If you try to use a file system to 99% full --- and it doesn't matter whether it is a 10GB file system or a 10TB file system, you will see significant performance penalties as the file system gets badly fragmented.
Not true, I've checked. I have plenty of Linux ext3 servers running for many years that routinely drop down to 1% free space for extended periods before being cleaned-up, which still have essentially zero fragmentation. You can create plenty of 10MB files on a multi-terabyte volume that has under 1% free space, as that's still tens of gigabytes to work with.
Obviously at some point you'll hit a severe problem and it's best to avoid taking a chance, but a fixed percentage really isn't the best measurement to tell you where that horizon will be.
It's true, but it's more true for some file systems than others. When you write a file larger than the contiguous available space after its starting point, a file system must break the file into "extents" (chunks). The less space available, the smaller the extents tend to be, and the more fragmentation you will impose for continued writes. It's just math.
Different file systems have wildly different strategies and data structures behind this process, however. Some drop to their knees over 92-93%. Some can write to the last byte with reasonable efficiency—but it'll never be as fast as when it was empty. Copy-on-write systems like ZFS tend to do poorly under near-full conditions.
> When you write a file larger than the contiguous available space after its starting point, a file system must break the file
And why would having a 10TB file system force you to write 100GByte files, whereas a 10GB file system would write 100MByte files instead?
Because that's the topic you're responding to... GP said it doesn't matter whether it is a 10GB file system or a 10TB file system, it "gets badly fragmented" when you get to "99% full".
Like most things, it depends on your workload. If all of the files are written all at once when they created (e.g., no slow append workloads) and the files are all the same size, then that's a very "friendly" workload that will be much less likely to suffer fragmentation. But if you are creating and deleting files that have a large range of sizes, and some files grow gradually over time, then free space fragmentation will tend to occur much more quickly.
Had anyone succeed in looking up the motivation behind the reserved space idea in ext filesystems, e.g a commit message from the time this feature was introduced? I've tried but failed miserably, got lost in many different git repositories.
One fun fact I learned though - one of the reasons is "for quota file". Since ext stores quota information in an ordinary file, there could be an issue where user fills up a disk but the information about it isn't written in quota file due to lack of space.
I know about this, but I do think it's not a bad idea doing what he does because the reserved block count is for root and most server processes still run as root. And it's usually them that are causing the disk to fill. Though I suppose this also makes the problem itself more prominent in the first place. I guess if you run into this a lot, stricter monitoring would be a better solution.
The way I found out about it originally was because I was using external storage drives and I was never able to fit as much as I expected :D
Luckily you can easily change this without reformatting.
What servers usually run as root? Some may start as root, but usually drop privileges for the actual server processes quickly, eg. apache, nginx, sshd.
Nothing that actually does the "serving" or accesses data should be running as root.
Is that true for all logfiles? I still have plenty of daemons (by default) writing directly to some file in /var/log eg EXIM, Apache, and the like. Also plenty of system stuff still write to files in that directory. And yes this is a machine that uses systemd.
But those daemons don’t usually have their own log writer processes running as root, do they? Instead, either the log file is accessible by the user the daemon is running as, or the daemon opens the log file as root before dropping privileges for the rest of its operation.
Most vendors (Debian/Ubuntu, RHEL/clones, etc.) add a hook into rsyslog to be a partner with the systemd logger and write out text files next to the journal - they realize that a lot of people dislike dealing with journalctl (I'm one of them) and provide an alternate hook already installed and working for you behind the scenes.
This is for daemons using syslog methodology, not direct writers like apache/nginx/mysql/etc; think more like cron, systemd, chrony, NetworkManager, and so forth. The vendors are not all aligned on what goes where (example: on RHEL, pacemaker/crm write to their own logs buy on openSUSE they're sent to syslog) - the actual results differ slightly from vendor to vendor.
DIY distros like Arch do not implement the rsyslog backend by default, you have to set it up yourself following the wiki - only journalctl is there by default.
My rule of thumb to avoid these issues is that application/server data gets its own dedicated volume that contains nothing else: logs get their own volume, and root its own. It's an especially bad idea for an application to put its data and logs in the same directory where its binaries reside.
That way, even if your log volume or root somehow fills up before monitoring had a chance to react, your service is unaffected. You can even catch issues pre-emptively by keeping log volumes small so that weird behaviour is likely to trigger an alert before anything goes truly wrong.
On cloud instances, it's silly to put anything on the instance root volume (on AWS, I keep them at the default 8 GB; it's never been a problem) when you can just attach an arbitrary number of additional disks. Container systems would use persistent volumes, and with physical servers, you use LVM or equivalent. This solves most disk allocation issues and makes operations easy when you need more space.
It used to be common, before "the cloud", to have many apparently unnecessary partitions in a server install. One for /, one for /var, one for /home, one for swap at the low sector numbers...
The idea is that /var filling up would not make the system unrecoverable.
> The minimum amount of space guaranteed to a dataset and its descendants. When the amount of space used is below this value, the dataset is treated as if it were taking up the amount of space specified by its reservation. Reservations are accounted for in the parent datasets' space used, and count against the parent datasets' quotas and reservations.
ZFS, like ext[34], also has a reserved space allocation for the entire pool to allow you to still do certain operations when it's reporting "0" free space (...like deleting files or snapshots, to free space).
Unlike ext[34], that reservation is not available for root's general use, but it's there.
If you don’t have monitoring to tell you when the disk is more than X% full, then you’re at risk for more failure scenarios than just a full disk (usually trivial to buy time by deleting old logs).
If the problem arises during a migration or other significant event, which it sounds like Marco's did, the alert will usually be triggered just in time to tell you why you're already in a world of pain.
I have an empty leader on my hard drive so that I can recover if I accidentally nuke the front of it with dd while making a live usb. So it's not a bad idea, and it's super effective so far it hasn't been tested, and hopefully I never will need to.
I think that in the past I saw that when creating a file with e.g. ...
dd if=/dev/zero of=deleteme.file bs=1M count=8196
...the "free space" shown by "df" slowly decreased while the file was being created, but then once the operation completed that "free space" magically went back to its original value => the big existing file (full of "0"s) was basically not using any storage.
Is this what you mean?
I just tried to replicate this behaviour but, dammit, I cannot demonstrate that right now as the behaviour so far was the expected one (free storage decreasing when creating the file and sticking to that even after the completion of the operation).
I strongly believe that that's what I saw in the past (when I was preallocating image files to then be used by KVM VMs), but now I'm wondering if I'm imagining things... :P
EDIT: this happened when using ext4 and/or xfs (don't remember) without using any compression.
You have to beware if you're on a filesystem (such as ZFS) that has compression enabled. A file of all zeros compresses quite well, and may not get you the space you need when you remove it.
The disk filled up, and that's one thing you don't want on a Linux server—or a Mac for that matter. When the disk is full nothing good happens.
I had this happen few times on a Mac and every time I was shocked that if disk gets full you cannot even delete a file and the only option is to do a full system reboot. I was also unable to save any open file, even to external disk and suffered minor data loss every time due to that.
What is the proper way of dealing with such issue on macOS? (or other systems, if they behave the same way)
I don't know with Mac, but this is why many Linux distros recommend putting /home is on a separate partition. If it fills, it won't lock up the whole system.
Fun story with this. Ubuntu now has an experimental root-on-zfs feature. I installed it and started playing with some docker containers, trying to compile a certain version of pytorch. Suddenly, my computer crashed. Apparently, my root partition filled because docker installed everything on the same partition as my OS, crashing everything immediately.
I even hacked my MacOS to disable the message. Computers shouldn't nag their owners repeatedly, even if it's in their best interest, unless the computer is about to catch fire.
Wouldn't it be better to take the system's advice and clear off some space rather than playing russian roulette?
I've had the mispleasure of pointing a large video export to the wrong drive and other misdeeds that allowed a drive to fill up. It's not pleasant. A simple reboot sometimes frees up the swap space to allow for more spring cleaning, but I typically just resort to booting into recovery mode and searching the web for the proper terminal command to decrypt/mount the root volume for spring cleaning.
I had this happen few times on a Mac and every time I was shocked that if disk gets full you cannot even delete a file and the only option is to do a full system reboot. I was also unable to save any open file, even to external disk and suffered minor data loss every time due to that.
This just happened to me. I got the best error message I've ever seen. Something akin to "Can not remove file because the disk is full." This wasn't from the Finder, this was command line rm.
On the Mac it's also exacerbated by the fact that swap will use the system drive and can fill up the disk, and can not be stopped. If you have some rogue process consuming RAM, among other things, your disk will suffer until it is full. And, as mentioned, macOS does not behave well with a full disk.
And, even if you've remedied the swap issue (i.e. killed the process), there's no way I know to recover the swap files created without restarting.
Just seems like the design is trouble waiting to happen, and it has happened to me.
When this last happened, somehow it managed to corrupt my external Time Machine volume.
I've been living with this for the past few years. The only remedy is to do a full system reboot. Sometimes I reboot a few times a night.
One way to buy yourself some time is to disable the sleep file. I'm not sure what it's called -- it's a file that MacOS uses to let the computer hibernate when there's no power. It's a few GB, which (like the blog post stated) is a nontrivial amount of freeable space.
> I'm not sure what it's called -- it's a file that MacOS uses to let the computer hibernate when there's no power. It's a few GB, which (like the blog post stated) is a nontrivial amount of freeable space.
Should be /var/vm/sleepimage and the same size as your RAM.
I ran into this on OpenWRT back in the day. It had a similar filesystem behavior where you could not delete from a full FS. The solution was to truncate a file that was at least 1 block big, thus freeing up a few kilobytes. Then you can rm a large file, and then you can resume normal cleanup.
So far, my first stop to temporarily get more disk space was to reduce the size of the swapfile which on a lot of servers seems to be allotted >1x the requirement.
Will be switching to this hack! Perfect illustration of the KISS principle (Keep it simple, stupid).
Back in my early university days the disks always seemed to be full at inconvenient times on the shared Unix systems we used. Some students resorted to "reserving" disk space when available. Which of course made the overall situation even worse.
> he had put aside those two megabytes of memory early in the development cycle. He knew from experience that it was always impossible to cut content down to memory budgets, and that many projects had come close to failing because of it. So now, as a regular practice, he always put aside a nice block of memory to free up when it's really needed.
If true, I hate that story. Think of the better art assets that were needlessly left behind. How is it that said block of memory had never been identified by any profiling?
> Think of the better art assets that were needlessly left behind.
Consider how long it takes to edit or recreate art assets to reduce their size. Depending on the asset, you might be basically starting over from scratch. Rewriting code to reduce its size is likely to be an even worse option, introducing new bugs and possibly running slower to boot. At least smaller, simpler art assets are likely to render faster.
This is also the kind of problem that's more likely to occur later in the schedule, when time is even more scarce. Between these two factors (lack of time and amount of effort required to get art assets which are both decent looking and smaller), I think in practice you're actually more likely to get better quality art assets by having an artificially reduced memory budget from the outset.
1. Deal with possibly multiple issues possibly involving multiple people with the politics that entails resulting in a lot of stress for all involved as any one issue could render it a complete failure.
2. Have extra space you can decide to optimise if you want. You could even have politics and arguments over what to optimise, but if nothing happens it all still works so there is a lot less stress.
Better PMs do this today by having buffer-features they can cut when needed. It'll handle the not-enough-memory issue as well as a meddlesome VP who think you're over-subscribed and wants you to cut to meet your dates.
Also, don't forget you're hearing decades-later retellings of someone else's story. I don't doubt that they trickled this extra space out as changing requirements mandated it, but that they kept from doing so until the team had actually reached a certain level of product-maturity and reclaimed all of their own waste first.
Remember that the PMs goal is to ship. Them blocking some assets but actually shipping is a success. Better 95% of the product than 0%.
There's a difference between "The server is not responding right now. We're loosing customers.", and "Low resources during product development". Actually the latter may be a case of enforcing premature optimization.
So no, it's not the same idea.
I think we are thinking of a different baseline. You are thinking along the lines of "this should run, we can reduce server costs later", I would suggest (if I may) "the app needs to run on any Android device with 2GB RAM". And then you develop a game to run on a 1.5GB RAM phone, expecting that it will eventually fit into 2GB RAM budget.
In my work it is very common to make the memory map a little smaller than it has to be. If you can't ship an initial version in a reduced footprint you will have no hope of shipping future bugfixes.
Many years ago I spent a couple of weeks fixing a firmware bug. The firmware was only a few dozen bytes shy of the EEPROM. I just #ifdef'd out a bunch of features to focus on debugging what was broken, but to get the fix released I had to manually optimize several other parts of the code to get everything to fit in the 2MB or whatever it was.
Would've been nice if someone had reserved some space ahead of time. Maybe they did, but nobody was around who remembered that codebase.
My favorite part of that story is how the initial question about overflow should make it obvious that what they're doing doesn't work, but nobody noticed.
I'd read in 'Apollo: Race To The Moon', Murray and Cox, that the booster engineers had done something similar with their weight budget, something the spacecraft engineers wound up needing. Contingency funds of all sorts are a great thing.
Back in the late eighties a colleague of mine was making a game for the Atari ST and he purposely put in some time wasting code in the game loop so that he had to work against a smaller budget which gave him some contingency for later on when he needed some extra cycles.
having an 8gb file you know you can delete isn't really all that helpful if everything has already gone disk-full-fracked. you should really have an alarm on free space, especially if you're an indie.
Sure, but sometimes the disk filling up is caused by something runaway and fast. If your "60% full" alarm goes off and the disk fills up 2 minutes later, you're still stuck.
With a "ballast file" (as another commenter termed it), you can decide exactly when processes get to start consuming disk again, and that can give you some headroom to fix the problem.
This is an old trick for when you need to deploy to media with a fixed size - floppy/CD-ROM/etc. Make a file that is 5-10% the size of your media and don't remove unless you're running out of space in crunch time.
This is like carrying around a pound of beef because you refuse to look up the address of a McDonald's 7 minutes away.
Setup quotas or implement some damn monitoring -- if you're not monitoring something as simple and critical as disk usage, what else are you not monitoring?
Not all environments require a stringent SLA. I have some servers that don't have a stringent SLA and aren't worth being woken up at night over if their disk is filling up fast.
It allows me to remove that big file, then I'm able to run sudo since I don't allow root ssh and sudo won't work with a full disk, then I can clear up space on the system, bring it up again, then update log rotate or do whatever to prevent that case from happening again.
That sounds a lot more complicated (and time consuming) than just having monitoring in place, realizing the disk is filling up and fixing it before it leads to downtime.
Monitoring is in place and usually it is caught in time. Downtime is acceptable in this environment, I don't think its worth being woken up in the middle of the night when it can just be resolved in the morning.
Monitoring doesn't prevent random things from spiking, and something like this makes it easier to recover.
Quotas are tricky to set up when things are sharing disk space, and that could easily give you a false positive where a service unnecessarily runs out of space.
When I worked at SevOne, we had 10x500 MB files on each disk that were called ballast files. They served the same purpose, but there were a couple nice tools built in to make sure they got repopulated when disk space was under control, plus alerting you whenever one got "blown." IIRC it could also blow ballast automatically in later versions, but I don't remember it being turned on by default.
As hacks go, it's a good one. I also like it because you don't have to be root to implement it and you don't have to reconfigure your file system params in ways that might or might not be great for other reasons.
But the idea isn't that bad. Name it properly, like saudi-arabia-customer-data.sql.pgp next to a directory named pgp-keys and fill it with /dev/random.
An alternative approach here... make sure (all) your filesystems are on top of LVM. This reduces the steps needed to grow your free space. Whether you have a 8gb empty file laying around, or an 8gb block device to attach...LVM will happily take them both as pv's, add them to your vg's, and finally expand your lv's.
Yes LVM can help here. Another approach would be when you create the logical volume to intentionally under allocate. Perhaps only use 80-90% of the physical volume.
If you are using LVM on all of your filesystems, it seems like a bad idea to use a file residing on LVM block device as another PV. And actually I'd be surprised if this was even allowed. Though maybe it is difficult to detect.
You'd effectively send all block changes through LVM twice (once through the file, then through the underlying block device(s))
LVM is just fancy orchestration for the device-mapper subsystem with some headers for setup information.
For block operations it's no different from manual setup of loop-mounted volumes, that also need to travel a couple of layers to hit the backing device.
Though there is an important caveat - LVM is more abstracted, making it easier to mistakenly map a drive onto itself, which may create a spectacular failure (haven't tried).
This points to a much more serious problem. This is 2021 and the technology is from the 90s, with a really poor user experience design. Your car warns you when you're low on fuel, but your server doesn't if you're low on critical resources.
Assuming we're talking about VMs (2021 etc.), for a SME is there any downside to giving 2TB of space to your discs and let dynamic allocation do the work?
Perhaps consolidate/defrag once a year. Even monitoring total usage more often than that is probably not worth the effort - just buy ample cheap storage.
Also, there was a tradition to split drives into OS, DB, DB Logs. That was mostly a rust performance thing and these days is probably just voluntary management overhead.
One VM using excessively more disk space than it's supposed to can potentially cause data corruption in all the other VMs on that system. For just spinning VMs up and down for testing, you probably won't run into that issue, but on a production system, it could potentially cause some massive downtime
Virtual machine disk space (e.g. Xen, Linode, AWS EC2, or similar) does not work this way. Each VM gets a dedicated amount of disk space allocated to it, they don't all share a pool of free space.
Yes they do with the "dyanmic allocation" the parent comment mentions; VMware datastore has 1TB total, you put VMs in with dynamically expanding disks they are sharing the same 1TB of free space and will fill it if they all want their max space at the same time and you've overprovisioned their max space.
And if you haven't overprovisioned their max space, you may as well not be using dynamic allocation and use fixed size disks.
Even then, snapshots will grow forever and fill the space, and then you hope you have a "spacer.img" file you can delete from the datastore, because you can't remove snapshots when the disk is full and you're stuck. It's the same problem, at a lower level.
I see, a VMware feature, thanks for clarifying. I suppose it's a nice idea in theory, but you'd have to be crazy to use that in production, or for any workload that you care about. It would just be a ticking time bomb.
Hyper-V can do that too, and so can you under Linux. It's called thinly-allocated disks, sparse files, or the dm-thin device mapper target. Professional SANs also allow you to overallocate the total size of the iSCSI volumes offered.
Yes, I've seen that time bomb go off on multiple occasions. Never on my watch though.
If you are using less space than the underlying datastore, there's no benefit to dynamic allocation, you may as well give the servers larger fixed disks. If you are thinking that one server might need more than the fixed size for a sudden growth, then you need to be monitoring to deal with that because that will run out of your space. If you are overprovisioning the datastore, you have the same problem at a level lower, and need to be monitoring that and alerting for that instead (as well).
> "just buy ample cheap storage"; "That was mostly a rust performance thing and these days is probably just"
In the UK a 6TB enterprise rust disk is £150 and a 2TB enterprise SSD is £300, it's 6x the price to SSD everything, and take 3x more drive bays so add more for that. And you can never "just" buy more storage than you ever need - apart from the obvious "when you bought it, you thought you were buying enough, because if you thought you needed more you would have bought more", so that amounts to saying "just know the future better", but it can't happen because Parkinson's Law ("work expands so as to fill the time available for its completion") applies to storage, the more there is available, the more things appear to fill it up.
Room for a test restore of the backups in that space. Room for a clone of the database to do some testing. Room for a trial of a new product. Room for a copy of all the installers and packagers for convenience. Room for a massive central logging server there. What do you mean it's full?
Everyone has this kind of alerting set up, but that's not the point. The beauty of this solution is that it's dead simple and will never fail. Alerting can fail or be ignored.
It's the same as old VW beetles which had a reserve gas tank. When you ran out of gas you opened a valve and you could limp to a gas station. Less likely to fail versus a 1950's era gauge that is telling you you're low. Also impossible to ignore it.
It's the same as old VW beetles which had a reserve gas tank. When you ran out of gas you opened a valve and you could limp to a gas station
In scuba diving there used to be "J-valves". When you had 50 bar left in the tank they would cut out. Then you would pull to reenable your air and return to the surface. Unsurprisingly they are no longer popular.
> The beauty of this solution is that it's dead simple and will never fail. Alerting can fail or be ignored.
It's not that straightforward IMO. Would this file be deleted before the space is filled? If so, there is alerting in place, and it assumes there's a way to delete files before space fills up. If this file is deleted after space fills up, how is this different from not having the file, other than making finding files to delete easier? Then what happens after that? If you delete the file and realize there's nothing else to delete, you'd have to solve the problem the same way if you didn't use this method.
ramdisk? It wouldn't be the first time I'd extract a .deb to tmpfs to resolve a temporary issue.
Don't think I've ever encountered a critical issue where "add more swap" would be a serious disaster recovery solution. I've certainly seen situations where swap was nearing 100% full, and although I would have minutes off wall-clock time to formulate a strategy, those minutes have never allowed me to input more than a handful of characters or so.
The 'beauty' artificially chokes your HDD and produces the same problems that you are trying to avoid.. not a sane way to proactively manage your disk usage.
Same was true of most motorcycles until rather recently, though with motorcycles it was rare that there was a fuel gauge at all. A sputtering engine was how you knew it was low. And I believe that like with motorcycles, the "reserve tank" in an old Beetle is really the same tank - there are two hoses located in the tank at different heights.
Linux servers aren't like mass consumer products. It's assumed users know what they're doing and can build and configure what they need on top of it.
> This is 2021 and the technology is from the 90s
I don't see how this is a valid point. Is integrated circuit technology outdated because it was developed in the 60s?
That's a quality problem. Your car can absolutely drop from "alarm" to "empty" in 30 seconds if there's a leak in the tank. We just don't build fuel tanks that don't spontaneously develop leaks, partly because the manufacturer can be held liable.
There's no reason not to have multiple fail-safes. Receiving the alert on a device at 3am would still mean he could free up 8gb immediately and have breathing room to solve the problem. And remember this is for a single admin. Asking such a person to be on call 24-7 all year, vacations, holidays, weekends... Having a quick way to get breathing room can significantly reduce the stress & cognitive load of worrying about such things in your off-time.
Alerting is also a hack really. In 2021 the operating SYSTEM should work as a system - complexly managing it's resources and make intelligent decisions. Ideally OS should dynamically reserve as much resources as needed on it's own.
714 comments
[ 4.5 ms ] story [ 317 ms ] threadi do wonder how many FS would actually allocate the 8GB if you, for example, opened a file, seeked to 8GB mark, and wrote a character. many file systems support "sparse files"[1]. for example on btrfs, i can run 'dd if=/dev/zero of=example.sparse count=1 seek=2000000' to make a "1GB" file that has just one byte in it. btrfs will only allocate a very small amount in this case, some meta-data to record an "extent", and a page of data.
i was expecting this article to be about a rude-and-crude overprovisioning method[2], but couldn't guess how it was going to work. SSDs notably perform much much better when they have some empty space to make shuffling data around easier. leaving a couple GB for the drive to do whatever can be a colossal performance improvement, versus a full drive, where every operation has to scrounge around to find some free space. i wasn't sure how the author was going to make an empty file that could have this effect. but that's not what was going on here.
[1] https://wiki.archlinux.org/index.php/sparse_file
[2] https://superuser.com/questions/944913/over-provisioning-an-...
This is true. If you are replicating this, copy from /dev/urandom rather than using an empty file.
Also I keep databases on their own partition so that nothing else can accidentally fill up the space and lead to data loss.
> On Linux servers it can be incredibly difficult for any process to succeed if the disk is full.
You won't feel too clever if you come to grow your LVM volume into the free space and it won't work because there's no free space on the filesystem! :)
(I don't actually know if this would fail or not - but the point is "rm spacer.img" is pretty much guaranteed not to fail).
I think the ideal ideal is tuning the reserved blocks in your filesystem.
That's only a problem if your memory is full as well, and even then, I've never encountered a server that uses a swapfile instead of a swap partition.
On Linux, this is not necessarily the case, as you can change the file on disk while the executable is running. I don't know if Linux just keeps executable code in memory all the time, or if it is smart enough to detect whether a copy of executable pages still lives on disk.
What you can do is delete and then recreate the executable. Then the deleted data simply sticks around on disk until it's no longer referenced
If you haven't customised this, in a pinch you can still lower it down a bit to buy some time.
It's important to note that mkfs doesn't care if you are formatting the root partition or a data volume partition, it will still reserve space for the kernel.
Disk space is cheap enough that even 5% of a 14TB disk is really not that much money --- and if you see bad performance, and then have to pay $$$ to switch to an SSD, maybe it would have been much cheaper to use a HDD with a larger free space reserve....
Not true, I've checked. I have plenty of Linux ext3 servers running for many years that routinely drop down to 1% free space for extended periods before being cleaned-up, which still have essentially zero fragmentation. You can create plenty of 10MB files on a multi-terabyte volume that has under 1% free space, as that's still tens of gigabytes to work with.
Obviously at some point you'll hit a severe problem and it's best to avoid taking a chance, but a fixed percentage really isn't the best measurement to tell you where that horizon will be.
Different file systems have wildly different strategies and data structures behind this process, however. Some drop to their knees over 92-93%. Some can write to the last byte with reasonable efficiency—but it'll never be as fast as when it was empty. Copy-on-write systems like ZFS tend to do poorly under near-full conditions.
No, still completely untrue.
> When you write a file larger than the contiguous available space after its starting point, a file system must break the file
And why would having a 10TB file system force you to write 100GByte files, whereas a 10GB file system would write 100MByte files instead?
Because that's the topic you're responding to... GP said it doesn't matter whether it is a 10GB file system or a 10TB file system, it "gets badly fragmented" when you get to "99% full".
One fun fact I learned though - one of the reasons is "for quota file". Since ext stores quota information in an ordinary file, there could be an issue where user fills up a disk but the information about it isn't written in quota file due to lack of space.
The way I found out about it originally was because I was using external storage drives and I was never able to fit as much as I expected :D
Luckily you can easily change this without reformatting.
Nothing that actually does the "serving" or accesses data should be running as root.
This is for daemons using syslog methodology, not direct writers like apache/nginx/mysql/etc; think more like cron, systemd, chrony, NetworkManager, and so forth. The vendors are not all aligned on what goes where (example: on RHEL, pacemaker/crm write to their own logs buy on openSUSE they're sent to syslog) - the actual results differ slightly from vendor to vendor.
DIY distros like Arch do not implement the rsyslog backend by default, you have to set it up yourself following the wiki - only journalctl is there by default.
That way, even if your log volume or root somehow fills up before monitoring had a chance to react, your service is unaffected. You can even catch issues pre-emptively by keeping log volumes small so that weird behaviour is likely to trigger an alert before anything goes truly wrong.
On cloud instances, it's silly to put anything on the instance root volume (on AWS, I keep them at the default 8 GB; it's never been a problem) when you can just attach an arbitrary number of additional disks. Container systems would use persistent volumes, and with physical servers, you use LVM or equivalent. This solves most disk allocation issues and makes operations easy when you need more space.
The idea is that /var filling up would not make the system unrecoverable.
> The minimum amount of space guaranteed to a dataset and its descendants. When the amount of space used is below this value, the dataset is treated as if it were taking up the amount of space specified by its reservation. Reservations are accounted for in the parent datasets' space used, and count against the parent datasets' quotas and reservations.
* https://openzfs.github.io/openzfs-docs/man/8/zfsprops.8.html
These are done on a per dataset basis (basically a directory delineated boundary).
Unlike ext[34], that reservation is not available for root's general use, but it's there.
Was also thinking of customising the login greeting to mention the file (/etc/motd).
Is this what you mean?
I just tried to replicate this behaviour but, dammit, I cannot demonstrate that right now as the behaviour so far was the expected one (free storage decreasing when creating the file and sticking to that even after the completion of the operation).
I strongly believe that that's what I saw in the past (when I was preallocating image files to then be used by KVM VMs), but now I'm wondering if I'm imagining things... :P
EDIT: this happened when using ext4 and/or xfs (don't remember) without using any compression.
About the blogpost itself:
The disk filled up, and that's one thing you don't want on a Linux server—or a Mac for that matter. When the disk is full nothing good happens.
I had this happen few times on a Mac and every time I was shocked that if disk gets full you cannot even delete a file and the only option is to do a full system reboot. I was also unable to save any open file, even to external disk and suffered minor data loss every time due to that.
What is the proper way of dealing with such issue on macOS? (or other systems, if they behave the same way)
Fun story with this. Ubuntu now has an experimental root-on-zfs feature. I installed it and started playing with some docker containers, trying to compile a certain version of pytorch. Suddenly, my computer crashed. Apparently, my root partition filled because docker installed everything on the same partition as my OS, crashing everything immediately.
I've had the mispleasure of pointing a large video export to the wrong drive and other misdeeds that allowed a drive to fill up. It's not pleasant. A simple reboot sometimes frees up the swap space to allow for more spring cleaning, but I typically just resort to booting into recovery mode and searching the web for the proper terminal command to decrypt/mount the root volume for spring cleaning.
This just happened to me. I got the best error message I've ever seen. Something akin to "Can not remove file because the disk is full." This wasn't from the Finder, this was command line rm.
On the Mac it's also exacerbated by the fact that swap will use the system drive and can fill up the disk, and can not be stopped. If you have some rogue process consuming RAM, among other things, your disk will suffer until it is full. And, as mentioned, macOS does not behave well with a full disk.
And, even if you've remedied the swap issue (i.e. killed the process), there's no way I know to recover the swap files created without restarting.
Just seems like the design is trouble waiting to happen, and it has happened to me.
When this last happened, somehow it managed to corrupt my external Time Machine volume.
One way to buy yourself some time is to disable the sleep file. I'm not sure what it's called -- it's a file that MacOS uses to let the computer hibernate when there's no power. It's a few GB, which (like the blog post stated) is a nontrivial amount of freeable space.
Should be /var/vm/sleepimage and the same size as your RAM.
Will be switching to this hack! Perfect illustration of the KISS principle (Keep it simple, stupid).
https://www.dodgycoder.net/2012/02/coding-tricks-of-game-dev...
> he had put aside those two megabytes of memory early in the development cycle. He knew from experience that it was always impossible to cut content down to memory budgets, and that many projects had come close to failing because of it. So now, as a regular practice, he always put aside a nice block of memory to free up when it's really needed.
Consider how long it takes to edit or recreate art assets to reduce their size. Depending on the asset, you might be basically starting over from scratch. Rewriting code to reduce its size is likely to be an even worse option, introducing new bugs and possibly running slower to boot. At least smaller, simpler art assets are likely to render faster.
This is also the kind of problem that's more likely to occur later in the schedule, when time is even more scarce. Between these two factors (lack of time and amount of effort required to get art assets which are both decent looking and smaller), I think in practice you're actually more likely to get better quality art assets by having an artificially reduced memory budget from the outset.
1. Deal with possibly multiple issues possibly involving multiple people with the politics that entails resulting in a lot of stress for all involved as any one issue could render it a complete failure.
2. Have extra space you can decide to optimise if you want. You could even have politics and arguments over what to optimise, but if nothing happens it all still works so there is a lot less stress.
I pick 2.
Also, don't forget you're hearing decades-later retellings of someone else's story. I don't doubt that they trickled this extra space out as changing requirements mandated it, but that they kept from doing so until the team had actually reached a certain level of product-maturity and reclaimed all of their own waste first.
Remember that the PMs goal is to ship. Them blocking some assets but actually shipping is a success. Better 95% of the product than 0%.
Would've been nice if someone had reserved some space ahead of time. Maybe they did, but nobody was around who remembered that codebase.
With a "ballast file" (as another commenter termed it), you can decide exactly when processes get to start consuming disk again, and that can give you some headroom to fix the problem.
Setup quotas or implement some damn monitoring -- if you're not monitoring something as simple and critical as disk usage, what else are you not monitoring?
Quotas are tricky to set up when things are sharing disk space, and that could easily give you a false positive where a service unnecessarily runs out of space.
some reading if LVM is new and you want to know more: https://opensource.com/business/16/9/linux-users-guide-lvm
edit to add: pv=physical volume, vg=volume group, lv=logical volume
You'd effectively send all block changes through LVM twice (once through the file, then through the underlying block device(s))
For block operations it's no different from manual setup of loop-mounted volumes, that also need to travel a couple of layers to hit the backing device.
Though there is an important caveat - LVM is more abstracted, making it easier to mistakenly map a drive onto itself, which may create a spectacular failure (haven't tried).
Perhaps consolidate/defrag once a year. Even monitoring total usage more often than that is probably not worth the effort - just buy ample cheap storage.
Also, there was a tradition to split drives into OS, DB, DB Logs. That was mostly a rust performance thing and these days is probably just voluntary management overhead.
RAM is another story.
And if you haven't overprovisioned their max space, you may as well not be using dynamic allocation and use fixed size disks.
Even then, snapshots will grow forever and fill the space, and then you hope you have a "spacer.img" file you can delete from the datastore, because you can't remove snapshots when the disk is full and you're stuck. It's the same problem, at a lower level.
Yes, I've seen that time bomb go off on multiple occasions. Never on my watch though.
> "just buy ample cheap storage"; "That was mostly a rust performance thing and these days is probably just"
In the UK a 6TB enterprise rust disk is £150 and a 2TB enterprise SSD is £300, it's 6x the price to SSD everything, and take 3x more drive bays so add more for that. And you can never "just" buy more storage than you ever need - apart from the obvious "when you bought it, you thought you were buying enough, because if you thought you needed more you would have bought more", so that amounts to saying "just know the future better", but it can't happen because Parkinson's Law ("work expands so as to fill the time available for its completion") applies to storage, the more there is available, the more things appear to fill it up.
Room for a test restore of the backups in that space. Room for a clone of the database to do some testing. Room for a trial of a new product. Room for a copy of all the installers and packagers for convenience. Room for a massive central logging server there. What do you mean it's full?
It's the same as old VW beetles which had a reserve gas tank. When you ran out of gas you opened a valve and you could limp to a gas station. Less likely to fail versus a 1950's era gauge that is telling you you're low. Also impossible to ignore it.
In scuba diving there used to be "J-valves". When you had 50 bar left in the tank they would cut out. Then you would pull to reenable your air and return to the surface. Unsurprisingly they are no longer popular.
It's not that straightforward IMO. Would this file be deleted before the space is filled? If so, there is alerting in place, and it assumes there's a way to delete files before space fills up. If this file is deleted after space fills up, how is this different from not having the file, other than making finding files to delete easier? Then what happens after that? If you delete the file and realize there's nothing else to delete, you'd have to solve the problem the same way if you didn't use this method.
What if the solution required some amount of free space? (eg. installing a package or swap)
Don't think I've ever encountered a critical issue where "add more swap" would be a serious disaster recovery solution. I've certainly seen situations where swap was nearing 100% full, and although I would have minutes off wall-clock time to formulate a strategy, those minutes have never allowed me to input more than a handful of characters or so.
> This is 2021 and the technology is from the 90s I don't see how this is a valid point. Is integrated circuit technology outdated because it was developed in the 60s?