It's a lot easier just to reboot and let fsck patch up your filesystem... flushing buffers isn't going to help you get back the last paragraph of your PhD thesis if Libre Office hasn't called write().
In any case, in ~8 years of using Linux, I don't think I've had a freeze that was the kernel and not just X.
One easy way to get kernel lockups is to use fglrx.
But agreed on 'just reboot': you should be using a resilient filesystem anyways, and MagicSysRq only works when it's enabked beforehand, and only for some kinds of lockups, and if the data is in the frozen application, syncing the disks isn't going to help.
And what if it's not your PhD thesis in Libre Office but a busy database server which can easily get corrupted if you don't flush.
> I don't think I've had a freeze that was the kernel and not just X.
Device drivers sometimes have bugs, especially when the device is not working properly (I had system freezes when there was a misbehaving capacitor on the graphics card).
Yes, I agree, but just several weeks ago I had to recover a MySQL InnoDB database after a power problem. MySQL is very popular, and it turns out it is an unsafe database server according to your definition. Well maybe it is.
Edit:
Besides, hard drives have write caches, and they can report a successful write operation to the OS when the data is still in its cache physically.
MySQL is only ACID compliant under a very specific set of configuration parameters. This makes it even more dangerous than an RDBMS with binary acidity.
Yep, MySQL is an unsafe database. The fact that it's popular just goes to show how powerful marketing is on th emind of most people.
Hard drivers are supposed to flush their caches before they report the end of a flush operation to the OS (some flush into flash, but they flush). If your does not, it's defective. Go ahead and make use of the warranty.
Re HD lying, this is rare except with dubious cheap flash sticks. The caching command set is well defined and operating systems know how to issue the relevant SCSI/SATA commands. It's critical for correct functioning of journaling filesystems such as NTFS and Ext4.
> And what if it's not your PhD thesis in Libre Office but a busy database server which can easily get corrupted if you don't flush.
No properly configured and working ACID compliant RDBMS should lose any data when the server is reset or stopped. If it does, then it is either a problem with the hardware, OS, configuration or the RDBMS itself. The application must also be able to handle the DB disappearing, though. Sadly this is often not the case.
The disk may lie. The 's' will get the OS to send everything out to the disk. Actually writing it to the platter (or flash part, or whatever) is at the disk's discretion.
It's at the disk's discretion as far as the laws of physics are concerned, but this would a severely broken disk prone to losing data and if it was a major server disk vendor, the vendor would take a pretty serious hit to its reputation.
Yes, but my point was that I could send data to an ACID compliant server, and kill it before the commit happened and data will be lost. Just trying to point out to the parent poster that sending is not enough, you need to wait for the commiting.
I remember some sort of magic invocation like that years ago for our medical "device" product we had to manage hundreds of remote node instances of. Something along those lines. I don't remember why the developer (Gabe) came up with sync three times being magic number. Maybe three times was just paranoia. =)
Back in the bad old days of pre-UNIX Macs, it was a common troubleshooting step to reset your PRAM. This is battery backed up RAM that holds some basic settings, and if it got corrupted somehow it could cause weird problems. You'd reboot while holding down command, option, P, and R, then wait for the boot chime to sound a second time indicating that it had been reset, then release the keys and boot normally.
Somehow this advice got mutated so that you'd keep holding the keys until you heard two boot chimes (thus resetting the stuff twice). And then it started to grow. Three was common. Some people would advise more. I'm pretty sure that doing it more than once never helped anything, but there we are.
(The cmd-opt-P-R sequence still works on modern Macs and I actually used it to resurrect a machine that wouldn't start up just a month ago, but it's far less frequently needed now.)
It's the same with the battery stats resetting and the Dalvik cache wiping these days in Android land. You do it three+ times.
Or the "Repair permissions" thing in OS X. You do it several times as well.
It's like whenever there's this one-step fix thing that a system utility does, the Common Man will interpret it as needing to repeat 3+ times in order for it to be effective.
I believe the thinking is because sync isn't instant, especially on older slower hard drives, having to type it again give it time to actually complete.
Back in the day (old Unix), the sync call would return right away, and the kernel would sync in the background. Unless there was a current background sync happening -- then sync would block until the first one finished, which is why you would have two sync's in a row. The third sync was thrown in just for luck.
This is likely a sun thing. Halt on (SPARC) Solaris does not shutdown the OS. It issues a reset command to the firmware. The halt command on Solaris is roughly equivalent to pressing the reset button on a PC.
When your Sun is particularly hosed we used sync;sync;sync; halt to reset and (hopefully) not lose any data (sync forces OS write buffers to purge)
>Run it and be amazed how much your disks/raid/OS lie. ("lie" = an fsync doesn't work)
>It seems everything from PATA consumer disks to high-end server-class SCSI disks lie like crazy. Yes, that includes SATA there in the middle. I'll discuss fixing your storage components in a second.
I picked up the "sync three times" thing from an AIX kernel developer, who did it from before AIX had a "shutdown". (Yes, he would do "sync, sync, sync, power-off".) My theory was that using it three times gave the system time to actually sync the data.
when doing kernel development on embedded systems or on a real host (not within a virtual machine), it's sometimes useful to get system information when it crashes. the cool thing is that it's also possible to send the sysrq through a console serial port by first sending a break command.
anyway, someone already mentioned a few mnemonics, I learned one, a quick googling lead me here:
I wouldn't advise other people doing this as the filesystem can get seriously broken under certain circumstances.
For example, on a Linux laptop with the hard-drive encrypted with dm-crypt, I simply lost access to my drive due to repeated hard reboots. I don't know if I could have recovered my data from it or not, but after repeated attempts of googling for the error message and following advice I simply gave up and later reinstalled everything from scratch (it's a good thing I constantly make backups ;-)).
This would be where keyboards with hardware-level macro support (like the Kinesis Advantage) come to be useful.
... to be honest, I can't think of another scenario. I have an Advantage for every place I spend extended computer time in -- it's too big and clunky to carry around...
It's possible that their computer is frozen enough to need this but it's more probable that this person hasn't figured out how to disable DontZap in newer versions of xorg. You need to disable it in your xorg.conf in order to have ctrl-alt-bksp work again:
That was one of my favorite Magic SysRq keys. It's the Linux analog to Windows's Ctrl-Alt-Del.
It's the "Secure Access Key" (SAK): You press that key and it kills all programs hooked to the TTY (incl. X, in your case) and displays a proper login prompt so that you can know what you're about to login to was run by the system and not a clever malware trying to steal your password.
I learnt about this a few months back... and never had to use it. Seriously, on my AMD APU netbook running Mint, I've had no freezes whatsoever, despite the hardware being a bit odd. A neat trick though.
I'm using Linux since 20 years and used to do this all the time, back when Linux --or more specifically X Window system-- was less stable than today. I'd make sure to always compile the kernel with MagicSysRQ (you need it for the combo mentioned in TFA).
It wasn't even mandatory to restart the system: the trick was to use first MagicSysRQ and then issue a "vgareset" (IIRC I couldn't even see what I was typing, but the command was taken into account) and then, miracle: I could unlock frozen X Window sessions (more specifically: kill X, "reset" the GPU and then restart a new X Window session).
Note that very often X is fine: it's just X which is frozen. Heck, if you have another machine on your LAN and allow SSH in, you can SSH and kill X / vgareset without needing MagicSysRQ.
But since quite a few years X is so stable that such hackery ain't needed anymore. Moreover if I recall correctly "vgareset" did only exist for 32 bits system (at least at one point). Nowadays my Linux workstation regularly reaches 6 months of uptime (there are only very rarely know remote root exploits mandating a kernel upgrade) so I've kinda "forgot" how MagicSysRQ works ^ ^
Yes, I believe that's true. Stop-A pops you to a prompt (which I believe is all running Forth) where you can do all kinds of cool stuff. It completely suspends the OS. I remember I was once able to TFTP in a custom boot logo at that gets loaded right into the Bios.
Sort of. Stop-A drops you into the firmware prompt (OpenBoot), suspending the OS in the process. OpenBoot, of course implements a forth interpreter shell.
Ahh, the good old days... I once implemented a nice little boot device selecting utility in for for some of my Suns. All written in Fourth, and executed by the firmware on every boot.
Actually, I did a lot of RPN development on Suns. Not in Forth but in PostScript within NeWS/OpenWindows developing front ends for large scale Lisp applications.
>Mind you I don't think I ever used "Stop A" on a Sun for anything constructive...
Back at the dot-com I worked at, we used to have a SPARCstation 5 that we used for SPARC-II arch builds. It was a nice machine, but it had a dead NVRAM battery so it would lose its configuration every time we lost power in the building (which was a lot, because rolling blackouts used to be a thing in California).
Anyway, an interesting quirk about the SPARCstation machines is that the MAC address for the NIC is stored in NVRAM, and when you lose the NVRAM settings it defaults to ff:ff:ff:ff:ff:ff AKA the broadcast address. So by default, on boot, this machine would start sending out DHCP requests and other network traffic with a source address of the broadcast address. The switches we were using did not like that, and would start flooding all of their ports with traffic. The only way we found to fix it was to reboot the switches.
So, there is at least one constructive use for "Stop A": you can use it to configure a MAC address on your SS5 so that it doesn't inadvertently bring down the whole network in a massive broadcast storm.
I've also seen one graphical app hang X. If you Ctrl-Alt-F1 and switch to your getty session you can sometimes kill the one app and return to your X session.
This is not for the situation in the article, but a slightly different one. You have a remote server, there's an ssh session still open to it but it's somehow lost access to all its mount points including /
You have nothing to work with but bash builtins and /proc, you're 100 miles away and you need to get it up and running again NOW. Emergency reboot -
echo 1 > /proc/sys/kernel/sysrq
echo b > /proc/sysrq-trigger
Sync never returns, all filesystems have been lost.
Yes, it's exactly like pulling and replugging, which was exactly what I needed!
--edit-- is sync even a bash builtin? Looks like it's /bin/sync on my systems. / had been lost (it was on a usb stick on an internal header on an unreliable usb3 card, I later found out.)
--edit 2-- if you meant also echoing s to the sysrq-trigger, it seemed to kill the session
How could that happen and leave your system in a state where it is still accessible and can be fixed by a reboot?
If you think you might face that kind of trouble you should keep a copy of Busybox and whatever other tools you might need on a RAM drive. You'd have an opportunity to figure out if rebooting would lead to a usable system.
The machine would occasionally stop responding completely. I got into the habit of leaving an open ssh session going from another box so I could try and poke around. The root drive was on a (very fast) usb 3.0 stick on an internal header on an add-in card.
There was a problem with the card or the driver as every so often something would go wrong and everything would stop responding again. The shell I left open revealed almost nothing as the root drive was gone, but it could be used to reboot the machine (thanks to the trick above), which would then be good again for another few days.
Control isn't necessary - I'm not sure why it would be specified. On a laptop you might have to hold the Fn key to hit PrtScr, but aside from that it's two fingers to hit 'Right Alt' and 'PrtScr' and the other hand can mash r,e,i,s,u,b.
In my workplace I have a Dell Keyboard that has the SysRq/Scroll Lock/Pause Keys on the top right, above the numpad. Holding Alt+SysRq with one hand is just impossible.
Yeah, with a typical ANSI layout it's a small stretch but definitely doable.
This is the layout that I have (and the only keyboard layout I'll ever buy, because every other throws the pipe key and backslash in a random spot along with randomly sizing the enter key)
The only way my linux machine has ever locked up is the usb driver seemed to crash in which case I couldn't do this anyways. The only other way I could have possibly restarted is to ssh to my machine from my phone, I opted for the power button instead.
unused
The server encountered an internal error or misconfiguration and was unable to complete your request.
Please contact the server administrator,
(etc...)
if your X is hung (ie, NOT "linux"), why not switch to a VC and fix it? yes, it's good to know about magic-sysrq, but most people don't even understand the layering of X on a VC, and the fact that other VCs are available.
of course, fixing why your X hung would be wise too, since for any normal distro and mainstream hardware, that's just not going to happen. if you really want to be fubar, have that f@cking POS systemd die on you... (yes, on-topic, since only sysrq saves the day.)
More often than not, it's your WM, DE or a GUI application eating up your RAM (Chromium used to be deadly for this) that's causing the freeze.
In more than a decade of running Linux on my desktop (and at work), I genuinely can't think of a single instance when I've not been able to pull a virtual console from a frozen desktop (albeit it often performs laggy).
When I learned this trick, it was because an X input driver was locking up[0]. The virtual consoles were completely inaccessible. Occasionally it would show me a blank screen for my efforts, but more often not even SysRq would work.
[0] I'm actually not 100% sure about that. Thank God I don't have to worry about it anymore.
Thanks for trying to diagnose my computer over a web forum, but I am competent enough to identify when and how my computer has failed. I'm not interested anecdotes from users. I develop video drivers. Am I allowed to experience these lockups now?
Firstly, I'm not just a Linux user. Like you, I'm a developer too. Given the demographic of this forum, it would pay for you not to assume that you're the only one on here that works in the industry (in fact I even hinted at that when I said I use Linux at work - but never mind)
Secondly, I was making a general comment about peoples desktops rather than talking specifically about your example (given the lack of details you posted, it would be insane of me to assume I could diagnose your fault with any precision). My point was that generally when people think their computer has locked up / X has crashed, it's actually one of the items I mentioned earlier that's at fault.
The snappy reply was appreciated though </sarcasm>. But given just how unusual your circumstances are (assuming what you said is true) and how much you seem to hate it when others discuss these topics with you; it might be an idea if you clarify your position a little better the next time such a topic arises. Like maybe saying "my crashes aren't typical because I'm a kernel developer, but.....". This way people don't accidentally post something that hits one of those raw nerves you have and it saves us all from a lot of unnecessary condescension.
Let it go, let it go!
Can't hold it back any more.
Let it go, let it go!
Turn away and slam the door.
I don't care what they're going to say.
Let the storm rage on.
The server never bothered me anyway.
That's because the default behaviour of CTRL + ALT + DEL is to issue a proper reboot.
That, and because X consumes those keys, making them useless when X goes bad (what is about all the times that Linux freezes and it's not hardware fault).
146 comments
[ 2.9 ms ] story [ 151 ms ] threadIn any case, in ~8 years of using Linux, I don't think I've had a freeze that was the kernel and not just X.
But agreed on 'just reboot': you should be using a resilient filesystem anyways, and MagicSysRq only works when it's enabked beforehand, and only for some kinds of lockups, and if the data is in the frozen application, syncing the disks isn't going to help.
Yup. Changed my GPU to nvidia, no more core lockups.
> I don't think I've had a freeze that was the kernel and not just X.
Device drivers sometimes have bugs, especially when the device is not working properly (I had system freezes when there was a misbehaving capacitor on the graphics card).
Wouldn't that make it an unsafe database server?
Edit:
Besides, hard drives have write caches, and they can report a successful write operation to the OS when the data is still in its cache physically.
Hard drivers are supposed to flush their caches before they report the end of a flush operation to the OS (some flush into flash, but they flush). If your does not, it's defective. Go ahead and make use of the warranty.
No properly configured and working ACID compliant RDBMS should lose any data when the server is reset or stopped. If it does, then it is either a problem with the hardware, OS, configuration or the RDBMS itself. The application must also be able to handle the DB disappearing, though. Sadly this is often not the case.
I remember some sort of magic invocation like that years ago for our medical "device" product we had to manage hundreds of remote node instances of. Something along those lines. I don't remember why the developer (Gabe) came up with sync three times being magic number. Maybe three times was just paranoia. =)
Somehow this advice got mutated so that you'd keep holding the keys until you heard two boot chimes (thus resetting the stuff twice). And then it started to grow. Three was common. Some people would advise more. I'm pretty sure that doing it more than once never helped anything, but there we are.
(The cmd-opt-P-R sequence still works on modern Macs and I actually used it to resurrect a machine that wouldn't start up just a month ago, but it's far less frequently needed now.)
Or the "Repair permissions" thing in OS X. You do it several times as well.
It's like whenever there's this one-step fix thing that a system utility does, the Common Man will interpret it as needing to repeat 3+ times in order for it to be effective.
"sync; sync; sync; halt"
Presumably so you had thinking time before automatically restarting a possibly sick system.
When your Sun is particularly hosed we used sync;sync;sync; halt to reset and (hopefully) not lose any data (sync forces OS write buffers to purge)
>Run it and be amazed how much your disks/raid/OS lie. ("lie" = an fsync doesn't work)
>It seems everything from PATA consumer disks to high-end server-class SCSI disks lie like crazy. Yes, that includes SATA there in the middle. I'll discuss fixing your storage components in a second.
anyway, someone already mentioned a few mnemonics, I learned one, a quick googling lead me here:
http://fosswire.com/post/2007/09/fix-a-frozen-system-with-th...
For example, on a Linux laptop with the hard-drive encrypted with dm-crypt, I simply lost access to my drive due to repeated hard reboots. I don't know if I could have recovered my data from it or not, but after repeated attempts of googling for the error message and following advice I simply gave up and later reinstalled everything from scratch (it's a good thing I constantly make backups ;-)).
On Linux REISUB has been my friend.
... to be honest, I can't think of another scenario. I have an Advantage for every place I spend extended computer time in -- it's too big and clunky to carry around...
(...from painful experience...)
It's the "Secure Access Key" (SAK): You press that key and it kills all programs hooked to the TTY (incl. X, in your case) and displays a proper login prompt so that you can know what you're about to login to was run by the system and not a clever malware trying to steal your password.
It wasn't even mandatory to restart the system: the trick was to use first MagicSysRQ and then issue a "vgareset" (IIRC I couldn't even see what I was typing, but the command was taken into account) and then, miracle: I could unlock frozen X Window sessions (more specifically: kill X, "reset" the GPU and then restart a new X Window session).
Note that very often X is fine: it's just X which is frozen. Heck, if you have another machine on your LAN and allow SSH in, you can SSH and kill X / vgareset without needing MagicSysRQ.
But since quite a few years X is so stable that such hackery ain't needed anymore. Moreover if I recall correctly "vgareset" did only exist for 32 bits system (at least at one point). Nowadays my Linux workstation regularly reaches 6 months of uptime (there are only very rarely know remote root exploits mandating a kernel upgrade) so I've kinda "forgot" how MagicSysRQ works ^ ^
You can force a kernel panic by 'echo c > /proc/sysrq' if you really want.
http://en.wikipedia.org/wiki/Magic_SysRq_key
Mind you I don't think I ever used "Stop A" on a Sun for anything constructive...
Ahh, the good old days... I once implemented a nice little boot device selecting utility in for for some of my Suns. All written in Fourth, and executed by the firmware on every boot.
Back at the dot-com I worked at, we used to have a SPARCstation 5 that we used for SPARC-II arch builds. It was a nice machine, but it had a dead NVRAM battery so it would lose its configuration every time we lost power in the building (which was a lot, because rolling blackouts used to be a thing in California).
Anyway, an interesting quirk about the SPARCstation machines is that the MAC address for the NIC is stored in NVRAM, and when you lose the NVRAM settings it defaults to ff:ff:ff:ff:ff:ff AKA the broadcast address. So by default, on boot, this machine would start sending out DHCP requests and other network traffic with a source address of the broadcast address. The switches we were using did not like that, and would start flooding all of their ports with traffic. The only way we found to fix it was to reboot the switches.
So, there is at least one constructive use for "Stop A": you can use it to configure a MAC address on your SS5 so that it doesn't inadvertently bring down the whole network in a massive broadcast storm.
You mean "Linux" is fine?
https://en.wikipedia.org/wiki/Sync_(Unix)
Yes, it's exactly like pulling and replugging, which was exactly what I needed!
--edit-- is sync even a bash builtin? Looks like it's /bin/sync on my systems. / had been lost (it was on a usb stick on an internal header on an unreliable usb3 card, I later found out.)
--edit 2-- if you meant also echoing s to the sysrq-trigger, it seemed to kill the session
How could that happen and leave your system in a state where it is still accessible and can be fixed by a reboot?
If you think you might face that kind of trouble you should keep a copy of Busybox and whatever other tools you might need on a RAM drive. You'd have an opportunity to figure out if rebooting would lead to a usable system.
There was a problem with the card or the driver as every so often something would go wrong and everything would stop responding again. The shell I left open revealed almost nothing as the root drive was gone, but it could be used to reboot the machine (thanks to the trick above), which would then be good again for another few days.
I now have / on a proper SSD...
I think I need an extra hand to do this
Source: http://en.wikipedia.org/wiki/Magic_SysRq_key
Maybe it's easier in saner keyboards.
This is the layout that I have (and the only keyboard layout I'll ever buy, because every other throws the pipe key and backslash in a random spot along with randomly sizing the enter key)
http://commons.wikimedia.org/wiki/File:ANSI_Keyboard_Layout_...
unused The server encountered an internal error or misconfiguration and was unable to complete your request. Please contact the server administrator, (etc...)
Did their Linux get frozen?
[edit] it works again
of course, fixing why your X hung would be wise too, since for any normal distro and mainstream hardware, that's just not going to happen. if you really want to be fubar, have that f@cking POS systemd die on you... (yes, on-topic, since only sysrq saves the day.)
In more than a decade of running Linux on my desktop (and at work), I genuinely can't think of a single instance when I've not been able to pull a virtual console from a frozen desktop (albeit it often performs laggy).
[0] I'm actually not 100% sure about that. Thank God I don't have to worry about it anymore.
Secondly, I was making a general comment about peoples desktops rather than talking specifically about your example (given the lack of details you posted, it would be insane of me to assume I could diagnose your fault with any precision). My point was that generally when people think their computer has locked up / X has crashed, it's actually one of the items I mentioned earlier that's at fault.
The snappy reply was appreciated though </sarcasm>. But given just how unusual your circumstances are (assuming what you said is true) and how much you seem to hate it when others discuss these topics with you; it might be an idea if you clarify your position a little better the next time such a topic arises. Like maybe saying "my crashes aren't typical because I'm a kernel developer, but.....". This way people don't accidentally post something that hits one of those raw nerves you have and it saves us all from a lot of unnecessary condescension.
Each one of these commands (r, e, i, s, u) takes or can take a few seconds to complete successfully, so let them do their thing.
In particular, if you have HD-activity LEDs, watch and wait until they stop blinking - after the 's' (sync) key.
That, and because X consumes those keys, making them useless when X goes bad (what is about all the times that Linux freezes and it's not hardware fault).
I'll need three hands.
If a Windows box froze, and you had a (somewhat slim) chance of gracefully shutting it down, would you not use it?
Would you call pressing F8 to access magical boot options in Windows, a reason non-technical people wouldn't use Windows?