HN Explorer

42 comments

[ 6.6 ms ] story [ 101 ms ] thread
cmorrisrsg
We saw this on every Ubuntu Lucid machine we run. Restarting the JVM was insufficient - we ended up fully rebooting the machines.
silviosantoz
Leap second + Java + Linux fix (without reboot):

/etc/init.d/ntp stop; date; date `date +"%m%d%H%M%C%y.%S"`; date;

-- then restart java

timdoug
Works wonders for me. Thanks!
zship
Thank you! Was banging my head against the wall on RHEL6's tomcat. Went on Hacker News to stop thinking about it for a few minutes. Son of a bitch, the solution was right on the front page.
_thekev
A lot simpler:

date --set "`date`"

larelli
On a localized box, I had to run "unset LANG" first, because otherwise date doesn't seem to understand its own output.
juiceandjuice
Doesn't work for everyone (Didn't work with our java processes on RHEL 6.2)
henrikschroder
Where were you 11 hours ago when we rebooted all of our cassandra and mysql machines? :-)
aSig
I saw the same across all of our Ubuntu servers. Some running Tomcat others running SmartFox Server (a java based game server). A reboot + app restart fixed it.
rbanffy
I experienced no problem today. Is this Java-specific?
NDizzle
Yeah this is what is slaying linkedin at the moment.
cedrichurst
MySQL is also apparently affected, according to the #mysql IRC channel.
codeka
Yes, this happened on my development MySQL instance and also one of my production instances of MySQL -- but not another one that I have. I'm not sure why it only affected one, maybe different versions of MySQL?
thristian
No, it's a problem with futexes. If your production code doesn't use many (any) threads, you should be fine.

People are reporting the same issue with MySQLd, I had it with Firefox.

spullara
Has anyone seen this not on Ubuntu?
cmorrisrsg
We saw it pop up on Centos6 as well. JRE 1.6 and 1.7.
jerdfelt
I saw this on my Fedora 17 laptop. Chrome started eating up 100% CPU and my laptop started getting really hot. I tried restarting Chrome, updating Chrome, etc. Nothing worked. Rebooting fixed the problem.

Surprisingly, none of my servers have had problems, but I don't run Java anywhere.

andrewcooke
opensuse (mysql supporting the kde desktop search).
stevencorona
Even if you're not running java in your stack, it's worth a quick check on your servers. Our SoftLayer servers use an Adaptec RAID card and the monitoring software uses the JVM and will suck up all of your CPU. All of our MySQL/Cassandra/Redis/Kestrel servers needed to be bounced because they had RAID cards. Frustrating.
Maxious
Even if you don't have java installed but you're running on shared infrastructure that might be running MySQL/Java elsewhere it's a crazy day. My inbox is full of performance-below-acceptable-levels/high-load alerts.
Negitivefrags
I also use Softlayer and have a pair of servers with Adaptec RAID cards. Checking up on them they were not affected.

Having looked in to it, it seems our server admin decided to install OpenNTP to replace the standard NTP daemon. I'm not sure why he did that, but it seems OpenNTP just ignores leap seconds and compensates after the fact just as if your clock had drifted a second out of sync in the usual way.

_joe
Same problem here. I am supposedly on vacation in Barcelona and was summoned to help :/ Gotta love your java.
Sharlin
Except that it's not a Java problem, it's a Linux futex problem.
DEinspanjer
Fun that a bug for my team happens to be the thing linked in this article. :)
matthewlmcclure
What's the explanation of that wacky date format?

minute day hour month year . second

emmelaich
It's actually month day hour minute.second

Originally (and perhaps still) the month day is optional; allowing you to set the date with just hour and minute.

_thekev
I was able to restore my java app boxen (and all other boxen) by only setting the date to the current date. No restarts necessary.
ryanpers
its the futex() call that is bugged, so therefore things that are highly multi-threaded... for example mysql and java!

Many unix things are not as multithreaded as those examples.

rasur
I had a lovely morning (2am-4.30) rebooting about 50 boxes by hand. All wearing some shade of java.
gouranga
At the risk of shooting myself, all our jvms were fine on windows and Ubuntu.
burningout
I was just wondering why everything stopped working at night. System.exit() didn't work anymore in any of the java processes, or just random hangs.

wow.

[dead] RahulAgrawal
Thank you guys.. since yestartday night i was working on it.. searched lots of places.. some of servers were going down and some were working great.. never thought this would be the reason.
nickzoic
I didn't really think about it at the time, but right about then (I'm in GMT+10) a Minecraft server (OpenJDK 64-bit, Ubuntu 11.10) I had running got rather upset ...

     2012-07-01 09:59:59 [WARNING] Time ran backwards! Did the system time change?
     2012-07-01 10:01:14 [WARNING] Can't keep up! Did the system time change, or is the server overloaded?
... and stuck on 100% CPU until I noticed it and restarted it this morning. Thankfully no other issues that I've noticed.