HN Explorer

40 comments

[ 2.8 ms ] story [ 105 ms ] thread
guiambros
Uh, this is indeed weird. Hope all is ok.

For those curious and without showdead, the first link was posted 1h ago, and shows:

"For ok whatfix swaptoAPP blahhhh"

quenix
Super bizarre, doesn't really seem like the behaviour of a hacked account either (those usually use their fifteen seconds of fame to promote cryptocurrency scams or to troll).
ssiddharth
If I were a betting man, I'd probably say either a pocket typing sort of situation (for the latter two) or a kid or animal gallivanting on/around the keyboard for the first.
aaws11
then the question arises. are you a betting man?
[deleted]
(comment deleted)
aaws11
maybe he's trying to code some kinda automation and running it on prod and the code is misbehaving.

Hope all is ok.

NhanH
Seems like production repl access went wrong somehow
[deleted]
(comment deleted)
acqbu
There's nothing to worry about guys, he's always been like that. You're simply paying too much attention - new year's goals, eh?
anigbrowl
Mescaline, I can tell by some of the pixels and having seen many weird trips in my time
tgv
Might be a test in production? The first link is dead now.
dang
Correct. Production is too tempting.
avinassh
testing in the production is the true way of testing.
LinuxBender
There are a few variants of t-shirts for this. "I don't always test my code but when I do, I test in production" I knew a director of operations and engineering that wore this shirt which made it funnier.

The best and weirdest bugs are found by the public in production.

acheron
Everyone has a test environment. Some places also have a production environment.
LinuxBender
Everyone has a test environment.

Yup, the weird bugs are rarely ever found in the test/dev/staging/qa environments. Sometimes it takes longer to even figure out how to reproduce some odd bug a customer found in Prod.

As one example we had one blade server that was causing financial data corruption. The senior java developers and architects debugged java down to the CPU and found 1 CPU core that was making mathematical errors under specific conditions. This would never in a million years have been found in dev, test, staging, QA, pre-prod environments. I was given the instruction by leadership to "drive over the server" in the parking lot as neither upstream OEM's were interesting in debugging it further.

Another fun one I ran into was using VLAN tagging on CentOS 5 on specific hardware with a specific NIC of whom shall not be named could get into a situation where partial frames were being transmitted in a loop from the ring buffer at the highest physical packet rate of the NIC which is surprisingly very fast. It brought down the access switches and the very big distribution routers. We debugged it pretty far down but hit a wall when one of the three vendors involved would not sign the 3-way NDA. The NIC was not Intel.

Or here is another one. We had a multi-million dollar storage solution that was never needed and in production it caused an IO fencing issue that broke the production databases in a glorious way. That bug would never have been tickled in dev or staging. It required a specific series of failures in the storage that triggered a logic failure on their controller of raid controllers. The vendor should have caught this one.

I could probably write a book about all the weird bugs that "should never happen" but only happen under real-world Production use cases.

toast0
> specific NIC of whom shall not be named

Inquiring minds want to know. CentOS 5 was a while ago, so maybe it's old enough you can talk about it?

> The NIC was not Intel.

I found intel's reputation for stable, reliable NICs to be well deserved. (I've heard their 2.5G nics are an exception, at least early ones) But I run into one where one if the rxqueues didn't work. That's fun to debug, but dropping traffic is way better than writing line rate garbage.

LinuxBender
Inquiring minds want to know. CentOS 5 was a while ago, so maybe it's old enough you can talk about it?

I would like to but if I go any deeper into that one a certain company will be embarrassed and will respond very poorly. When the day comes that they and their legal entities cease to exist I will share more of the ugly details.

I found intel's reputation for stable, reliable NICs to be well deserved.

Same here, at least for their server NIC's. I've found that some of their consumer NIC's tend to lean conservative on power management by default and will start having issues negotiating speeds on very long ethernet cable runs to which I have to work around by changing ACPI settings, BIOS settings and sometimes even through a L2 switch at one end. Most of those seem to use the igb driver.

toast0
Sorry for this. I guess you can only speak in broad strokes, you can't give us any real details or put anything out into the sunshine. Although you did give us a bit of intel. I guess you'll keep your NDA in force.

Not sure if I missed any vendors there. The nvidia one is a reach ;)

dang
Write that book!
LinuxBender
It is on my long-term to-do list but there are some things I have to knock out first. I've never written a book but I am sure there are people here that have written threads on the best ways to go about it and the pitfalls. I should search for those.
spike021
One time my company was working on something and we had a liaison at Splunk. After a meeting with this engineer I think we wrote in about some kind of bug and he replied back with the Austin Powers meme and that quote. This was several years ago now.
tgv
That's what a hacker would say!
dang
Wow you guys don't miss much do you? Two different things were going on here:

I was working on some spam filters and posting some test comments to verify some of that. That was 34285936. I posted a few and deleted them but apparently neglected to delete that one. I've done so now.

The other two were related to https://news.ycombinator.com/item?id=33855594, which I sent a repost invite for. I was going through the previous stories about Paul Otlet and noticed that http://nautil.us/issue/21/information/the-future-of-the-web-... doesn't exist any more - so I posted an archive.org link in those two threads. This will be useful if and when we have a new thread about Otlet's Mundaneum.

rkagerer
Well, this was a fun thread. Glad to hear all is alright :-).
bingo-bongo
..and thx again for all your hard and dedicated work!

imho, you are one of the main reasons why HN keeps being a high quality tech forum (which is extremely rare these days on the Internet) - so thx dang!

hdjjhhvvhga
> Wow you guys don't miss much do you?

That was super-euphemism for 'Wow you guys really don't have better things to do, do you' :D

Again, thank you for your great work, one can appreciate it even more in moments like these.

anotherevan
Of course, if someone was clever enough to hack dang, they'd have a clever cover story prepared...
smcl
It's probably just two accidents (making a mistake when pasting a link, then the HN comment equivalent of pocket-dialling) in a row.

edit: ah nm, he's posted a comment :D

adenozine
Why was this thread removed from the front page?
dang
It's off topic.
singularity2001
dang using live environment / account for dev / staging work, just as the rest of us;)
AnimalMuppet
"I don't always test my code. But when I do, I test it in production."