375 comments

[ 3.0 ms ] story [ 263 ms ] thread
Same goes for work. If you join a well oiled machine that just seems to be working great, you may never understand how it works, since you won't be exposed to various people diagnosing issues. Join a place that used to work and is now creaking, or a place that never worked, and there's more pain but also more learning.

Also when does a Factorio system ever work? There's permanently a pressure for it to do more, stuff stuck on the wrong belts, not enough of some input...

> Also when does a Factorio system ever work? There's permanently a pressure for it to do more, stuff stuck on the wrong belts, not enough of some input...

Sounds like a growing SaaS company.

I think this is the classic "when it is working well, no one notices"
If you use a calculator and then carefully design out the logistics & geometry of your factory, you can build stuff that basically "just works" once you iron out the missing inserters & power poles. You get bottlenecks when you focus on one product at a time instead of looking at how the system as a whole functions.
What does it have to do with Factorio?
The linked tweet is part of a larger thread which details an issue with how they set up transmitters in factorio which led to the destruction of the entire base. The thread then pivots to how it applies to other areas.
Ah thank you, that makes sense now. I'm not too used to how Twitter is organized.
(comment deleted)
Oh cool, a random tweet from the middle of a twitter rant.

Unleash the chaos monkey's just in case, not to get too confident.

One reply tweet notes "a complex system always operates in a failure state", but to find more discussion on this point it's worth noting that this is a restatement of:

> "The Fundamental Failure-Mode Theorem (F.F.T.): complex systems usually operate in a failure mode."

-- John Gall, General Systemantics (aka Systemantics, aka The Systems Bible), 1977 < https://en.wikipedia.org/wiki/Systemantics >

Searching for "fundamental failure mode theorem" will provide interesting further insights on this.

'complex systems usually operate in a failure mode' has a somewhat different meaning from "complex system always operates in a failure state".

They usually operate in a failure mode because it's easy to be deceived by the superficial appearances of complex systems, thus it seems like a big bother to drill down the layers, until a failure has occurred. As excellently demonstrated by this 'Factorio' player.

The older I get and the more systems I encounter, the more I'm convinced everything is in a state of failing. The question is only how long before it becomes noticable.
Experienced sysadmins see the occasional chatter of problems in logs and emails as the pulse of the system, the EKG hooked up to the patient letting you know how things are going.

The right level of logging is usually enough to be annoying but not so annoying you need to reduce it. That means you're hopefully thorough enough in the logging that you're reporting what you care about and the things you see are the the things you can manually fix or ignore, and fit a frequency level you can live with.

Until the stuff you're managing becomes too large for that to make sense anymore, because that only scales so far (but surprisingly far for diligent admins). Then you need to move to something more complex, which is a big job and requires an entirely different way of reviewing (like Prometheus). You're probably better doing that from the beginning, but it's time and effort but everyone can take, or may not have existed at that time.

> because that only scales so far (but surprisingly far for diligent admins)

Surprisingly far. We log terabytes of data a day, most of it is unnecessary. I've been trying to tell people that it is not a good idea but so far I've had limited success. It's easy for people to just "printf" everything (we are lucky if logging libraries are used with proper levels).

Right now, we have the most expensive 'add' operation in history. K8s pod logs some data. That is eventually noticed by fluentd, which is watching the filesystem. Fluentd sends to our logging servers over the network. That goes to kafka, then is shipped to the system that actually does the indexing. And then, there's a query running, which will then count the number of times that string appeared and updates a counter.

All of which could have been avoided if the apps just added +1 to a counter, and exposed that as /metrics so Prometheus could scrape.

mtail is handy for such cases, when you really only care to extract a few metrics from the logs. https://google.github.io/mtail/

Generally though I've found once you start aggregating logs you find many more uses for them.

I don’t know if we worked for the same company but your logging pipeline sounds almost identical to the logging pipeline at my last company. That said, logs are definitely abused often, usually all it takes is an engineer to say to themselves “well, I know this gets logged 1000x a second per host, but, someday during an outage I could potentially use this!”, without realizing that one log call can cost thousands or more a month. Tools like ElastAlert unfortunately dont help in that it makes people comfortable using logs as a fail signal. It seems like the best way to get teams to limit their logging is to give them the exact dollar amount their log call costs; ie, “this one log line accounts for 5% of all log traffic, well that costs $X a month, and it’s only been queried for N times”
One of the consequences of this is when something goes very wrong, and the people involved say it really shouldn't have because something like 4 separate things have to fail at the same time for that to have happened.

What they don't realise is that, thanks to the FFT, 2 of those 4 things actually failed over a year ago and the system has been continuing to work ever since without anyone ever noticing. One of the other things actually fails intermittently a couple of times per month, and while a couple of people have noticed that and reported it, either it's not been investigated at all, or when it was investigated everything looked fine and the report was put down as an unexplained glitch. All that was actually needed was for the 4th thing to fail, to cause this "can't happen" failure mode to suddenly ruin everyone's day.

All of this is further exacerbated by the modern business take on capitalism of "optimize away work that doesn't positively produce".

Just because you're paying someone to go check and they don't find anything 99% of the time, you don't get rid of that. If you do, there goes the pnly sensor capable of propagating that signal to the rest of the control network. Applies equally well in electronics/circuits/or human domains.

If only C-levels and Boards viewed such jobs as a good excuse to provide someone with a living as opposed to poor worker utilization hurting their bottom line
Time to market. Nothing else matters, use whatever stack/library works and whatever I can copy/paste of stackoverflow to get it to work, we can fix the problems in software as we go and we only have to support it for 2 years then we can make another.
Thanks! I read this years ago and have been sharing the concept with colleagues, but couldn't remember where it came from.

It's a really useful concept to keep in mind when designing or maintaining complex systems.

(comment deleted)
The number of interactions of a system with n components increases according to O(n^2).

The probability of an interaction not going wrong decreases exponentially with each interaction.

Hmm, what does he mean "I woke up the next day"? Did he leave Factorio running, or did it get an online component while I wasn't looking?
Probably left it running? I've definitely left my factory to idle to build up resources while I'm gone, when I'm playing on peaceful mode.

You could also run headless version of the game on a server and just connect to it for the same effect.

Remember - Factorio is just an idle game with extra steps.

Ah, makes sense, thanks.
*they, this is why they hate their tweets being posted here
It's done by some people intentionally to be offensive, and it's impossible to tell if someone doing it is doing it as an insult or out of ignorance. And, believe it or not, there's enough people out there that do it intentionally that assuming ignorance would be incorrect more often than it'd be correct
Really? Huh, that's shitty.
Go around your office using "she" to refer to your male co-workers and see how long before someone gets cross.
The chance you get the gender correct when you know the real name, the voice, the look and the behavior of a person is significantly higher. Thus it makes more sense to raise an eyebrow when someone misgenders in real life.
They don't teach that in college.
(comment deleted)
There are two good lessons here:

1) Robust system design involves identifying the parts of your system that are mission-critical and always monitoring them. NASA missions have great automation and a 24/7-staffed mission control.

2) If a system failure can result in massive secondary damage, isolate that system. Warehouses receiving orbital payloads should probably be nice and far away from the base you care about.

> Warehouses receiving orbital payloads should probably be nice and far away from the base you care about.

As mentioned elsewhere in the thread, shades of 'The Moon is a Harsh Mistress' by Heinlein.

All your base are belong to dust.

Sorry, having never read that story (nor played Factorio since it went interplanetary) I don't understand the reference.
> 1) Robust system design involves identifying the parts of your system that are mission-critical and always monitoring them. NASA missions have great automation and a 24/7-staffed mission control.

You should alert on critical parts but you should monitor anything and everything you can. It might be critical in finding out why system broke later on. Easier said than done for hardware but easy for software

A more narrow lesson: this is why all SQL statements involving NULL are always false.

> in any case, losing power means the transmitter stops transmitting.

> and here's the fun part: your circuit which controls that inserter is set to insert "if [ICE] < 8000"

> and GETTING NO SIGNAL AT ALL counts the same to it as ICE=0.

> 0 is < 8000.

(comment deleted)
In industrial automation, one old and popular analog signal scheme is the 4–20 mA current loop [1]. Why 4mA instead of 0mA? Because that way, you know 0mA specifically indicates a fault caused by a failed sensor, break in the line, etc.

[1] https://en.wikipedia.org/wiki/Current_loop

A CHECK constraint treats NULL as true.
What drama with the Factorio devs is he talking about at the end (and namesearchers?.. I don't get Twitter sometimes)

I've never heard anyone say anything about the Factorio devs except praise their productivity and professionalism.

Kovarex promoted Uncle Bob in the past, who has generated a drama vortex of his own.
One dev said in one post that they like Uncle Bob.

I think that’s it.

> "saying the name of the game uncensored is going to get namesearchers and I have a transflag in my name so guess what kind of response I'd immediately get?"

I believe kovarex made some anti-trans statements once, and gamers in general are prone to making abuse pile-ons, so it's a mitigation against that.

> I believe kovarex made some anti-trans statements once

Nah, publicly advertised for and defended some right-wing Youtuber. [0]

[0] https://knowyourmeme.com/memes/subcultures/factorio

(comment deleted)
Uncle Bob? Stating he is just "some right-wing youtuber" sounds mildly reductive, the guy co-authored the agile manifesto[0].

I won't go as far as defending his views, but recommending his talks and books seems pretty normal to me.

[0] https://en.wikipedia.org/wiki/Robert_C._Martin

He doesn't even have a whole "getting cancelled" section on his Wikipedia page yet, can't be that bad.
Might as well cancel most programmers while we're at it. It's insane to lambast him for this when all he did was recommend some of Uncle Bob's videos on project management and programming.
(comment deleted)
Great article, made this exact same mistake in space exploration before learning to transmit demand and not supply. There are so many principles like this I learned from playing through the game. One area I've been focusing on improving is trying to anticipate how any given subsystem might go wrong and add even simple circuitry to detect condition and alarm/signal to the factory dashboard. Has cut down on the times where I realize some part of factory hasn't been working properly for like 5 hours. Applying this to software development has made me better about getting better about how I use rollbar logging
Also a good reminder why exceptions and optionals exist. Bonkers that "I didn't get a response" defaults to 0.
It's not that "no response" defaults to 0, it's that in Factorio's circuit network, "no signal" and "signal with a value of 0" are exactly synonymous.
One thing I hated about working with Go. POSTed JSON bodies missing a field could not be distinguished from those where that field pointed was present and pointed to the empty value for that (an empty array or string or something). This may be entirely work-around-able and simply a failure of how that job worked, but it was a whole thing.
In this case, instead of, say, an int, you could have the field be a pointer to an int.
And now you've traded one problem for another.
My favorite is, after fixing a system in prod, asking "how did that possibly work in the first place?"
I've totally asked that many, many times. But sometimes the answer is actually quite simple: no one was ever paying attention to the outputs of that piece of the system.
well, I'm the sort of person who digs around logs of ostensibly working systems just to find problems, which has been very useful in identifying long-tail problems (like, 1% of your network cables are blipping but the application just runs slower instead of failing).
Tests that output booleans (pass/fail) are an antipattern.

Tests and their dashboards must distinguish “the testing system worked and the test failed” from “the testing system failed”.

Most testing frameworks distinguish between failure (aka assertion violation) and error (invalid or nonexistent test result).

But there may be mistakes writing the tests (false success result). To check the tests, I break the implementation, then check that they fail.

I dunno, I think regression tests are inherently boolean. Either it passed or it didn't. The code path that the system takes to get to a green light or not is arbitrary. If you start using an enum to return the result of the test, folk tend to add more and more enums over time until it's overly complicated. Processes return an 8 bit integer, but 99% of the time all anyone checks is whether it was 0.

There's certainly practical benefit to being able to quickly triage a test failure. Tests that only output booleans are annoying. When there is a failure, it's helpful to have some out-of-band information about why it failed. That's typically provided via console output, but you can make whatever complex logging system you want.

If a regression test fails, you alert the programmer who kicked it off and maybe roll back the checkin.

If the regression testing system has an error while running a test, you page the oncall SRE and maybe try to rerun it. You certainly don’t roll back any checkins.

Wow, Factorio is approaching Full Time Job levels of detail and work required to keep everything running. Incredible game that I will never play, I feel I’m back at my job!
This is a mod that seems extremely boring and tedious. The base game is not like this.
Space exploration is a mod which has extra automation challenges compared to the base game, and it's probably got some of the most interesting ones compared to other mods which tend to just have the same basic concepts as the base game, just with many more steps piled on, which are probably more on the tedious side (pyanodons mods being the most infamous in this regard).
> This is a mod that seems extremely boring and tedious.

It doesn't have to be, though. I'm currently playing Space Exploration + Krastorio 2. Krastorio modifies the early/mid game, Space Exploration meddles a little with the early game, but most of it is late game.

There are a few more inputs required for things so they can get more complicated. And with Krastorio the early game is extended (you have "burner" versions of the science lab and assembly machines). So yes, earlier game is a little slower. On the other hand, it gives you stuff to make up for it. You don't require as much science(at least, before space) and there are things like core miners, which provide infinite resources so you don't have to keep moving miners. Also new power generation.

I guess that it depends on what you want to do. Base Factorio you either play until you launch the rocket, or you move on to megabases. If you want to do that, it's perfectly fine. However if you want to keep playing, it's a lot of the same, only bigger. Space Exploration essentially _starts_ once you launch the rocket. I don't want to do megabases so it seems ok with me.

Also, spaceships :)

Burner things is from space exploration too. Krastorio doesn't change the very early game.
That's very subjective. It also adds a ton of new gameplay components that push factorio players to better think about how to manage their logistics which is fun for some players (not all for sure)
It definitely can be. I don’t think it needs to be as long as it is. With 5 space science categories, each with 4 tiers, and each tier requiring 4 unique data cards with one common ingredient and one catalog recipe for the tier, one insight per category, there are ~125 intermediate products to manufacture that don’t really pose any new interesting logistical challenges. For example, once you set up iridium processing, the 20 intermediates you need to manufacture to finish all 4 tiers of materials science are pretty straightforward. Sure, dealing with the scrap generated in the process of making those intermediaries is a logistical challenge that becomes harder the deeper you progress, but they probably could have presented that logistic challenge without requiring you to set up production for 25 separate products — most of which are ONLY useful for making materials science packs.

I have really enjoyed the puzzles it has posed in the form of circuit network logic for automating rockets and spaceships. The issue described in TFA is just the tip of the iceberg. My factory has deadlocked in a number of interesting and unique ways over the course of the 600 hours I’ve played thus far. I’ve enjoyed the mod. It has some very interesting ideas and challenges. It’s a shame those puzzles are hidden behind so much slog.

Exactly why I stopped playing the game fairly quickly.
For most of these things, you just make a blueprint and stamp it down repeatedly.
Usually games or hobbies that feel like a job are much more fun because there's nothing wrong happening if you fail at it, or if one day you decide not to show up.
Definitely with you on your second point, but not on your first. Failing is extremely stressful for me!
Jobs get pretty fun too if nothing seriously bad happens if you fail at it. That's why so many people who are financially independent keep working.
And you don't have to fill out a bunch of paperwork whenever you do or change anything.
To be fair - this specific issue is with Space Exploration, a mod that's specifically for people who decided that Vanilla didn't hurt quite enough.
I think "never" is too polarized, maybe "be skeptical of" is a better suggestion.
(comment deleted)
foone's pronouns are they/them
(comment deleted)
If your code has side effects, it only ever seems to be working.

----

> the newbie says "aww, why isn't it working?"

> the intermediate says "yay, it's working!"

> the expert goes "hmm, why is it working?"

I've also seen this as

My thing isn't working and I change X and now I get error J instead of error K, so now I have to change X back because it's still not working.

Where K is a "better" error than J, such as K = failed to connect and J is Server 500 error. With J you are at least talking to a webserver.

Working with beginning programming students, "Yay, now we have an error message that makes sense" has become my catchphrase.
"Read the error message, no wait don't close that go back" when helping people is a definite feeling.

Take a deep breath. Take off your headset. Go outside. Never look at a computer again. Start driving a bulldozer. Life is good.

I’ve lamented here before, but the number of technical people who absolutely freeze up at the sight of an error message is daunting. Being able to interpret errors correctly is a huge step up in this industry; to me it feels like table stakes, but clearly isn’t.
> I’ve lamented here before, but the number of technical people who absolutely freeze up at the sight of an error message is daunting.

"freeze" is still a much better reaction than "quickly dismiss without paying attention". I can understand "freeze"; it's possible to learn and train from there to "good, well done, now read everything and think". I do not understand "quickly dismiss without reading and paying attention".

(I know about the studies for "people dismiss something that isn't getting them their goal", but I still don't understand the mindset that can dismiss something like that even when theoretically trying to figure something out.)

Part of the problem is that we've trained everyone, especially nontechnical people, to ignore and dismiss stuff.

You're using a program. A thingy pops up or unexpectedly shows up that interrupts you.

80% chance it's some stupid ad or newsletter thing. Dismiss with prejudice.

19% chance it's a "oopsie poopsie we did and whoopsie, the server monkeys are working VERY HARD to fix this, sowwy!!" and then the app still works apparently just fine, or if it doesn't you just force close and re-open the app.

1% of the time it's an ad again.

In the remaining floating point error's worth of %, it's an actual actionable thing a user could action to fix the issue.

I don't really blame people for just predicting that a pop up will be useless because it probably will be.

Usually error messages are somewhat nice, What I have problems with at work is "error 395" and such. Where the documentation for the error codes can't be found easily.

(I hate progress openedge)

It's amazing the number of junior engineers I've taught to simply read the error message properly.
Or a junior dev will see "Error 422: thing_id is required" and instead of updating the request they sent to include thing_id they'll report it as if it's a 500 Internal Server Error until you pick up the ticket and read the error message to them.

Of course, they didn't actually put the error message in the ticket, they'll write "backend crashed" and put a screenshot of the front-end crashing because they didn't do any error handling at all.

Or its close cousin: "Yay, now we have a different error message!" When something is totally broken and you have no idea why, this is a promising signpost.
Similarly, the joke that the engineer’s second worst nightmare is, “it doesn’t work but it should”, while their worst nightmare is, “it works but it shouldn’t”.
That’s the surest way to nerd snipe some people. This code doesn’t work, and yet it does. After reading the commit history, it in fact never worked, and yet it has been up until yesterday. What in the actual fuck.
The buggy behaviour of A caused buggy behaviour of B that caused buggy behaviour of C, just so happened to align enough to "look like it is working".

Double fuckup if the thing is backups and your data is gone. Reason why we check not only "does the backup job finished" but also "does the size looks right", because backup with 0 files still returns OK...

Haha yeah, been there. I forget all the details, sadly, but there was some long-present bug with how the frontend architecture word in the dev environment that would keep the changes from being propagated. But every member of the team had set up a workaround so that, on their machine, it would load the correct assets some other, undocumented way, so they wouldn't notice the bug.
My favorite was C code that was roughly (all functions and variables h:

  int bar;
  
  ...
  foo()
  {
    if(bar) {
      doX();
    else
      doY();
    }
  }
Note that uninitialized globals in C are implicitly zero-initialized.

doY() had been broken for years but nobody ever set bar to be true; doX() was nevertheless being called. Setting bar to zero caused the buggy doY() to happen. Setting bar to one seemed to work, but would occasionally segfault.

Then someone explicitly initialized bar and got a linker error. Turns out in ye olden days of C, before "extern" was a thing there was a feature where variables that were initialized in no more than one compilation unit would be all coalesced by the linker. Before doY was broken, someone defined a function named bar, which then caused bar to have a non-zero (i.e. true) value. Someone later added some broken feature to "doY" and it passed all the tests because changing doY() didn't have any effect.

You're missing the } and { around your else, btw. I'm only mentioning that because I immediately saw it and got ready to read a horrifying story involving the text "#define else".
oh gosh I am.
I thought that was the bug on my first read. Naming collisions suck.
I used to do industrial tech support and this gives me the raging anxieties.
I have an exceptional ability to fix things: often I can fix things before determining why they are not working.

This is immensely frustrating because I like to understand mechanical and electrical devices and take pride in my craft, so if I fix something too easily I don't trust that the fix will hold without a mental model of the thing.

No news is bad news.

I don’t like code that compiles cleanly or runs cleanly on the first try. The worst is when I know that code isn’t close to done yet. Without errors my list of actionable tasks is low. I know there are a certain number of problems, and if I can’t see any it’s because they’re invisible, not because I don’t have any.

I came here to say this exactly. There's nothing more stressful to me than when a lot of freshly-written C++ compiles without error or warning on the first try, and then seems to run without issues. I get the distinct feeling that there is a problem that I'm just not seeing yet.
Eh, that heavily depends on language. Some have better tooling to get most of the typical errors out before the compile.

That being said I'm always suspicious and start to fiddle with them if I see test + code pass on first time

Was thinking the same. Once you exceed like 5 tweets in a thread, it becomes a small article and you would've been better off just writing an article in the first place. If you check the thread reader below you can see just how unstructured and incoherent it is compared with a simple article.
If you're genuinely curious:

> I have ADHD: it makes writing blogs hard but Twitter easy. So it gets quickly tiring that every time I'm posted there [to HN] that the comments immediately go "why isn't this a blog? Twitter is a bad site for this kind of writing and I hate it blarg"

https://twitter.com/foone/status/1440695857699962893

This exact comment of yours is a prime example of why foone hates it when their tweets are posted here.

Thanks, that's interesting. I can see why it's easier for him to tweet in that case. But still doesn't change the fact that an article is easier to consume for the reader.
Foone is a them, not a him.

And yeah, you're right, it's definitely easier for the reader, but the author isn't required to make it easy for their audience -- if it was, Judith Butler wouldn't Judith...Butle and James Joyce wouldn't James Joyce. Foone writes tweets, 149k people have chosen to read their tweets. Personally, while I agree a blog'd be nice, I'm just glad foone makes their knowledge available.

How does that make sense? You write the same amount of text in the end.
The dopamine hit from click the send button can help overcome the ADHD tendency to wander off task. Plus, even if you have half an essay posted to Twitter and still get bored, it's much more difficult to just leave it halfway finished than to leave an unposted blog post in your drafts. And besides that, I'm really not sure why people get so upset at people putting content into the world. Wasn't the point of the internet to make it easier for people to share their knowledge with the world?
Personally I don't care too much.

But on the other hand I have ADD and there's a chance I get bored with a twitter thread and move on before finishing it while a blog post allows me to keep it open in a tab and read it in steps.

Foone also has a few hundred Tweet Twitter rant about why they are using Twitter, so at least they are consistent.
Just want to note that the novel noted in the first post, The Moon Is A Harsh Mistress, is a great read. I think I read it after seeing it recommended in Mary Robinette Kowal's Lady Astronaut series, but it's a nice balance of technical and science fiction with politics, almost like a condensed version of KSR's Mars Trilogy. It's mostly (not entirely) devoid of the more typical Heinlein sexism/objectification.
It seemed to me to be a handbook on how to conduct a revolution. Much as Heinlein's Stranger from a Strange Land was a handbook on how to start a religion (hello Scientologists).
It's been a long way since I've read it, and it certainly had an interesting theme of politics and oppression. What bugged me was the description of the resistance movement: it pretended to be cleverly divided into cells, but ultimately the computer knew everything so it was pretty questionable security theater, and it didn't seem the book was particularly self-aware of that fact.
This is why 4-20 mA is a common signaling standard in industrial automation. 4 mA means zero and 20 mA means one. 0 mA means broken transmitter!
Interesting. That makes sense, but what’s the reason for using current instead of voltage as the variable?
Current flow is independent of wire length, the sender only has to turn up the voltage high enough to overcome the wire resistance.
Also, there's greater immunity to interference and poor connections.
This happens because the real insulators are better approximations of ideal insulators, than real conductors are approximations of ideal conductors.

In electrical circuits made with normal conductors and insulators, e.g. copper wires and organic polymer insulators, it is possible to ignore the currents that branch from the wires and close through the insulators, because they are extremely small (e.g. in the femtoampere range), but frequently it is not possible to ignore the voltage drops along the copper wires, which are caused by the resistances of the wires, and which may be in the millivolt range.

I have a bachelors degree in electrical engineering and work experience in hardware design and in the back of my mind I've always known this to be true, but this is the first time in my life I've ever heard anyone say this explicitly. EE pedagogy is badly broken in places.
Indeed, when discussing theory we love pretending that there's no voltage drop from the wires, which can really bite you when you get into design or deep failure analysis. All good technicians and engineers learn this reality fairly soon, and it's one of the things we check especially when dealing with indeterminate failures.

As much as I love academia, I really wish there were more instructors with real experience that could say things like, "for math purposes assume a zero-loss conductor" while still emphasizing that real life is not so simple.

Maybe it has to do with the culture at specific schools. As I mentioned in another comment, my instructors loved to include wire resistance in their problems, and always explicitly stated the zero-loss assumption when applicable.

We did have a healthy incubation program, where instructors helped companies with product development, so perhaps that is what brought the real-world experience.

As another EE, I had the opposite experience. Voltage drops due to wire resistance and transmission losses were drilled into our heads at any possible opportunity.
Difference between focusing on digital vs analog, maybe? As another EE, I vaguely recall this being pounded into us more in power systems than any of my digital electronics courses.
Sounds like you missed the “actually, all those digital wires are transmission lines” class :)
> EE pedagogy is badly broken in places.

Generally, the biggest problem with math, physics and chemistry is that if you miss just one lesson (no matter if you physically miss it, or in worst case miss out understanding a concept for a couple minutes), you can easily end up "left behind" for years. With history, arts, languages, a number of computer science topics or most of biology you can either skip out on a certain subject or catch up easily on your own, but not getting a specific part in math/physics/chemistry and lacking the resources to follow up is setting oneself up for exponential learning gap later on.

The societal problem is that we've not set up schools or universities to deal with this "failure mode". Privileged kids can have after-school teaching or siblings helping out, but most will sooner or later resign and hate these subjects for the rest of their lives - and that is incredibly sad.

If only we had the technology to somehow record lessons for later viewing or for refreshing knowledge later, and additional catch-up materials & deeper dives into the topics. This seems like extremely low-hanging fruit.

Why do I get a sneaking suspicion that few in the machinery of education want this, as it is a fast-track to continuous accountability?

You will also get voltage drops from the sensors using power along your current loop as well, no? Whereas current will be the same along the entire loop all the time.
1-5 V is also common. You can convert between the 2 with a resistor.
Charge is conserved whereas voltage (i.e. electric field differential) is not.
High impedance (low current flow) signals are more prone to electrical interference. This is particularly bad when you're running long wires across a factory with big electric machines running. And if your signals are low impedance (significant current flow), then you get voltage drop across your long wire, so you can't use voltage-based signaling.
or why we build a stop command with normal-closed contacts. a broken wire results in the same signal as a "stop" command.
If you're doing three-wire control [0], it also means you can add additional "stop" buttons by wiring them all into the circuit in series.

A nice-to-have addition to your table saw is a big red stop button you can reach with your foot so you don't have to take your hands off of your work pieces to shut the saw down if things have gotten more interesting than you expected them to. Three wire control makes this super easy to add (once you've found the big red mushroom-head n/c switch, of course).

[0] https://twcontrols.com/lessons/control-wiring-3-wire-control...

I think for a foot pedal, it'd make more sense to have an enable rather than e-stop. Altough, that also depends on if you only work from one location or many. (Presses in industry often use such a design for hands, so that it ensures you hands aren't in the press.)
Semantics are really important in these situations. We have a system with an E-stop (emergency stop) circuit. Any break in the loop and the system will shut off. It needed to work together with one particular vendors board that insisted on having a "hardware enable" signal that it could control. Rather than adding an additional circuit, someone decided to have "hardware enable" flip a normally open relay in the E-stop circuit. Technically this "works" because enable clears the E-stop condition and allows the thing to function. From a practical point of view this has caused no end of odd problems, particularly when diagnosing issues where things fail because the meaning of this signal has been compromised.
This reminds me of what I've seen happen with the internal health-check endpoint often added to APIs. Simple at first, but eventually there are endpoints to cause the health-check to synthetically fail for deployment purposes; the health-check is also monitoring downstream dependencies; the health-check becomes much more and much less than it was intended to be.
It is also why white traffic lights no longer mean "go".

White means "the red glass lens in front of the incandescent bulb shattered again".

Related, from train signals:

> The invention of the track circuit in 1872 by William Robinson permitted automatic block signals. Most simply, a low voltage battery current runs up one rail and down the other. As long as current flows from the battery through one rail to the relay and back through the other rail, the relay remains energized and routes current from another battery to the green lamp of a clear signal. But when a train is present, the circuit is short circuited by the steel wheels and axles of the cars; a broken rail or an open switch will also break the circuit. As a result the relay is opened and the signal displays red. The system is arranged so the failure of any component results in a red indication or a dark signal (which must be interpreted as the most restrictive aspect that signal can display).

https://www.trains.com/trn/train-basics/abcs-of-railroading/...

Train air brakes are similar in failing safely for the same written-in-blood reasons:

https://en.wikipedia.org/wiki/Railway_air_brake

Some military trucks are designed the opposite way on purpose - because they’d rather have no brakes and be able to move than be stuck on the battlefield.
That’s a great example of how you always have to design for a context.
Yeah, any design is going to be various trade-offs, and you have to understand them (and how they may relate to your mission) when determining what to do and what would be a bad decision.

For example, cars have "fail open" brakes but have independent cylinders so that it is relatively hard to have all four wheels fail at the same time (older cars had single master cylinders) - and one of the tradeoffs is that people want cars to get going immediately and not wait for brake cylinders to "charge up".

As more and more things get computerized, you have more options for "intelligent failover" - anyone who used to drive an old manual car knew you could slow down using the engine - but now the computer could detect brake failure and do things for you to help recover.

And as seen from other comments, the modern US military prioritizes "on-road safety" over "under ongoing attack ability" - partially because of changing strategies I suspect, but also from the realization that the US military kills more members of the US military each year via accidents, etc, than anything else.

Pretty much every car made since the 1980s has had dual-circuit brakes with usually one front and one rear wheel cylinder on each circuit. Volvo being Volvo took it further with their 2-series cars in the 1970s and 1980s, where the front discs had four-pot calipers configured as two sets of two-pot calipers. Each braking circuit had one half of each front caliper and one rear caliper, so in the event of one half of the brakes failing it would still stop in a straight line.

You can use engine braking in automatic vehicles too, by simply changing to a lower gear.

Yeah - but as we saw with the floormat/unintended acceleration, many people don't think of it in an emergency (all the fatalities could have been avoided by shifting into neutral in those cases, might have blown up the engine but wouldn't have crashed/killed).
It's something you're taught to do as part of your driving test, or at least you were when I sat mine.
There is one subtle difference between automatic transmissions and manual ones and that is the manual one will allow a shift into a lower gear even when engine damage is likely.

An automatic one will not, and will hold the current gear until car speed falls within shifting range.

Ask me how I know :)

Yeah, newer ones can be too smart for your own good (but not theirs) - older ones had no problem with you slamming it into reverse or park at freeway speeds - ask me how I know!
Ahhh, the classic money shift.
I am reminded of a friend who changed his steering wheel plastic trim, in the process he bumped the wheel position sensor out of calibration. No worries, trigger a recalibration. Uh oh, won't recalibrate.

This of course meant the entire cars safety and stability control systems, ABS and traction control were all disabled. It drove like a tank.

The fix, obviously, was to replace the brake booster primer pump (hybrid car). It had worn enough to no longer recalibrate. Still perfectly functional, and would have kept working for a long time, bit it wouldn't quite meet the factory expected parameters so it failed it's test.

According to the other guy that founded the Range Rover forum I run, in the 1990s when they were getting P38s as traffic cars the training for "really really stop right now" was to throw it into reverse at 70mph and plant the throttle.

It appears that ZF 4HP20/4HP22 gearboxes can do this an unlimited number of times without apparently being damaged, but I'm not trying it on either of mine.

I have always wanted to try that! I'm not going to try it on any car I own either. It is all just super high risk, and I just don't have a spare car laying around that has the beans to pull that off.
Depends on the gearbox. Really modern ones won't, anything more than about ten years old will downshift if you select a lower gear.
Well, I have a 2000 Honda that won't do it. And I had an 03 Kia that also won't do it.
> For example, cars have "fail open" brakes but have independent cylinders so that it is relatively hard to have all four wheels fail at the same time (older cars had single master cylinders) - and one of the tradeoffs is that people want cars to get going immediately and not wait for brake cylinders to "charge up".

And emergency brake that's wholly separate circuit.

> Some military trucks are designed the opposite way on purpose - because they’d rather have no brakes and be able to move than be stuck on the battlefield.

Do you have any details on that? Or a source I could read? I'm pretty curious now.

I drove trucks in the military - specifically, FMTV, HEMTT, HMMWV, and 5 tons. None of them had the reverse brakes you describe.

The deuce/half air-over-hydraulic brakes fail open (loss of air pressure leaves you with hydraulic brakes alone, loss of hydraulic leaves you with no brakes at all). However, there is a failure mode that leaves the brakes full on if the air cylinder jams.

At the time it was an acceptable trade-off; later military vehicles have more in common with trucks than tanks, I suspect.

Losing brake boost in a failure condition is the typical failure mode in most vehicles with hydraulic brakes. A hydroboost failure in modern medium trucks reduces braking performance just as much as an air pack failure in the deuce and is just as scary..

The brakes, unboosted, are supposed to still be capable of stopping the vehicle. This is also why hybrid and electric vehicles with regenerative braking, that normally use brake-by-wire with a pedal feel simulator, fail through to using the PFS as a direct unboosted hydraulic cylinder.

Losing all brakes in a hydraulic circuit failure is just because those trucks are so old they only have single circuit hydraulic brakes. Part of the A3 upgrades was changing over to a modern dual circuit hydraulic system.

So basically, the issue is that medium trucks are in a weird in-between. Not heavy enough to require full air brakes, but heavy enough that some of the safety assumptions we make about light vehicles are potentially less true.

They have a way to change tire inflation to suit the terrain. I wonder if they can change the failsafe direction of the brakes to suit the situation.
Not in the US. US trucks are built using standard truck parts, including off-the-shelf air brakes that fail safe.
they didn't always fail safe.

I remember a house at the bottom of a steep mountain grade that got plowed into on a regular basis especially during that time.

EDIT: hmm... seems freezing might still be a way to cause failure?

Failure at the bottom of a steep hill is a different type of failure caused by brake fade due to the brake drum swelling with heat as the truck goes down the hill.
But I believe at some point air brakes had to be reversed by law.

In other words, the brakes would apply on loss of pressure, instead of via pressure.

We have white traffic lights now in Canada for "transit advance", when buses are allowed to proceed before the rest of traffic.
Yes but it's also a different symbol - a vertical line rather than the full circle (at least in my experience).
Would they have breaks (gaps) to segment the track into distinct stretches of conductors?

How far down the line would the supply voltage run (or how far away would a train complete the circuit) before resistance losses degrade the system?

Yes, insulators were (and still are) used between track blocks.

The maximum length of a block with this system (versions of which are still used today) varies greatly depending on local conditions. I've seen 2k to 10k feet as a general performance range.

Similarly train brakes are failsafe. They operate by compressed air. Being connected to the locomotive and hence control allows them to be released. If the carriages become disconnected, brakes get applied.
Alarm systems use the "End of Line" resistor concept. The alarm triggers when a sensor is opened and current flow stops. Closed is ambiguous because it could mean power is flowing through all closed sensors OR there is a short (alarm panels handle shorts). A resistor allows the panel to consider 0 Ohms a short and some specific number of Ohms as closed and working. It apparently also provides a way to detect tampering of the wires.

https://www.alarmsystemstore.com/pages/all-about-resistors

(comment deleted)
Another example is how garage door opener trip-beams work (the sensors that detect if something is in the way before lowering the door).

The transmitter and receiver are wired in parallel and connected to a power supply through a series resistor. When the receiver receives the signal from the transmitter, it pulls the supply voltage down, which then causes the transmitter to stop transmitting, which then causes the receiver to stop pulling the supply down, and the cycle continues. The end result is that if there's no obstruction, the supply voltage pulses and the system checks for this. If there's any other type of condition, the signal is either 0 or VCC. Basically, the system is designed so that false positives are extremely unlikely since that's the worst case scenario.

There's some capacitors in the transmitter/receiver to control the frequency.

Gas central heating boilers use something similar to control the gas solenoid - the solenoid is fed through a biggish capacitor, which has a diode and a capacitor to "smooth" the pulses from the microcontroller. If the microcontroller output latches itself on, the capacitor charges fully, there are no more pulses, and the solenoid drops out.
That sounds very complicated. What advantage does it have over wiring the two in series and having an open circuit indicate an obstruction?

edit: it gets you protection against one of the two failing closed, duh

Also why a standard keyboard has a Break key: used to be that a telegraph station would have a bell that rang if it detected a broken connection (a “break condition”), then people figured that deliberately inducing a break was a handy way to alert the remote operator. I think the first automated telegraph systems actually used current-loop signalling, but I’m not sure about the first practical ones; in any case, the concept carried over into modern (well, 70s-vintage) asynchronous lines—most UARTs nowadays have a break interrupt that you can enable.
I have a standard keyboard and it has no break key.
To clarify, by a “standard” keyboard I mean a (usually US) QWERTY PC keyboard with 101 to 105 keys (depending on the amount of Win/Super and Menu keys and the presence of an additional punctuation key to the left of Z). Those have three rows of three keys each above the arrows (not counting any “media keys” such as Power); the topmost one of those is PrintScreen / SysRq, Scroll Lock, and Pause / Break, though some of those labels may be omitted. I honestly have never seen a full-size PC keyboard without those. (How am I supposed to hit Ctrl-Break or Win-Break otherwise?)

Strictly speaking, there is no Break key in the same sense that there is no SysRq key—there are only special Break and SysRq scancodes that AT and PS/2 keyboards send instead of Pause and Print Screen when Ctrl or Alt respectively is depressed (thus the combined labels), while USB keyboards don’t even have those, they just send normal modifier sequences. But I don’t think that’s what you had in mind :)

I know what you meant. I was implying that a 101-105 key keyboard has not been standard for a decade or more.

I would be willing to bet that none of the ten most popular/best-selling computers in the country for the last 5-10 years have had a key that says “break”.

You're talking about laptops which you're right, are probably more popular than desktops, but taking a look right now at a few different laptops near me shows they have break as a function key. Take 3 data points as you will.

I'll take that bet and offer up a break key from my active keyboard if I'm wrong.

The 10 most popular/bestselling computers in the last 10 years definitely would include the Dell optiplex line that comes standard with a full size keyboard that has a Pause/Break key.
Nope, they’re all laptops.
Even on laptops, there's still usually a Pause/Break key, it's just merged with something else such that you have to use Fn.

For example, on most modern Thinkpads, Pause is Fn+P, and thus Break is Ctrl+Fn+P.

It's also named "Pause key", but one could argue that if your keyboard doesn't have any of these two keys, it cannot really be called "standard".
That's why in our monitoring we used nagios scheme + 1 (0 for invalid, 1 for ok, 2 for warning, 3 for crit, 4 for unknown)
Also issue with golang (+ protobufs) where you can't distinguish a populated zero value from the default "not present" value.
This drives me absolutely bonkers. We had a situation in a machine learning context where we should have been treating the output of a detector as NaN (it was deliberately opting out), but instead it was falling back to zero.

Last I checked (this was years ago) the way to deal with this is to wrap it in a sub-message (which are actually nullable), which is just kinda gross.

Thats why JavaScript has both undefined and null!
Do we really need to differentiate between undef and null?
Not necessarily, but it does allow a level of robustness to be able to distingish between values which have deliberately been assigned null, and values which have not been initialized correctly.
Lua does fine with only a nil, although it gets a tad annoying when you actually want to store a nil value in an array and end up creating a hole instead. (Arguably, the solution to that problem is to use your own marker object and not a nil when you need some sort of unspecified or default value which is not a literal absence of one.)
C# linqs method - first or default has the same problem
The method itself just uses the default value of the corresponding type. If you have a sequence of ints, and you cast it to Nullable<int> first before applying FirstOrDefault, then you can distinguish between null and 0.
That's one reason, but the big one is that it makes it easy to remote-power the sensors. It gives them a 4mA budget to Do Their Thing even when outputting a zero.
I would say the independence of conductor length to voltage drop since current is measurable at any point in the wire is more important. another big advantage to it is that it also is resistant to noise. (though you should be using grounded on on end TSP anyway) nowadays it's pretty common to use cat5 and read a bunch of things from the sensors though via various protocols. 4-20mA sure is a lot simpler to program against though.
This is a very dramatic illustration of one of the benefits of a pull vs push-based architecture.
I’ve built a pretty successful software monitoring business off of this basic premise.
While I have not really played SE (past initial launch, it was too daunting to build the ship in space, etc) I have run into these kind of "issues" in Factorio. While they can have disastrous effects, I often really enjoy finding the core problem and finding a solution to make it "fail-safe".

Sometimes I wish I could go back and wipe all my Factorio knowledge and start from scratch again, most importantly refusing to use blueprints from the internet except for basic things like balancers. Finding early/mid/late game malls/blueprints sort of ruined the game for me. Min/maxing is fine by myself but once I'm "competing" against the internet or feel obligated to find the most efficient/best green/red/blue circuit factory, or science packs, etc it really ruins the game for me and makes it feel more like a job.

I got a good thousand or two hours out of the game before I hit that point and someday I want to try playing it again with self-enforced limits on what I'm "allowed" to get from the internet and what I need to just figure out on my own. The first game I played was pure bliss (and I played with Bob's mods and a number of others, yes that was stupid for my first playthrough but damn it was fun), I'd love to recapture that.

I want to reinforce your plan. Somehow I played factorio for a few hundred hours before finding online communities and blueprints. It was a wonderful experience of discovery and exploration. I came up with a few "novel" designs during that time that I'm still proud of.

Once comparing my own work to the best online it took a lot of joy out of the game. But I found that the memory was short lived, once I cut myself off from those online communities it was easy to fall back into the "let me pull out my sketchbook and solve this problem" mindset. I think half the fun is that design process.

I'm well north of 1000 hours now, and still discovering new ways to play and solve problems. I've come full circle and now build spaghetti bases because they are so visually fun.

I've heard this story repeated many times now and I'm glad to hear people are able to get back into it after having the joy sucked out. I think next time I play I will refuse to use a "main bus" until much later in the game if at all. That was the one of the biggest joy-killers, trying to plan for a mega-base essentially from the very start. It's not fun to plan out your bus and leave lots of room for red/blue circuits or more science packs while you are still building MK1-level things. That or trying to setup production lines with room for MK2-MK3+ versions of things from the start. It's complete overkill especially since there is no advantage to having a bus next to where you start. Better to situate it further away where minerals are more dense/plentiful and later in the game when it actually makes sense. Looking back I can't believe how stupid it was for me to setup 4 lanes of iron, 2 of copper, and a handful for each circuit type from the very start of the game, it sucked all the fun out but I didn't notice the reason.

The game largely felt like I was stuck in the same cookie-cutter, always trying to never have to tear up lines or rebuild elsewhere (trying to do it perfectly from the start). That was easily half the fun of my initial runs, realizing I needed to expand or rebuild instead of trying to perfect the build order.

> I think next time I play I will refuse to use a "main bus" until much later in the game if at all.

I mostly played through without any internet advice until I shared with a friend who did use internet advice and the main bus was his advice. Thankfully I already had a base so I tried this for a specific process/product (rockets ofc but still). I think a lot of “late game” stuff by experienced players will likely converge towards a bus since you’ll be already learned scale and order by then, but maybe with less precision in layout. The problem is that you can’t un-see it once you start building with it once.

I agree that fuck perfect from the start, build at the right scale at the right time. It’s way more fun. Conversely, if you like multiplayer games, this is a game where it’s really fun to play with people who are new. You can use their naïveté to override your knowledge or build completely different looking based each time. It’s fun to guide them through rediscovery (eg wtf why did all the trains dead lock!)

I found I enjoyed the process of putting together a mall from a blueprint, but leaving the rest of the factory up to me. I find the logistics and planning part of factorio more interesting than the "build up all the parts you need to make the factory" part.