Human-in-the-Loop Is Not the Same as Judgment-in-the-Loop (kunyuan.substack.com)

That's what worries me about AI. Not that it will replace human judgment, but that it may gradually remove the friction that judgment depends on.

My main worry about AI is people misunderstanding its limitations and then deferring their subject matter expertise judgement to a hallucinating, amnesiac used car salesman bot. Or taking humans out of the loop such as orders for lethal systems to kill humans without rigorous intelligence.. a decade ago, US mil drones absolutely required an officer to give the command. Ukraine, modern NATO/US.. I'm not so sure that boundary is respected anymore. Judgment and responsibility cannot be outsourced morally/ethically in a multitude of situations. Perhaps it's fine if AI manages meeting schedules or decides what you're having for lunch.

[dead]