> That 20ms is a smoking gun - it lines up perfectly with the mysterious pattern we saw earlier!
Speaking of smoking guns, anybody else reckon Claude overuses that term a lot? Seems anytime I give it some debugging question, it'll claim some random thing like a version number or whatever, is a "smoking gun"
> Keystroke obfuscation can be disabled client-side.
please never do that (in production)
if anyone half way serious tries they _will_ be able to break you encryption end find what you typed
this isn't a hypothetical niche case obfuscation mechanism, it's a people broke SSH then a fix was found case. I don't even know why you can disable it tbh.
> Obviously forking go’s crypto library is a little scary, and I’m gonna have to do some thinking about how to maintain my little patch in a safe way
This should really be upstreamed as an option on the ssh library. Its good to default to sending chaff in untrusted environments, but there are plenty of places where we might as well save the bandwidth
The guy in charge of Go's security decreed that SSL 1.3 (which he was a contributor to) was so secure that silly programmers should not be able to override what algorithms are allowed or not allowed, because why would they possibly need to do that, because he's such a genius, and even if someone DID find a security vulnerability, well....they can just wait for Google to publicly disclose it and release a patch, compile the new version, update their code to work with that version of Go, rebuild their containers, put stuff through testing, and then release it into production.
Versus...seeing there's a vulnerability, someone adding a one-line change to disable the vulnerable algorithm, compile, image update, test. And a lot less testing because you're not moving to a new version of the language / compiler.
The man has no practical experience in running a production network service, an ego the size of a small moon, and yet was a major contributor to a security protocol now in use by billions of people.
But hey, you can be a handbag designer and end up head of design at Apple soooooooo
Lots of the real world vulnerabilities out there exist exactly because of people choosing to support a range of crypto algorithms.
Sure, if it's an internal tool you can recompile both ends and force a universal update. But anything else and you need to stay compatible with clients and anytime you allow negotiation of the cryptosuit you open yourself up to quite a few subtle attacks. Not saying that choice about go is clearly a good one but i don't think it's obviously wrong.
Very interesting, I hadn't heard of this obfuscation before so it was well worth clicking.
Another good trick for debugging ssh's exact behavior is patching in "None" cipher support for your test environment. It's about the same work as trying to set up a proxy but lets you see the raw content of the packets like it was telnet.
For terminal games where security does not matter but performance and scale does, just offering telnet in the first place can also be worth consideration.
The really mysterious part is how ~10,000 packets per second costs ~20% of a core. That would mean SSH is bottlenecking in its code at ~50,000 packets per second per core which would be ~500 Mbps per core (assuming full packets) which is ludicrously slow. It is trivial to do 10x that packet per second rate. Is SSH really that poorly designed?
> I am working on a high-performance game that runs over ssh.
Found your problem.
But it is an interesting world where you can casually burrow into a crypto library and disable important security features more easily than selecting the right network layer solution.
One thing you notice if you have ADSL is that some services are built as if slower connections matter and others are not. Like Google's voice and audio chat services work poorly but most of the others work well. Uploading images to Mastodon, Bluesky, Facebook, LinkedIn, Instagram and Nextdoor is reliable, but for Tumblr you have to try it twice. I don't what they are doing wrong but they are doing something wrong and not finding out what they're doing wrong because they're not testing and they're not listening to users.
Nobody consulted me about their decision not to run fiber by my house. If some committee decides to make ssh bloated they are, together with the others, conspiring to steal my livelihood and I think it would be fair for me to sue them for the $50k it would take to run that fiber myself.
It's OK if you work for Google where there is limitless dark fiber but what about people in African countries?
It's the typical corporate attitude where latency never matters: Adobe thinks it is totally normal that it takes 1-5s for a keystroke to appear when you are typing into Dreamweaver.
In 2023, ssh added keystroke timing obfuscation. The idea is that the speed at which you type different letters betrays some information about which letters you’re typing. So ssh sends lots of “chaff” packets along with your keystrokes to make it hard for an attacker to determine when you’re actually entering keys.
Now that's solving the problem the wrong way. If you really want that, send all typed characters at 50ms intervals, to bound the timing resolution.
I wonder if this is the same reason why Microsoft's Remote SSH plugin on VS Code is so flaky even with a decent internet connection. Every couple of months I try to give it another go and give up due to the poor keyboard latency I inevitably experience. And the slow reconnects whenever I glance away from my computer monitor briefly. This is on a fiber connection with a 20ms ping to the remote machine.
The reliance on LLMs is unfortunate. I bet this mystery could gave been solved much quicker by simply looking at the packet capture in Wireshark. The Wireshark dissectors are quite mature, SSH is covered fairly well.
* Yes, I clearly know what tcpdump is / how to capture network traffic
* It has been several years since I have looked at a pcap
* I don't have wireshark installed on this computer
* I've done the thing where you decrypt TLS with wireshark exactly once, years ago, and I found it frustrating for reasons I can't remember[1]. Wasn't sure if I could do this with ssh
* When I started investigating this, I didn't remotely think that ssh was the root cause. I thought it was a quirk of my game
* I *did* make a client that printed out all the data it was receiving, but it was useless because it was operating at the wrong layer (e.g. it connected over SSH and logged the bytes SSH handed it)
* I'm experimenting with Claude Code a lot because it has a lot of hype and I would like to form an opinion
* Looking up flags is annoying
* Being able to tell an agent "look at this pcap and tell me what you see" is *cool*
So idk. I'm sure that you would have solved this much more quickly than I did! I'm not sure that (for me) opening up the packet in Wireshark would have solved this faster. Maybe reading the SSH spec would have, but debugging also just didn't take that long.
And the big leap here was realizing that this was my SSH client and not a quirk of my game. The time at which I would have read the SSH spec was after I captured traffic from a regular SSH session and observed the same pattern; before that I was thinking about the problem wrong.
I don't think that this is unfortunate. In fact, I think I got what I wanted here (a better sense of Claude Code's strengths and weaknesses). You're right that an alternative approach would have taught me different things, and that's a worthy goal too.
[1] I suspect this is because I was doing it for an old job and I had to figure out how to run some application with keys I controlled? It would have been easier here. I don't remember.
I don't see how Claude helped the debugging at all. It seemed like the author knew what to do and it was more telling Claude to think about that.
I've used Claude a bit and it never speaks to me like that either, "Holy Cow!" etc. It sounds more annoying than interacting with real people. Perhaps AIs are good at sensing personalities from input text and doesn't act this way with my terse prompts..
> I am working on a high-performance game that runs over ssh.
Step one, run https://www.psc.edu/hpn-ssh-home/introduction/ instead
Step two, tune TCP/IP stack
Step... much later: write your own "crypto". (I'm using quotes because, before someone points out the obvious, packets-per-keystroke isn't, itself, a cryptographic algorithm, but because it's being done to protect connections from being decrypted/etc, mess with it at your own peril.)
Depends on what kind of security. They might care about connection integrity. If a faulty (or malicious) router in-between client and server starts malforming packets, `nc` will display those malformed packets. SSH will only show you what the server intended, or nothing.
Not related to SSH, but does the eieio.games website make anyone else's monitor flicker? When the website is fullscreen it overwhelms something. I thought my monitor's backlight was going.
>>> That makes a lot of sense for regular ssh sessions, where privacy is critical. But it’s a lot of overhead for an open-to-the-whole-internet game where latency is critical.
Switching to telnet instead of SSH might be an option.
63 comments
[ 2.6 ms ] story [ 72.1 ms ] threadSpeaking of smoking guns, anybody else reckon Claude overuses that term a lot? Seems anytime I give it some debugging question, it'll claim some random thing like a version number or whatever, is a "smoking gun"
Disabling TCP_NODELAY would also reduce number of packets + be portable & simpler to implement - but would incur a latency penalty.
Also I was unfamiliar with SSH being vulnerable in the past to keystroke timing!
please never do that (in production)
if anyone half way serious tries they _will_ be able to break you encryption end find what you typed
this isn't a hypothetical niche case obfuscation mechanism, it's a people broke SSH then a fix was found case. I don't even know why you can disable it tbh.
This should really be upstreamed as an option on the ssh library. Its good to default to sending chaff in untrusted environments, but there are plenty of places where we might as well save the bandwidth
Versus...seeing there's a vulnerability, someone adding a one-line change to disable the vulnerable algorithm, compile, image update, test. And a lot less testing because you're not moving to a new version of the language / compiler.
The man has no practical experience in running a production network service, an ego the size of a small moon, and yet was a major contributor to a security protocol now in use by billions of people.
But hey, you can be a handbag designer and end up head of design at Apple soooooooo
Sure, if it's an internal tool you can recompile both ends and force a universal update. But anything else and you need to stay compatible with clients and anytime you allow negotiation of the cryptosuit you open yourself up to quite a few subtle attacks. Not saying that choice about go is clearly a good one but i don't think it's obviously wrong.
I mean, for modern version of Openssh it's not exactly wrong. The failure was to tell you why that is the normal behavior.
WAT. Please no.
Another good trick for debugging ssh's exact behavior is patching in "None" cipher support for your test environment. It's about the same work as trying to set up a proxy but lets you see the raw content of the packets like it was telnet.
For terminal games where security does not matter but performance and scale does, just offering telnet in the first place can also be worth consideration.
Also, port 21 is often blocked.
Found your problem.
But it is an interesting world where you can casually burrow into a crypto library and disable important security features more easily than selecting the right network layer solution.
One thing you notice if you have ADSL is that some services are built as if slower connections matter and others are not. Like Google's voice and audio chat services work poorly but most of the others work well. Uploading images to Mastodon, Bluesky, Facebook, LinkedIn, Instagram and Nextdoor is reliable, but for Tumblr you have to try it twice. I don't what they are doing wrong but they are doing something wrong and not finding out what they're doing wrong because they're not testing and they're not listening to users.
Nobody consulted me about their decision not to run fiber by my house. If some committee decides to make ssh bloated they are, together with the others, conspiring to steal my livelihood and I think it would be fair for me to sue them for the $50k it would take to run that fiber myself.
It's OK if you work for Google where there is limitless dark fiber but what about people in African countries?
It's the typical corporate attitude where latency never matters: Adobe thinks it is totally normal that it takes 1-5s for a keystroke to appear when you are typing into Dreamweaver.
> And they’re sent to servers that advertise the availability of the [email protected] extension. What if we just…don’t advertise [email protected]?
Now that's solving the problem the wrong way. If you really want that, send all typed characters at 50ms intervals, to bound the timing resolution.
My thinking was:
So idk. I'm sure that you would have solved this much more quickly than I did! I'm not sure that (for me) opening up the packet in Wireshark would have solved this faster. Maybe reading the SSH spec would have, but debugging also just didn't take that long.And the big leap here was realizing that this was my SSH client and not a quirk of my game. The time at which I would have read the SSH spec was after I captured traffic from a regular SSH session and observed the same pattern; before that I was thinking about the problem wrong.
I don't think that this is unfortunate. In fact, I think I got what I wanted here (a better sense of Claude Code's strengths and weaknesses). You're right that an alternative approach would have taught me different things, and that's a worthy goal too.
[1] I suspect this is because I was doing it for an old job and I had to figure out how to run some application with keys I controlled? It would have been easier here. I don't remember.
I've used Claude a bit and it never speaks to me like that either, "Holy Cow!" etc. It sounds more annoying than interacting with real people. Perhaps AIs are good at sensing personalities from input text and doesn't act this way with my terse prompts..
Step one, run https://www.psc.edu/hpn-ssh-home/introduction/ instead Step two, tune TCP/IP stack Step... much later: write your own "crypto". (I'm using quotes because, before someone points out the obvious, packets-per-keystroke isn't, itself, a cryptographic algorithm, but because it's being done to protect connections from being decrypted/etc, mess with it at your own peril.)
Vibe coders man...
For example, "nc" (netcat) is pre-installed on all platforms where ssh is.
Switching to telnet instead of SSH might be an option.