Well this is certainly interesting, I wonder how their secrets were being stored, I assumed they were stored in vault and then injected with a sidecar, but I guess not
Seen multiple folks in the Railway Discord confirming that their secrets stored in Railway have been used by third parties, so it definitely sounds like the attackers were able to read decrypted environment variables.
Environment variables are generally fine for secrets. Environment files are problematic. Modern systems that inject secrets into the environment launch the program itself.
(pre-caveat thing to not be massively annoyingly pedantic throughout, I'm not a security buff and I know who it is I'm talking to, I'm not trying to correct them)
I'm not sure I get the difference - the issue I would have thought was having the system code have access to the secrets themselves? Env file vs env vars injected at runtime seems the same there - either can be exfiltrated. The difference would be having it injected when used.
Is the problem with files about what happens outside that, being committed, etc?
Everything can be exfiltrated. If you never store the environment in a durable file, it's not meaningfully easier to exfiltrate from the environment than from process memory. Most of the wisdom about the insecurity of environment variables falls out of people setting them in .profile or in .envrc files --- that's very bad, yes, as are configuration files (the thing 12 Factor Apps was responding to).
The headline here is that there's no reason to believe "environment variables" were the reason this data was breached; presumably, the database those variables came from was what actually got hit.
You can say "everything can be exfiltrated" but capturing my PC's password is about nigh unto impossible. The only way it can be captured is if a program has active control over the keyboard controller/associated drivers when I type in the password to unlock it.
I think some of this conversation is circling around the concept of 'allowable risk'...
We have a couple of VMs that are not mission critical, and they don't hold sensitive data. We just inject the secrets into the runtime. Access to those machines is restricted to a single IP, there's not much more protection in front of them.
A meaningful attack on one of those machines would require a LOT of effort, and almost nothing in return. Some systems don't NEED to have government level security, as a developer it's my job to provide the best solution for each scenario.
(IE, I wouldn't put 12 security cameras and an armed guard around a fruit stand, but I might around a jewelry store)
> Use environment variables to point to where secrets are located, not to store their values.
Now your apps need to be able to read the secret that is pointed to, as opposed to the task runner populating the secret and the app would not know about the details.Not to mention the changes it entails to development.
It doesn't sound like they got them via a shell environment, they just hacked into the place that stored them. If the place you store secrets can be hacked into, it doesn't really matter if they are injected into environment variables or not.
You may think that sounds like a very nice salary but $4MM annual gross is a very small business. I can’t imagine how Railway would be in business if that were considered a large customer.
Sure, I guess there are a ton of dumb people. But please name a few sites that aren't slop ecom fly by nights... I cant think of any serious business that runs on raleway.
I dont understand using any PaaS that essentially functions by arbitraging aws to people that are scared of an aws dashboard at a 3x markup.
Seems like a decent platform for toy projects not for enterprise.
Maybe I'm finally old, but I don't see how it's surprising that Raleway is at all successful. Heroku built the same business ~15 years ago and had plenty of "enterprise" clients. I don't really understand if there's that much more value in Raleway vs Vercel, and Vercel claims to have OpenAI and Meta as clients.
Never underestimate the power of good UX and tooling. I’m not really familiar with Railway, but Heroku (and Vercel) made their business around ease of use and good DX and tools. It’s sadly not a great long term business strategy because there a price point where it’s cheaper to pay someone to deal with AWS or Azure for you.
300K ARR means you get 300K in annual recurring revenue.
Subtract costs for employees. Which, given current economy and inflation, that gets you a founder scraping on their savings, one or two other people (remember that if you're paying someone unless it's totally offshored through a house which has it's own inflation, you have other factors) and then your own taxes... yeah... much more than that and you are in the black.
As a general factor I keep in mind, when I worked at a 'normal' but toxic place before I was an actual software engineer, my manager's rules were something like '40 percent L2R' (might have been lower) for in-house employees and 70-
80% L2R for contractors.
IOW, for every dollar billed for an actual employee, we paid the employee 40 cents. For every dollar billed for a contractor, we paid them 70-80 cents.
I've heard similar figures from others (e.x. 'in house employee has a real cost of 1.5-2x their actual pay').
And yes it sucks and yes the structure at least here in the US is a factor.
I do think Railway's founders have been a little overconfident with some statements like this:
Technical people should understand they're going to get their ass beat by non-technical people right now
You don't need to know how to code to write Rust anymore
Stop pearl clutching and start problem solving
I guess you can't vibe "make sure my secrets store is secure".
The uncharitable take (for them) is that it turns out, knowing how to code is important after all.
The cynical take for the state of the industry, which is maybe more aligned with Railway's statements, is that even people who do know how to code can get hacked, given how good the LLMs are at attacking targets now.
I imagine both camps would agree that without a red team that's leveraging the frontier, you're much more likely to fall victim to an attack.
I find it very funny when people put the link of their company in their bio (that's fine and normal) and next to it the ARR it brings (who cares ??). It *really* makes you look like those influencer-entrepreneur-scammer who claim to have 10 side projects making 2B ARR but still somehow spend their time selling you courses on how to be like them instead of taking care of their companies.
Your client do NOT care about your ARR (well to an extent), so why is it something you are so keen on advertising
it's called building in public, e.g. see http://trustmrr.com. they're not putting it there for clients, but for others in the huge community of indie hackers/saas entrepeneurs on x
I'm already tired, between pushing back on vibe coded internal projects being put onto the internet and then the constant barrage of CVEs that mean nothing.
Then you have real security incidents that you need to tune into constantly.
It’s funny that every security breach disclosure now has a mandatory “the attacker used AI!” as if it absolves you from having to maintain a secure system. If anything it makes the situation more damning. If it was possible to find exploits using AI then your own red teaming should have found them before the attacker did.
> your own red teaming should have found them before the attacker did
Easier said than done. The defender has to do everything right, all the time, to prevent a headline like this. The attacker only needs to be right once, and it's game over. AI or no AI.
It's a nice saying, but if you did enough things right, the attacker still needs multiple things to line up in order to get anything of value, if you get everything right. No one competent who's doing everything right is leaving open S3 buckets open to the world. Doing some things right means using multiple buckets, using multiple time limited-keys with limited read-only access, using special write-access, not having admin keys on developer laptops, a SIEM system that monitors buckets for traffic going in & out and sending emails (both amount and who it's going to) to the person on call. A firewall (inc IP-address-based blocks so eg China can't access the customer data from China at all). A WAF.
That's not an exaustive list but if the attacker gets one of those "right" once, it is not game over. (They aren't of equal strength, mind you.)
Defending is so much harder than attacking. You don't even have to focus on one target, as you can just mass-AI-script a lot of attacks wherever it's possible, and see what sticks. Even if you did everything right, there's always the social-engineering angle, or a single compromised device, or just accidental bad upgrade through supply chain. If you can mass-spread the seed, and plant something that "calls home" constantly, you can just have an agent that would start the attack the minute it gets infected, and go deeper.
It's really a wild west right now, have no idea how we fix it. I'm assuming everything will get more concentrated in big co's hands, as having a very robust security is extremely hard.
Also, we have to agree that younger people will start companies. There will be small companies that cannot afford red teams. There will be companies that are just not equipped, and their profit margins cannot sustain constant attacks.
I didn’t actually see that in this post at all. They mention that the end goal of the attackers might be to get to AI inference providers. But I don’t think they are crying “AI was used” here.
We (Railway) make no claims about the tools the attacker used in the above post.
However, from discussions with partners, it resembled the pattern of attacks to secure AI API tokens and credentials. Preliminary forensics tell us that this was the primary goal of their attack. That said, I want to be careful here commenting until we have our formal public post-mortem.
> It’s funny that every security breach disclosure now has a mandatory “the attacker used AI!”
This isn't in the article? They do not at all claim "the attacked used AI!".
The only mention of "AI" is "We believe this is part of a series of attacks recently targeting platform companies with the aim of compromising AI inference providers." This is the same (apparently) end-goal as the Opencode breach last week, and is not blaming or finger-point at AI. It is stating (rightfully) that AI inference is very easy to resell and turn stolen credentials into cash.
how are the secrets decrypted, I would like some explanation so that I can have some understanding here, I am starting to feel like all these companies are just meme-ing with their security posture.
Like did someone vibe code too hard, how do env secrets leak for everyone...
They absolutely are. Every ceo sees security as a massive money pit (it is) but there’s a reason it’s expensive and hard to do well. Those compliance frameworks they show off are meaningless and can be manipulated extremely easily. There are a lot of security leaders who are going to be having a very bad time soon.
secret encryption is always a bit of a conundrum for a running cloud system. the secret needs to be plaintext when you use it, so you have to decrypt at least on every deploy. but that means the keys have to be ready at hand, no passphrase, somewhere an automated system can get them. that makes it very hard to actually protect them from someone with full access
The security team at my company is working with a frontier lab to scan all our code/systems. They are in a borderline panic to close every possible issue, even innocuous ones that are buried under defense in depth. Whatever they are seeing from our partner has them spooked -- as in "acceptable risk" is no longer a thing anymore.
>We are going back to static htmls for public pages...
God yes, please. I backpack and mountaineer in the PNW and have been trying to research trip ideas up in British Columbia, and as a result I discovered https://bivouac.com.
Here's a website that's still frequently used by a large number of people as a valuable resource, is regularly updated and maintained, and yet looks like it's straight out of 1995.
Every pages loads instantaneously, everything is simple yet packed with detail, and it's such a basic, straightforward, delightfully pleasant experience to just browse through the site. I was so thrilled after just 5 minutes on the site that I threw them $25 for a year's full access.
Sorry for the tangent, but oh man, it's such a refreshingly stark contrast to the rest of the bullshit that fills the web these days.
I think they mean VPN as in WireGuard where each customer would have a separate private network to the service provider. This way the service wouldn't be exposed to the public Internet.
Of course, it was a giant scam that tested for nothing and optimized for how to pass a test.
Let's see if companies are ready to hire more people who know what they are doing, against a super-intelligent swarm of millions of agents with cyber capabilities.
We have gotten a lot of comments about why didn't we just message the breach sooner when people have noticed compromise with their accounts. I think that crit is fair and justified, and although we have been up for days on end- I don't blame you if you don't have sympathy for us.
Our thought process was that we didn't want to recommend that people roll keys when we couldn't possibly confirm that we confidently plugged the breach and rolled the mitigation(s). I am comfortable to publicly say that this was a highly motivated attacker. Meanwhile, our threat team mitigates DDoSes, CVEs, while been patching everything immediately as dealt with exploits.
As for this breach in particular, we tried to balance a duty to warn, which we take very seriously... However, it took a significant amount of time on the order of hours to verify, reviewing, and keep up with the attacker as they attempted to move laterally around, and briefly, within the system. We didn't want message anything beyond a shadow of a doubt that there were no further exploit paths.
We take the view that we would like to "avigate, navigate, then communicate" when we deal with incidents like these then we prepared communications to the public and all users. While it was ongoing, we were in contact with external partners and vendors to revoke keys (the source of some online chatter) with investigators that required time and coordination.
On the incident side of the house, it's a suite of bad options, we're still in the middle of it making sure customers are safe. But, hopefully this offers a view into our thought process.
Formal postmortem after we fully restore service and complete the investigation. Apologies to all, I personally I feel like I let everyone down.
IMO "aviate, navigate, then communicate" does not work in this situation. There is more than one pilot. The moment you detected a breach, hell, even knowing that an attacker was inside your system, you should have pointed to anyone. Hell, even AI, to draft a disclosure with more info coming.
"aviate, navigate, then communicate" only matters if you are the one in control of outcomes.
Customers could have simply revoked keys and not re-launched services with rolled credentials (that from your post, may have been re-stolen), but instead you let this fester.
What a disaster for Railway. I hope you get through this stressful moment, but this should be the end of Railway.
> and although we have been up for days on end- I don't blame you if you don't have sympathy for us.
From my timezone, this started last night and is ongoing into today. Has this been going for longer than the status page and email indicated? If so, why was notification delayed so long?
I love your product, have an incredible amount of sympathy for you as humans (having been in similar sleep-consuming nightmares) and was willing to forgive the GCP-triggered outage as an understandable teething pain, but sadly this response has really knocked my confidence.
I understand the hard reality of security issues -- "urgently rotate your keys, they're already compromised" is an important message to share -- but "sorry, that will also require your site going down because we've blocked deploys" (presumably as a response to the incident...) is an incredibly frustrating situation to find ourselves in.
I also want to briefly note that communicating, from the outset, "this is a significant incident, your keys are compromised, the attack is ongoing and you cannot enter new keys as _those too will be compromised_" is a better response than leaving people to figure out "my keys are compromised and oh surprise, I can't deploy".
I would rather know I should put a notice on my site, or my own social media, or notify my own customers ASAP and disable some functionality (or make my own choice about porting somewhere else as a mitigation) than get caught out like this after the horse has bolted and my key has been compromised for hours.
Railway / OpenAI: I had a key compromised this morning. It hit our limit, so I revoked the key... and figured, well, we'll have to track down the leak.
Then this security notice came out, telling all of their customers to rotate all their keys... As the guy whose spent the entire evening rotating keys and etc. doing
this is... it's a huge message to send.
The cost and implications go really far.
We have many production services that we've taken offline or disabled because we can't deploy to fix the credentials. because of the railway deploy problem...
Railway was stuck between some pretty nasty tradeoffs. I get that the few hour delay in communication caused some of us more cost, issues, and potential breach... and maybe larger suff.
But I also realize that rolling the keys while things may still be compromised, with deployments blocked, could make thins even more sticky
leaves only very bad options. Also we don't know how much they we're spending time actively stopping the attack. the time and focus could be very much consumed. if it was 5% of exposure and they prevented the 95% exposure.. i'm sure glad they aviated instead of communicated.
We make the best decision we can in these moments.. and they all just suck.
I was a loyal railway customer for a while, but this response smacks of "we didn't want people to know" instead of "we wanted to take responsibility."
The options railway has given us are "rotate your keys and bring down your service" or "keep your service up and let the exposed keys remain at risk." Kind of a crap position, no? Mistakes are going to happen, but when the suggested mitigations are to stab yourself in the leg or stab yourself in the ass, it's hard to be sympathetic.
I have to imagine quite a few users are asking their agents to port their stacks over to alternate providers like I am. The cost of switching has never been cheaper thanks to AI.
> Our thought process was that we didn't want to recommend that people roll keys when we couldn't possibly confirm that we confidently plugged the breach and rolled the mitigation(s). I am comfortable to publicly say that this was a highly motivated attacker. Meanwhile, our threat team mitigates DDoSes, CVEs, while been patching everything immediately as dealt with exploits.
> As for this breach in particular, we tried to balance a duty to warn, which we take very seriously... However, it took a significant amount of time on the order of hours to verify, reviewing, and keep up with the attacker as they attempted to move laterally around, and briefly, within the system. We didn't want message anything beyond a shadow of a doubt that there were no further exploit paths.
Once you realize there is compromise, you message accordingly, as soon as you are able. If you can't immediately determine exactly what subset of customers were affected, this means you have to tell everyone. Then you provide follow-up updates to fill in gaps as you develop your understanding of scope+impact and work on remediation.
What you should never do is hold off on reaching out to possibly affected stakeholders that they may be affected by a compromise because you still have uncertainties about current status.
A situation like this does not in any way a constitute a dilemma or tough balance regarding when or to whom to do initial disclosure - you do it as soon as you can, to everyone except those you know are not impacted.
Right just as I take my first day off (for the entire year!) from work to fly out for a vacation. The mother of all security bombs starts dropping all around stacks and services that I am responsible for. How else could have taken time off go for me…
Thanks, yes I will. Left without the laptop(s). Let the chips fall where they lay. If they need me back so urgently, they will have to go through the full cancellation of booked flights and hotels first. Lets enjoy the time off.
Yes. Basically, if there is 1 person that knows all the critical stuff and you will have BIG issues after they get hit by a bus... well, the rest is self-explanatory. :-)
Brother do not cancel anything for these people, they do not care about you, it's their fault for not considering bus factor in their org chart. Why can't the person above you handle that?
Rob: They're not real. You get that, right? None of it is real. The critics aren't real. The customers aren't real. Because... this isn't real.
Chef Finway: [laughing awkwardly] Okay...
Rob: Derek, why do you care about these people? They don't care about you. None of them. They don't even know you because you haven't shown them. Every day, you'll wake up, and there'll be less of you. You live your life for them, and they don't even see you. You don't even see yourself.
[long pause; Derek's tight smile has slowly faded into a look of sorrow and regret]
Rob: We don't get a lot of things to really care about.
I have used it because of the nicer developer experience and it's a little bit more friendly to my clients who find AWS too confusing to navigate; but unfortunately this has destroyed all of my confidence in Railway. Nothing personal, and I'm sympathetic to the engineers who are navigating this situation the best they can. I simply refused to believe something this egregious would happen with Secrets Manager in AWS
If you are open to self hosting and want a railway like experience, i have been building stackdome: https://github.com/axernel-labs/stackdome . Its mit-licensed and can run on cheap vps's to large k8s clusters.
What about buckets? I tried to regenerate my secrets but deploys are paused. I don't know if they actually rotated or not because the old keys seem to still be working! Dear attackers, please don't read or mutate my files while my app is offline :)
> Earlier this morning, Railway detected anomalous activity involving systems that store customer environment variables. ... Environment variables may contain secrets such as API keys or database and registry credentials.
Can I ask why the *$@%@^T you would trust somebody else's server with this task? Has software engineering really gotten this lazy and stupid, or am I missing something about this being a good idea? I saw a tech-focused social media account advocate doing this, and I came close to coming after him in comments, but I decided not to; I figured this is just one of those things the kids are doing that I'll never understand.
”An external incident response firm reviewed PingCAP’s production systems and found no evidence that the attacker ran commands or maintained persistent access in the systems reviewed”
The evidence is in the blog post explaining how they did it. However seeing at how bad their operations were I doubt they have evidence of anything themselves
Being that guy, I say cybersecurity is an oxymoron with the messy and heterogenous implementation of identifying the authorised actor. For example passwords are known like two decades or more being frail and vulnerable, yet that's the backbone of security after all those years? So it had to be patched and scaffolded with various techniques like 2FA and one time codes. Still, different ways in all places, so users resort putting all their eggs in one basket just to be able to get moving, storing secrets and keys in one place, handing over to someone on the internet, while calling it progress and the edge of technology. It feels like for a frustrated user who needs to fight through daily identifications in at least one or two of the dozens of systems (distracting from productive work) that the poor person is forced to use as everything must be put on the net regardless if it is better that way or not, it is modern and good for the developer. All in the name of security, so poor user starts to write things down or put all the eggs in one basket in other ways, aggravating the password situation and exposing/risking his life and livelihood. As we see from the frustrated folks in various threads here on HN too.
I do not thing there is (any) security in cyberspace, and this is exposed big now with the use of AI.
I use linode and have nothing but praises for their service.
>Actually I will move my projects away from Railway
If you want a railway like experience, i have been building an opensource alternative to railway: https://github.com/axernel-labs/stackdome . Its mit-licensed and can run on a single vps to full-scale k8s clusters.
288 comments
[ 6.3 ms ] story [ 139 ms ] threadI'm just going to say it, since having said it non-stop for over 20 years, nobody seems to listen and we keep circling this drain.
The 12 Factor manifesto was 100% wrong on this point.
Use environment variables to point to where secrets are located, not to store their values.
I'm not sure I get the difference - the issue I would have thought was having the system code have access to the secrets themselves? Env file vs env vars injected at runtime seems the same there - either can be exfiltrated. The difference would be having it injected when used.
Is the problem with files about what happens outside that, being committed, etc?
The headline here is that there's no reason to believe "environment variables" were the reason this data was breached; presumably, the database those variables came from was what actually got hit.
These problems were all solved a long time ago.
We have a couple of VMs that are not mission critical, and they don't hold sensitive data. We just inject the secrets into the runtime. Access to those machines is restricted to a single IP, there's not much more protection in front of them.
A meaningful attack on one of those machines would require a LOT of effort, and almost nothing in return. Some systems don't NEED to have government level security, as a developer it's my job to provide the best solution for each scenario.
(IE, I wouldn't put 12 security cameras and an armed guard around a fruit stand, but I might around a jewelry store)
Now your apps need to be able to read the secret that is pointed to, as opposed to the task runner populating the secret and the app would not know about the details.Not to mention the changes it entails to development.
I don't have to enter my Mac's password every time I open an app.
https://x.com/wickedguro/status/2113015432974405658
I dont understand using any PaaS that essentially functions by arbitraging aws to people that are scared of an aws dashboard at a 3x markup.
Seems like a decent platform for toy projects not for enterprise.
https://railway.com/customers
tripadvisor and bilt, maybe?
It's pretty common to have "toy" infrastructure in a non-tech business
Of course "uses" doesn't mean they run critical services.
Median annual revenue for a small business (or just business) is a paltry $280k.
Subtract costs for employees. Which, given current economy and inflation, that gets you a founder scraping on their savings, one or two other people (remember that if you're paying someone unless it's totally offshored through a house which has it's own inflation, you have other factors) and then your own taxes... yeah... much more than that and you are in the black.
As a general factor I keep in mind, when I worked at a 'normal' but toxic place before I was an actual software engineer, my manager's rules were something like '40 percent L2R' (might have been lower) for in-house employees and 70- 80% L2R for contractors.
IOW, for every dollar billed for an actual employee, we paid the employee 40 cents. For every dollar billed for a contractor, we paid them 70-80 cents.
I've heard similar figures from others (e.x. 'in house employee has a real cost of 1.5-2x their actual pay').
And yes it sucks and yes the structure at least here in the US is a factor.
The cynical take for the state of the industry, which is maybe more aligned with Railway's statements, is that even people who do know how to code can get hacked, given how good the LLMs are at attacking targets now.
I imagine both camps would agree that without a red team that's leveraging the frontier, you're much more likely to fall victim to an attack.
Then you have real security incidents that you need to tune into constantly.
Easier said than done. The defender has to do everything right, all the time, to prevent a headline like this. The attacker only needs to be right once, and it's game over. AI or no AI.
That's not an exaustive list but if the attacker gets one of those "right" once, it is not game over. (They aren't of equal strength, mind you.)
It's really a wild west right now, have no idea how we fix it. I'm assuming everything will get more concentrated in big co's hands, as having a very robust security is extremely hard.
Also, we have to agree that younger people will start companies. There will be small companies that cannot afford red teams. There will be companies that are just not equipped, and their profit margins cannot sustain constant attacks.
Or did I miss something?
However, from discussions with partners, it resembled the pattern of attacks to secure AI API tokens and credentials. Preliminary forensics tell us that this was the primary goal of their attack. That said, I want to be careful here commenting until we have our formal public post-mortem.
This isn't in the article? They do not at all claim "the attacked used AI!".
The only mention of "AI" is "We believe this is part of a series of attacks recently targeting platform companies with the aim of compromising AI inference providers." This is the same (apparently) end-goal as the Opencode breach last week, and is not blaming or finger-point at AI. It is stating (rightfully) that AI inference is very easy to resell and turn stolen credentials into cash.
the speed of irresponsible deployments is infinite now, by design
Like did someone vibe code too hard, how do env secrets leak for everyone...
Fwiw it sounds similar to the hack over the weekend that opencode suffered.
B2C? too risky to publish services online haha.
http://www.tarsnap.com/spiped.html
God yes, please. I backpack and mountaineer in the PNW and have been trying to research trip ideas up in British Columbia, and as a result I discovered https://bivouac.com.
Here's a website that's still frequently used by a large number of people as a valuable resource, is regularly updated and maintained, and yet looks like it's straight out of 1995.
Every pages loads instantaneously, everything is simple yet packed with detail, and it's such a basic, straightforward, delightfully pleasant experience to just browse through the site. I was so thrilled after just 5 minutes on the site that I threw them $25 for a year's full access.
Sorry for the tangent, but oh man, it's such a refreshingly stark contrast to the rest of the bullshit that fills the web these days.
Not sure about VPN. Seems likely to provide a false sense of security.
Let's see if companies are ready to hire more people who know what they are doing, against a super-intelligent swarm of millions of agents with cyber capabilities.
We have gotten a lot of comments about why didn't we just message the breach sooner when people have noticed compromise with their accounts. I think that crit is fair and justified, and although we have been up for days on end- I don't blame you if you don't have sympathy for us.
Our thought process was that we didn't want to recommend that people roll keys when we couldn't possibly confirm that we confidently plugged the breach and rolled the mitigation(s). I am comfortable to publicly say that this was a highly motivated attacker. Meanwhile, our threat team mitigates DDoSes, CVEs, while been patching everything immediately as dealt with exploits.
As for this breach in particular, we tried to balance a duty to warn, which we take very seriously... However, it took a significant amount of time on the order of hours to verify, reviewing, and keep up with the attacker as they attempted to move laterally around, and briefly, within the system. We didn't want message anything beyond a shadow of a doubt that there were no further exploit paths.
We take the view that we would like to "avigate, navigate, then communicate" when we deal with incidents like these then we prepared communications to the public and all users. While it was ongoing, we were in contact with external partners and vendors to revoke keys (the source of some online chatter) with investigators that required time and coordination.
On the incident side of the house, it's a suite of bad options, we're still in the middle of it making sure customers are safe. But, hopefully this offers a view into our thought process.
Formal postmortem after we fully restore service and complete the investigation. Apologies to all, I personally I feel like I let everyone down.
Customers could have simply revoked keys and not re-launched services with rolled credentials (that from your post, may have been re-stolen), but instead you let this fester.
What a disaster for Railway. I hope you get through this stressful moment, but this should be the end of Railway.
From my timezone, this started last night and is ongoing into today. Has this been going for longer than the status page and email indicated? If so, why was notification delayed so long?
I love your product, have an incredible amount of sympathy for you as humans (having been in similar sleep-consuming nightmares) and was willing to forgive the GCP-triggered outage as an understandable teething pain, but sadly this response has really knocked my confidence.
I understand the hard reality of security issues -- "urgently rotate your keys, they're already compromised" is an important message to share -- but "sorry, that will also require your site going down because we've blocked deploys" (presumably as a response to the incident...) is an incredibly frustrating situation to find ourselves in.
I would rather know I should put a notice on my site, or my own social media, or notify my own customers ASAP and disable some functionality (or make my own choice about porting somewhere else as a mitigation) than get caught out like this after the horse has bolted and my key has been compromised for hours.
The options railway has given us are "rotate your keys and bring down your service" or "keep your service up and let the exposed keys remain at risk." Kind of a crap position, no? Mistakes are going to happen, but when the suggested mitigations are to stab yourself in the leg or stab yourself in the ass, it's hard to be sympathetic.
I have to imagine quite a few users are asking their agents to port their stacks over to alternate providers like I am. The cost of switching has never been cheaper thanks to AI.
> As for this breach in particular, we tried to balance a duty to warn, which we take very seriously... However, it took a significant amount of time on the order of hours to verify, reviewing, and keep up with the attacker as they attempted to move laterally around, and briefly, within the system. We didn't want message anything beyond a shadow of a doubt that there were no further exploit paths.
Once you realize there is compromise, you message accordingly, as soon as you are able. If you can't immediately determine exactly what subset of customers were affected, this means you have to tell everyone. Then you provide follow-up updates to fill in gaps as you develop your understanding of scope+impact and work on remediation.
What you should never do is hold off on reaching out to possibly affected stakeholders that they may be affected by a compromise because you still have uncertainties about current status.
A situation like this does not in any way a constitute a dilemma or tough balance regarding when or to whom to do initial disclosure - you do it as soon as you can, to everyone except those you know are not impacted.
Chef Finway: [laughing awkwardly] Okay...
Rob: Derek, why do you care about these people? They don't care about you. None of them. They don't even know you because you haven't shown them. Every day, you'll wake up, and there'll be less of you. You live your life for them, and they don't even see you. You don't even see yourself.
[long pause; Derek's tight smile has slowly faded into a look of sorrow and regret]
Rob: We don't get a lot of things to really care about.
Can I ask why the *$@%@^T you would trust somebody else's server with this task? Has software engineering really gotten this lazy and stupid, or am I missing something about this being a good idea? I saw a tech-focused social media account advocate doing this, and I came close to coming after him in comments, but I decided not to; I figured this is just one of those things the kids are doing that I'll never understand.
And it’s all just getting started imo.
https://docs.gitlab.com/releases/patches/patch-release-gitla...
[0] https://news.ycombinator.com/item?id=50173716
”An external incident response firm reviewed PingCAP’s production systems and found no evidence that the attacker ran commands or maintained persistent access in the systems reviewed”
hmm, I don't think that's true.
PingCap TiDB Leaks - https://news.ycombinator.com/item?id=50173716 - Oct 2026 (32 comments)
(It had only spent 3 hours on the frontpage so I re-upped it.)
I do not thing there is (any) security in cyberspace, and this is exposed big now with the use of AI.
Is that an accusation? As in, major AI players taking out competitors ?
What is the best VPS to use today? Is it still hetzner or ovh?
>Actually I will move my projects away from Railway
If you want a railway like experience, i have been building an opensource alternative to railway: https://github.com/axernel-labs/stackdome . Its mit-licensed and can run on a single vps to full-scale k8s clusters.