Here's probably a silly question: Shouldn't this work automatically? I just assumed they would have an intermediate CA or whatever it's called and have that certificate be signed by some widely trusted CA.
Or have they done it in a different way for security reasons?
For mysql at least you need to specify what CA or cert to trust. There is no default CA which mysql will try to use. So once a CA expires you will have to rotate it
Not silly at all. Certainly there are some database clients that will access a system default CA trust store. ISTR the PostgreSQL JDBC client may have this behaviour, although perhaps depending on which Java runtime & packaging is in use. And there'll be others besides.
And behold, Amazon are themselves a globally trusted root CA! (Have a peek in /etc/ssl/certs, on must current 'nixen I would expect to see the four Amazon roots). So why not have the Amazon root sign the RDS CA certificate as an intermediate CA, so that clients who could trust an RDS-issued cert that way, will magically trust it?
I've met some of the RDS team and they are smart cookies. If you will agree to my use of cookies, then I can think of at least three good enough reasons (and just to be clear, this is speculation and extrapolation, not insider chapter and verse):
1. Changes in behaviour during another potentially disruptive change are the province of the unwise, the unwary, and the otherwise soon to be fired for cause.
2. Not all clients can use a default trust store; even different clients for the same database may differ. For example, the PostgreSQL C client (as used by the command-line tools and many language bindings) cannot access the system certificate store; the bundle must be configured explicitly. For one client to have a magic behaviour that others do not, in the realm of cryptography and trust, is going to make security admins and auditors very unhappy. Inconsistency is the enemy of policy, and policy makes the DevSecOps world go round.
3. Not all clients support a certificate chain, so the RDS team already has to maintain a bundle of the regional intermediates. Making the RDS CA an intermediate just adds more complication to that activity, more special cases to document, and more potential for confusion in client configurations.
So these are security reasons, although they're more from a policy and human factors and software inadequacy perspective than anything to do with, say, large prime numbers. It's quite possible the service team have more reasons besides; I would certainly agree to their use of cookies or indeed any baked product. For me, those reasons alone would suffice, and I am not surprised to see that the new RDS root is another standalone self-signed CA.
As ever, the #1 tip to maximise confidence in your configuration is to automate it, write a failing test case, then fix the test.
> Any specific examples?
Some of the new-wave SSL implementations (including yaSSL, with which mysql-client can/could be built) didn't always support them (not sure where yaSSL is at now), and the AWS documentation hints that it's cropped up in some circumstance in GovCloud, but I think the most likely circumstances will be enterprise folks using appliances and/or appliance-like software that's configured only via a web UI, one in which they just didn't think chains were necessary (or that only supports a small bundle etc etc; the sky's the limit in this class of software gore). I've seen this exact issue, albeit for an LDAP client not a RDBMS.
Re: anything w/ security (esp how it works), if you don't know ASK. ASK. ASK.
PLEASE ASK.
Security is incredibly complex and minor/subtle mistakes can destroy companies and other potentially catastrophic consequences (especially to your employment.) If you're not sure about something, ASK. End-users are bad enough when it comes to security, but engineers should always feel empowered to "make sure" about all matters of security, large or small.
Obviously it's best to to be familiar with best practices, but if you're not a security "expert" and you're unclear about something, ASK.
There no dumb questions in security, just dumb practices. The dumbest thing of all is not asking your question because you think it's "obvious."
> Security is incredibly complex and minor/subtle mistakes can destroy companies
For better or for worse, that's not the case. These days, security mistakes just end up in 2 years of free credit monitoring. Ironic case in point, Equifax.
I'm not saying you shouldn't take security seriously. But it's not fair to say that insecurity could destroy your company. Mainly, most end users just really don't care (or have a choice).
It depends on the company. What you say is true for a consumer website, but much less true for a SaaS product which depends on customer trust. For example, imagine an AWS or Salesforce or Github security breach that released very private business data.
> Here's probably a silly question: Shouldn't this work automatically? I just assumed they would have an intermediate CA or whatever it's called and have that certificate be signed by some widely trusted CA.
Yeah, that would work if someone at AWS would make a bit of sense and would sign them using their pubic CA, but they decided not to and instead generate new RDS CA every 5 years.
Anyway at least with PostgreSQL clients that rely on libpq (the main postgres client library) by default while it prefers SSL when available it won't verify the certificate unless you explicit use verify-ca or verify-full sslmode.
> Regions – Rotation is needed for database instances in all commercial AWS regions except Asia Pacific (Hong Kong), Middle East (Bahrain), and China (Ningxia).
I wonder why?
Are these CAs subject to... shall we say, additional goverment oversight?
Yes, but if they're already in the network (although I've nothing to suggest they are), MITMing the database connection means they get the data immediately, rather than at the next quasi-judicial request.
Yeah, Ningxia is a weird region. It’s run under license by a Chinese company, and KMS isn’t available, which by extension means most encryption isn’t either since that’s all built on top of KMS these days.
Oh, fair enough, last time I looked into the China regions that didn't exist. Reading the announcement I'm loving the seeming contradiction between "
AWS KMS is a managed service that makes it easy for you to create and control the encryption keys used to encrypt your data, and uses Chinese government-approved hardware security modules to protect the security of your keys." and the very next sentence "The service is designed so that no one, including KMS service operators, can retrieve your plaintext master keys from the service."
Something tells me those master keys aren't quite as secure as you'd expect in other regions.
I am ex-AWS (forced out after inadvertently hiring and developing the Beast) and therefore qualified to comment. There's some mixup here I think, it's James Hamilton that owns a ship. Some cynics say that Jeff Bezos started Blue Origin just to upstage Hamilton, since Jeff can now say he owns a space ship. Even that isn't true, though. Jeff was just trying to earn thrust, as part of a long-term fitness programme - I'm sure you've seen the candid images of his guns - to get lean and bi curious.
-----
edit/codicil: the irony seemed fitting when the parent comment was published. Sadly for those of us willing to risk satirical puns on Hacker News when faced with an irresistible prompt, the context, being an earnestly hagiographical quotation from the Amazon Leadership Principles, was edited away.
They're different in more ways than you think. Fun fact, Jeff Bezos passed on acquiring Whatsapp because he's secretly biased against Acton. Talk about getting your signals crossed.
> Next, I update my client applications to use the new certificates. This process is specific to each app and each database client library, so I don’t have any details to share.
This is how I lost the last two days of work. Learned a lot at least.
Is it possible to simply concatenate the two `bundle` files (2015 and 2019) and feed that to client apps? That would allow updating the client first without changing the db, then updating the db without breaking the client. (Then step 3 would be to update the client to just use the 2019 bundle)
I created a new RDS instance via Cloudformation just last week and immediately had the notice that I should update the cert. Looks like the new cert will be the default on 1/14, but I thought that was really weird.
In the spirit of asking silly questions (as encouraged in some comments here), here's mine:
My small SaaS company's PostgreSQL RDS instance and app servers are in a VPC with security groups configured to only allow connections from the app servers to the DB (no public access to the RDS instance). My client (ruby-pg) on the app servers is connecting via SSL, but not currently with certificate validation (though I believe the cert date still needs to be valid [?], hence the need to rotate the PostgreSQL server's certificate).
In this scenario, how important is certificate validation? I understand the theoretical risk of clients not being able to fully trust that they're connected to the database I intend, but from a practical standpoint, it seems that if an attacker is able to poison the VPC's DNS and trick the app servers into connecting to something else, I'm already hosed and cert validation wouldn't do much to help me. Am I missing something obvious and very dangerous?
AWS claims that VPC traffic is immune from MiTM attacks. [0] (Although they strangely don't talk about this much, and I can't even find official documentation of it.) So within a VPC, you can forgo certificate validation.
However, in general certificate validation would protect you. The attacker wouldn't have a valid certificate, so your application would refuse to connect to the malicious endpoint.
So basically they intercept all ARP communication and add VPC header information to all Ethernet packets, which should rule out any MITM scenario.
Of course the keyword here is “should”, and depending on your threat mode you may or may not want to have an additional defense layer of SSL certificate validation.
35 comments
[ 2.9 ms ] story [ 87.2 ms ] threadOr have they done it in a different way for security reasons?
And behold, Amazon are themselves a globally trusted root CA! (Have a peek in /etc/ssl/certs, on must current 'nixen I would expect to see the four Amazon roots). So why not have the Amazon root sign the RDS CA certificate as an intermediate CA, so that clients who could trust an RDS-issued cert that way, will magically trust it?
I've met some of the RDS team and they are smart cookies. If you will agree to my use of cookies, then I can think of at least three good enough reasons (and just to be clear, this is speculation and extrapolation, not insider chapter and verse):
1. Changes in behaviour during another potentially disruptive change are the province of the unwise, the unwary, and the otherwise soon to be fired for cause.
2. Not all clients can use a default trust store; even different clients for the same database may differ. For example, the PostgreSQL C client (as used by the command-line tools and many language bindings) cannot access the system certificate store; the bundle must be configured explicitly. For one client to have a magic behaviour that others do not, in the realm of cryptography and trust, is going to make security admins and auditors very unhappy. Inconsistency is the enemy of policy, and policy makes the DevSecOps world go round.
3. Not all clients support a certificate chain, so the RDS team already has to maintain a bundle of the regional intermediates. Making the RDS CA an intermediate just adds more complication to that activity, more special cases to document, and more potential for confusion in client configurations.
So these are security reasons, although they're more from a policy and human factors and software inadequacy perspective than anything to do with, say, large prime numbers. It's quite possible the service team have more reasons besides; I would certainly agree to their use of cookies or indeed any baked product. For me, those reasons alone would suffice, and I am not surprised to see that the new RDS root is another standalone self-signed CA.
I just put my root certs in ~/.postgresql/root.crt and it all works, right? (unless the root itself changed, but this should be stable)
> Not all clients support a certificate chain
Any specific examples? psql and PostgreSQL JDBC both support chains.
What would even cause that? Handrolling an SSL library?
Also ensure sslmode is set to verify-ca or verify-full, either as a set parameter or via the PGSSLMODE environment variable. See https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/ssl-c...
As ever, the #1 tip to maximise confidence in your configuration is to automate it, write a failing test case, then fix the test.
> Any specific examples?
Some of the new-wave SSL implementations (including yaSSL, with which mysql-client can/could be built) didn't always support them (not sure where yaSSL is at now), and the AWS documentation hints that it's cropped up in some circumstance in GovCloud, but I think the most likely circumstances will be enterprise folks using appliances and/or appliance-like software that's configured only via a web UI, one in which they just didn't think chains were necessary (or that only supports a small bundle etc etc; the sky's the limit in this class of software gore). I've seen this exact issue, albeit for an LDAP client not a RDBMS.
Re: anything w/ security (esp how it works), if you don't know ASK. ASK. ASK.
PLEASE ASK.
Security is incredibly complex and minor/subtle mistakes can destroy companies and other potentially catastrophic consequences (especially to your employment.) If you're not sure about something, ASK. End-users are bad enough when it comes to security, but engineers should always feel empowered to "make sure" about all matters of security, large or small.
Obviously it's best to to be familiar with best practices, but if you're not a security "expert" and you're unclear about something, ASK.
There no dumb questions in security, just dumb practices. The dumbest thing of all is not asking your question because you think it's "obvious."
tl;dr: ASK!
For better or for worse, that's not the case. These days, security mistakes just end up in 2 years of free credit monitoring. Ironic case in point, Equifax.
I'm not saying you shouldn't take security seriously. But it's not fair to say that insecurity could destroy your company. Mainly, most end users just really don't care (or have a choice).
Yeah, that would work if someone at AWS would make a bit of sense and would sign them using their pubic CA, but they decided not to and instead generate new RDS CA every 5 years.
Anyway at least with PostgreSQL clients that rely on libpq (the main postgres client library) by default while it prefers SSL when available it won't verify the certificate unless you explicit use verify-ca or verify-full sslmode.
I wonder why?
Are these CAs subject to... shall we say, additional goverment oversight?
Something tells me those master keys aren't quite as secure as you'd expect in other regions.
-----
edit/codicil: the irony seemed fitting when the parent comment was published. Sadly for those of us willing to risk satirical puns on Hacker News when faced with an irresistible prompt, the context, being an earnestly hagiographical quotation from the Amazon Leadership Principles, was edited away.
This is how I lost the last two days of work. Learned a lot at least.
My small SaaS company's PostgreSQL RDS instance and app servers are in a VPC with security groups configured to only allow connections from the app servers to the DB (no public access to the RDS instance). My client (ruby-pg) on the app servers is connecting via SSL, but not currently with certificate validation (though I believe the cert date still needs to be valid [?], hence the need to rotate the PostgreSQL server's certificate).
In this scenario, how important is certificate validation? I understand the theoretical risk of clients not being able to fully trust that they're connected to the database I intend, but from a practical standpoint, it seems that if an attacker is able to poison the VPC's DNS and trick the app servers into connecting to something else, I'm already hosed and cert validation wouldn't do much to help me. Am I missing something obvious and very dangerous?
However, in general certificate validation would protect you. The attacker wouldn't have a valid certificate, so your application would refuse to connect to the malicious endpoint.
[0] https://kevin.burke.dev/kevin/aws-alb-validation-tls-reply/
https://twitter.com/0xdabbad00/status/995833462660612096
Of course the keyword here is “should”, and depending on your threat mode you may or may not want to have an additional defense layer of SSL certificate validation.