My feeling is that Malone has misconstrued a bunch of Colm MacCárthaigh's arguments. I will now do the same thing, by attempting to summarize MacCárthaigh's arguments myself.
First: TLS (and mTLS) create secure channels. A channel bears many requests and responses. In many designs, a single channel will bear requests for many different users. The job of a textbook secure channel is to prevent an attacker with control of the underlying network from tampering with requests and responses. Binding identities and authorization claims to requests is a different job. The textbook solution to that problem is authenticated requests, not authenticated secure channels. I wouldn't have used the word "layering violation", which I would contend Is Not A Thing, but the point MacCárthaigh makes is important. So: when MacCárthaigh says SQLI and Request Smuggling are "still things", he's not saying that mTLS introduces SQLI, but rather that authenticating at the level of secure channels means that a request smuggling attack is almost automatically a game-over for your application, because you aren't authenticating the requests independently of the channel. Even Basic Auth can potentially avoid that problem, if an attacker has to know a secret to slip into a request in order to forge a request on a compromised channel.
Second: TLS was designed for the WebPKI, and that's its most important application. A consequence of this is that a lot of TLS software is designed for WebPKI threat models. Like everything else the IETF produces, TLS has features that support a lot of other applications, like using heartbeat messages to debug remote processes. But what matters in the real world is the installed base of TLS software, and that installed base does not have sane defaults and safe ergonomics for mTLS. You have to be extra careful rolling it out that way.
Third: TLS is built on X.509, and X.509 is extremely complex and error prone. This complexity is deceptive, because we've spent 20 years filing down the complexity of X.509 in the WebPKI deployment mode, and developers mostly don't have to care about anything except the filenames for their certificates and keys. That's out the window with mTLS, where peers actually have to crack open certificates and look at them to accomplish authn and authz tasks.
Fourth: Revocation is a debacle. It's a debacle everywhere, but TLS complexifies it, because it has to work at Internet scale. If you're managing a fleet for even a large, complex application, you have recourse to designs that make revocation easier to handle.
Fifth: Fleet-wide credential rotation is much harder in mTLS than it is in simpler systems, where the root of trust might just be a simple secret you can quickly load on to a given machine and tabletop how to do a fleetwide deployment in a quick meeting. You can add mechanism to automate certificate issuance, but those systems have to be resilient to the total loss of the root of trust, which is something the TLS ecosystem is not generally good at; the WebPKI could suffer a total loss of all trust roots if we broke RSA, but not so much if we just broke OpenSSL.
Sixth: The standard modern solution to revocation, short-lifetime certificates, also relies on secure clocks. Breaking clocks Internet-wide might be difficult enough to be outside a reasonable threat model, but breaking clocks in a single data center is not.
Seventh: To make short-expiry work in a setting where a compromised certificate isn't just a potential coffee shop MITM for a subset of your users but instead a game-over compromise for the entire application, you might need lifetimes so short they're hard to operationalize without outages. Part of what we like about public key authentication schemes is that they don't require continuous access to a central trusted authority just to keep the system up and running!
Eighth: Serious applications want queryable audit trails for inter-service RPCs, which means having ready, oft...
Reading it over, I think we mostly agree on the facts. It's easy to do mTLS and x509 wrong. The question, then, is what's easier / more secure: doing mTLS/x509 right or doing something else? I think that's somewhat subjective: it depends on your requirements, your environment, and your skillset.
One point that I'd like to reiterate is this: if you want a consistent cryptographic solution that works everywhere, TLS is pretty much your only choice. You could use something else for client authentication, but you probably still need TLS.
As a strawman, here's a sketch of how I'd recommend doing TLS in a microservice system. I consider this "right" for most garden-variety microservices-in-cloud scenarios and don't think it's particularly hard to do. Most of this is already implemented in https://github.com/smallstep/certificates:
* Deploy the root cert via automation (so it's quickly rotatable) and/or keep it in a managed HSM/KMS. You might harden root rotation a bit by signing your new root with your old root. But, generally, trust config management or container orchestration to push root(s) (you already trust it to push code and secrets). Root rotation (and, thus, bulk revocation) is now as fast as secret rotation (secrets are generally pushed the same way).
* Issue short-lived certificates per logical entity. If it gets a box and a name in your architecture diagram, it should get an identity and each instance should get a certificate. Use domain names and email addresses that you control for names. Keep certs simple: one SAN. Certificates bind a name to a public key. That's it.
* Automate certificate issuance. ACME can work for this, but there are other options (single-use tokens issued by config management, cloud-managed instance identity documents or service accounts, an existing device certificate issued by a manufacturer, etc.)
* Automate certificate renewal. A simple mTLS HTTPS request works for this. This is easy to implement and easy to scale out with multiple intermediates. "Revoking" a certificate just marks it as "not renewable". To reduce risk of outage, in this architecture, it's safe to renew an expired certificate as long as it's not revoked (ACME-STAR basically does this, but it's push instead of pull).
* If you really need active revocation, fine. One good solution is to push CRL to a cloud storage bucket. Short-lived certs will keep your CRLs small. If you need to do a mass rotation, rotate roots (push new root, wait for rotation, pull old root).
* Use secure NTP for time.
* Index issued certificates. CT (trillian) is cool if you want to be fancy. Your existing database or SIEM also works. zcertificate can parse x509 and output a JSON representation of a certificate that you can map to something like an Elastic Search schema: https://github.com/zmap/zcertificate
I want to respond specifically to your first and final points.
On your first point: I understand that in theory an attacker could slip a request across a secure channel, and binding authentication to a request could in theory prevent that. I don't understand how that's likely to happen in the context I'm thinking of here. Which may be different than the context you're thinking of. So let me clarify.
Suppose I have `<end-user> -> <service-a> -> <service-b> -> <database>`. Let's focus on `<service-a> -> <service-b>`. I don't see how using end-to-end mTLS, terminating in `<service-a>` and `<service-b>` application code, would be any more vulnerable to this variety of attack than an HTTP Basic header like `Authorization: Basic base64(service-a:password)`. Surely, the logic in `<service-a>` is simply "insert HTTP Basic...
At some point, in most designs, you're going to end up with a client and server service where the server does something on behalf of a user. The identities in this design now include [client-identity, server-identity, user-identity].
I'll stipulate to mTLS resolving client-identity and server-identity. But the client is making requests of the server that pertain to a user. The certificate doesn't (and can't) attest to a user.
In most designs I've actually evaluated, what ends up happening is that the server just trusts the user provided by the client, and the client tries hard not to ask for things for the wrong user. But beyond whether the client has authorized every code path that generates a request, you have an additional problem here, because even after the client authorizes a path that generates a request, a bug in the client can give an attacker influence over the request itself. The server has no way of distinguishing between a corrupted request and a real request, even though you're relying on a secure channel.
Authenticated requests mitigate this problem: an attacker might be able to corrupt a request, but it's not enough to corrupt it; you need to know whatever secret authenticates the request itself.
To do this right, you now need two authentication schemes in play: one for the secure channel, and one for the requests themselves. But if you can reliably authenticate requests, why are we rigorously authenticating the secure channel? We're spending complexity chits to buy only marginal extra security.
I think it can make some sense to mesh up services with mTLS as a "you must be this high to get on the ride" mechanism. But since that's all it's doing, we don't need a lot of complicated mechanism to give precisely the right certificates to services, because even if something gets screwed up, request authentication, not secure channel authentication, should be what's protecting your application.
Especially when we think about short-lived certificates and ACME and rollover: what is this really buying us in a strong system with request authentication? A WebPKI ACME certificate will live for 90 days. If certificates matter to our design, we can't let a compromised cert live for 90 days. Our certificate lifespans have to be much shorter --- the duration in which we'd be OK having a compromised credential still viable for an attacker. Remember the differing consequences between a WebPKI certificate compromise and an internal credential compromise: one is a second-order flaw that lets attackers with some other vulnerability MITM a subset of users; the other is a game-over flaw. How long would you let a compromised developer prod SSH key live?
So now we're talking about very short-lived certificates, and you can see, this is a design that spends all the complexity chits asymmetric encryption costs, but is asymptotically approaching the operational resiliency of Kerberos. Yikes.
The WebPKI has to assume certificates might be compromised because there are 18 zillion different ways people might mishandle a certificate. But that's not at all true for a data center deployment of a microservice ensemble! Certificate keys are as secure in our system as the microservice itself is. If you lose one to an attacker, you lost the microservice too --- even if you rotate the cred, whatever flaw gave the attacker that certificate is just going to give the attacker the next one, too. If you stop trusting a certificate (and, by extension, the instance it was resident on), you zap the whole instance, revoke, and root-cause the flaw that caused the problem. Rotation just isn't winning you much.
At bottom the issue here is that the WebPKI has a very particular threat model, and it's a shitty threat model, and we have built lots of tools for that threat model, and many of them are by necessity shitty because we live in a fallen world. But your microservice ensemble wasn't born with original sin! You don&...
Someone just sent me a tweet from @halvarflake where he asked for the threat model under which service-to-service TLS is the best solution. I wanted to cross-link these threads since I think they're related: https://twitter.com/halvarflake/status/1337308903512678401
First, agreed. Certificates can't and shouldn't attest to the user's identity. You need another mechanism for that.
If you're securing requests, then securing the channel might be redundant. However, I do want authenticated service identity (not just user identity). The shape of policy I want to enforce is: "<service x> should be allowed to make <rpc> to <service y> in the context of responding to <request> from <user z>". I need the authenticated identity of both the service and the user to do that. Either that, or the authorization could take place higher up in the stack and I need a capability token passed through. Even with a capability token, it's nice to know the authenticated identity of the calling service for other purposes, like audit. Not to beat a dead horse, but I realize that macaroons satisfy all of these requirements :).
I also want confidentiality. Macaroons don't do this. This is an important point that I want to make more strongly this time, because I feel like it's getting lost in the noise. Authentication isn't the only thing people are using TLS for.
I appreciate your perespective. I think the best way to respond is to actually answer @halvarflake's question: what's the threat model that justifies service-to-service mutual TLS?
One answer is: that's the wrong question. Many people use mTLS for flexibility, not security.
Most microservice systems are born as a bunch of web services thrown in a VPC (or, more recently, a kubernetes). They can all talk to each other. Some of these systems are successful, and eventually outgrow their perimeter. Sub-components need to be able to communicate across untrusted channels (e.g., the web). TLS can give you a "cryptographic VPN" to do this.
There are a lot of VPN-y solutions with a lot of pros and cons. TLS isn't always the right way. The big pro for TLS, however, is that it works everywhere. Even if you don't control the host stack (e.g., functions as a service) or if you're on a constrained device, etc. TLS is also baked into most infrastructure: proxies, queues, databases, etc. You can easily terminate TLS easily at your perimeter if that's what you want. Alternatively, you can connect directly from a service in one perimeter to another service or to a piece of infrastructure like a database in another, without a proxy. Again, TLS is ubiquitous and it's flexible.
At this point I think we're at your "you must be this high to get on the ride" model. Why go further? This is where it makes sense to start talking about threat model.
First... I already mentioned this, but, once again: confidentiality. I think this is less controversial, and it doesn't require client certs. So I'll not elaborate further except to say that people need it.
The interesting topic is: service authentication. Why use client certificates for service authentication? I contend that service-to-service mTLS plus a bit of simple, coarse authorization actually does address a number of common security threats. Specifically:
* Malicious insiders: if you're using microservices and practicing DevOps and agile and have engineers on-call you probably have a lot of people and a lot of code that can get inside your perimeter. Coarse segmentation of service-to-service interactions can help here. An engineer who has the ability to `ssh` or `kubectl exec` or tunnel into your perimeter won't immediately have access to every sensitive service you run.
* Lateral movement: very similar to the above, but thi...
In a lot of this, I see you trying to frame the argument as "should we do mTLS or should we do no TLS at all". And while there are threat models where TLS doesn't make any sense (Halvar's same-host or same-rack K8s model), I don't think people are generally advocating against encrypting network traffic. Encrypt your network traffic. Obviously, you don't need mTLS to do this.
Another sleight of hand I see here is the ambiguity between network authentication, service authentication, and user authentication. When it's convenient for the argument, it seems, mTLS is just a way to ensure both ends of a network connection are part of the super best friends club. But later, when it's helpful to argue that mTLS is "flexible", certs carry credentials. Maybe even JWTs! There's a subtext to the "be wary of mTLS" argument, and it's that unified site-wide PKIs don't work well in practice.
Finally, you've lost me completely with this "revocation is less critical to the threat model than you think it is" stuff. The step-ca website is even explicit about this distinction you're trying to draw between "active revocation" and "passive revocation" (short-lifetime certificates). No. That doesn't work in internal application environments. step-ca defaults to 24-hour certificates. Have you considered how weird that is? 24 hours is short enough that your CA needs to be highly available --- ask around about production outages that have been caused by certificates expiring at much longer lifetimes --- but long enough that you can't possibly "wait out" a compromised cert.
I feel like something that is happening here is that WebPKI-style mTLS advocates think that a server cert and a client cert are two different instances of the same thing. They are not. They are wildly different entities. A server certificate prevents a MITM active attacker from intercepting a TLS connection. That's a rare attack, and, more importantly, a second-order attack. A client certificate is an access token for the application. Lose a prod-viable SSH key and then tell me I should wait 7 hours for it to expire.
The reality is that every application that deploys "passive revocation" is ultimately going to have to implement "active revocation", and, if they follow the advice on the step-ca website, they're going to have to do it on no notice, and in the most disruptive way possible, because when an internal access credential is lost, it has to be shut off immediately.
It's not intended to be sleight of hand. Confidentiality is the gateway drug. The argument is that, once you're using TLS for that, the incremental cost of adding client certificates and doing mTLS to reduce blast radius is lower. It's not zero. It is lower.
I'm also not arguing that some active mechanism to immediately remove attacker access is unimportant. Of course it is. I'm arguing that active certificate revocation is not the only way to achieve that. I gave two examples: remove access at the perimeter or push new policy to revoke access for the compromised entity. Most microservice systems already have tools here: they're typically adding mTLS on top of a perimeter model with decent privileged access management.
This is a bit of an asymmetric argument, in any case. I have all of my cards on the table. The opposition isn't really proposing any concrete alternative, so it's hard to make a meaningful comparison. If you want active revocation, implement active revocation. It's not easy, but OCSP and CRL exist and they do work. I even suggested a way to do it well: push a CRL to a cloud bucket, which is scalable and highly available. You can call that "cargo culting all of the idiosyncracies of the WebPKI", but I still haven't heard a concrete alternative. So where does that leave us?
The only concrete thing that's been proposed... that I had to propose myself, since no one else would even answer this very simple question, is macaroons. Which I've said repeatedly I think are a good idea. However, it's abundantly clear to me why people, in practice, are choosing client certs over macaroons. Macaroons can't layer into an existing system the way that mTLS can, and if all you want is "blast containment" you can get that with a fairly simple client certificate setup and a bit of coarse authorization.
Regarding CA availability: if you can't remediate an availability issue with a shared-nothing web service in 8 hours (the downtime you'd need for a cert to expire in a default step-ca setup) then you've got bigger problems than certificate management. The 24 hour default is long enough for people to remediate an outage and short enough to get people operationally comfortable with the idea of short-lived certificates. If I were to use shared secrets instead of asymmetric keys, and I created a system that rotated all of my service-to-service shared secrets automatically every 24 hours would you say that's a bad idea?
First, if you are in a place where the only two alternatives are (1) pervasive identity-bound mTLS or (2) Macaroons, you are wildly out of step with how security is managed in this industry. Most orgs, including some with very well-resourced security teams, don't do pervasive mTLS of any sort. And nobody uses Macaroons.
Secondly:
I'm also not arguing that some active mechanism to immediately remove attacker access is unimportant. Of course it is. I'm arguing that active certificate revocation is not the only way to achieve that.
An "active mechanism to remove attacker access" is revocation. That's literally what the word means.
You’re putting words in my mouth again. I didn’t say those were the only two options. I asked what your silver-bullet alternative to mTLS is that has no issues of its own. The position seems to be that mTLS has problems, so don’t use it. But everything has problems.
Re: revocation... the point is that you can revoke attacker access without revoking a certificate.
Or, if you want, you can revoke certificates. I’m not totally opposed to that. CRL is finicky, but architecturally it works well with short-lived certs and it’s not very different than secret rotation. If you can push new secrets, you can push a CRL. In fact, if you can push new secrets, you can also push new roots.
There are many people who use mTLS. Along with consistent service logging, it’s one of the main reasons people use service meshes. It’s also very common in IoT.
If you can revoke attacker access without revoking the certificate, the certificate isn't doing all the authentication stuff you say it is. If you have to inform all the services in your network that a certificate (or an identity bound exclusively to a certificate) is invalid, you've implemented revocation. You can see how this looks like jazz-hands, right?
The certificate is still containing blast radius when the attacker is in your network and limiting the insider threat from people who have access to some subset of production. Without a cert, they’d be able to access everything. With a cert, they can only access stuff that the identity bound to the certificate can access. It’s doing precisely what I want it to do.
If you find someone malicious in your network, you lock them out of your network. You don’t need to revoke a certificate to do this.
OR, you can do CRL. This isn’t jazz hands, it’s a choice. As you push mTLS harder, and need things like active revocation to satisfy your threat model, it’s worth reconsidering whether mTLS is the right choice. Sometimes it is, sometimes it’s not. Sometimes you don’t have a feasible alternative.
Wait a minute. You're not being coherent. I'm the one saying mTLS is blast-radius containment. What that means is: it's not your primary security control. It's certainly not your primary authentication modality. It's a thing that constrains the impact of another security flaw.
You can't "lock someone out of your network". That's handwaving. If you can do that reliably, you don't need any other security controls. Just lock all the bad people out!
The problem with mTLS in configurations where you rely on it for authentication is, if an attacker manages to obtain a client certificate key, it no longer matters if you nuke the machine from orbit, because they have the keypair! The keypair still works! They can use that keypair from any other place on the network they have access to (if there's no such place, you just defined away the need for mTLS).
Which is why you need active revocation if you're going to rely on mTLS: when you lose confidence in your sole custody of a client keypair, you need to actively revoke that keypair, immediately. You can't just wait 7 hours for the step-ca default key lifetime to expire!
"Or, you can do CRL" is literally active revocation. Your project has a prominent call-out about how most people don't need to do this. But that's literally the opposite of the truth. It's true in the WebPKI, but I think you've become confused about the difference between WebPKI server certs and internal client certificates. Telling people they should deploy lots of mTLS but not worry about revocation is malpractice.
What's going to happen to people in practice is that they're going to have to re-key their entire fleet any time there's any question about the integrity of a service. Or, more likely: they won't, because the mTLS deployment mode you're encouraging creates so much goddamn friction to protecting a key that people will roll the dice with the safety of their users rather than confront the fact that the correct engineering solution is an outage-inducing all-hands-on-deck rekeying.
What blows my mind about this is, at 24-hour expiry, in a medium-sized application, you're going to have machines needing to refresh keys practically every hour of the day; your CA will need to be available 24/7. At that point, you've basically reinvented Kerberos. Ops teams fucking hate Kerberos! And for all that effort, you still face fleetwide updates any time something sketchy happens anywhere.
I like mTLS for things like "we set up a Consul cluster; let's make sure just the machines that use Consul can reach it". It works fine for that. I don't think it's a good idea to take it much further.
In my response from three hours ago about threat model I said “service authentication helps control the blast radius if someone gets inside your perimeter”. I’ve been saying that I think we mostly agree for a while.
If you have a perimeter with well-defined entry points (e.g., a bastion for HTTPS and SSH) then you can lock people out at those entry points. Mutual TLS is defense-in-depth, and how you implement least authority for services inside the perimeter.
I know that CRL is literally active revocation. If you want active certificate revocation you can literally do active certificate revocation. Whether you need it depends on your threat model. And it’s not actually that hard in many scenarios. It is harder than not doing active revocation, and if you find yourself needing it to satisfy your threat model you probably should consider other alternatives for client authentication.
I’m not telling people they should use lots of mTLS but not worry about revocation. Any content I have written that discusses short-lived certificates is very careful to point out the fact that a compromised certificate will be considered valid until it is revoked. Most people who are using our open source toolchain are actually using it for internal server auth TLS, for narrow use cases like issuing certificates for database authentication or VPNs, or for mTLS to satisfy the sort of threat model I’ve described here.
We haven’t implemented anything around active revocation yet. But it is planned, because it is necessary in many scenarios.
If you’re worried about keeping a simple web service running 24 hours a day then you shouldn’t use our stuff. If you’re far enough along to need our stuff, you’ve figured out how to keep a web service running... you have to keep your own web services running 24 hours a day otherwise your microservice system is already going to have availability issues.
Can you describe a threat model that any medium-sized application --- low tens of microservices, mid-single-digits instances of most of them --- would ever not need revocation? Help me understand where your proposed 24-hour "passive revocation" actually works.
If you have a group of services running inside a perimeter with secure entrypoint(s), issuing client certificates allows you to coarsely segment service-to-service interactions within that perimeter to contain blast radius from a malicious insider or service compromise.
You don’t need revocation because the credential is useless outside of the perimeter. If you find a malicious insider or a compromised service, you fix the service or lock the insider out of the perimeter. You’re glad that you had client certificates in place as it reduced the scope of the compromise. It would be nice if you could actively revoke any compromised credentials, but it’s not critical because you’ve locked out the offending user and/or fixed the compromised service.
This is such a weird position to take: it's so important to keep certificates fresh that you encourage people to set up an infrastructure to run internal ACME and expire certificates in just a matter of hours, but so unimportant to protect certificates, because they're "worthless outside the perimeter", that it's fine to just wait out a compromised key.
There are many use cases for internal PKI. This is just one of of them. You’re nitpicking a default.
Credential rotation is good security hygiene. To suggest otherwise is malpractice. Our toolchain makes certificate rotation trivially easy. Why not rotate frequently?
Hopefully the threat model stuff made sense. It still feels like you actively want to disagree with me, and I’m still not sure why. But I agree that this is starting to feel unproductive.
I do appreciate the discussion. I understand your position on client certs better now. Your concerns are valid.
Maybe one day we can discuss over beers or something. It feels like that would be the right atmosphere.
No, arbitrarily frequent credential rotation is not universally good hygiene, and suggesting otherwise is not malpractice. Why not rotate frequently? Because doing so requires a high-availability CA, and reduces the reliability of the whole system, for marginal or no security benefit.
Counterpoint: if it hurts, do it more. That’s good operational hygiene. If you’re worried about certificate rotation failing, the surest way to make that process resilient is to do a lot of it. If you’re running the smallstep toolchain, you have an HA CA. That’s a sunk cost. That’s why it defaults to frequent rotation.
Furthermore, it’s not arbitrary. Credentials leak and services come and go. Having active keys around that aren’t in use is worse than not having them around. If someone accidentally commits a key to a GitHub repo or something, it’s nice to know that key will only be useful for a little while.
If you still want to rotate less frequently, change the default.
This really has very little to do with the topic at hand, so I’m not sure why we’re debating it. Do you want me to change the default certificate lifetime in step-ca? What do you think it should be?
I love this summary, which is better than I could have written myself, and feel no need to clarify. Instead I'll add the origin story of my dislike for mTLS.
Not too long into my tenure of being an active maintainer of Apache httpd I somehow ended up being the X509 person. I'd worked on some of the crypto code in Apache-SSL, which was maintained by non-US people only at the time, and X509 sort of fell into that bucket. I also got on well with Dirk-Willem van Gulik who wrote a lot of the original X509 code, and later he ended up hiring me, but that's a different story.
As part of being an "Apache x509 person" I got a consultancy gig to audit an "Enterprise" company's mTLS setup. On the first day, based on nothing more than tailing logs, it became clear that some people had access to things they really shouldn't. The CTOs Executive Assistant, who was part of the group trying out mTLS, actually had god level access to everything. EAs are gods and should be worshiped, they make companies run, but this level of access was not intentional. A few hours of root causing and I figured out that the EA had god-level access to everything just because the string "admin" was in their job title. Access was being granted based on a greedy PCRE regex match on a string. It wasn't even in the right field in the certificate, but the string matching didn't care. This is what x.509 does to your brain.
Similar gigs would find more problems; to an approximation, no one had working revocation, more than once I found setups that didn't even require client certificates, but no one had noticed. Over time I just got the impression that x509 was commonly set up by folks who were blinded by a cryptographic luster but wouldn't follow through. There's been a general absence of professionally built solutions that get the details right.
I'm biased because I work at AWS, and even more biased because I now work on our IAM team, but I really like AWS SIGv4 and IAM policy. SIGv4 signing itself is request-level, and is incredibly simple with a tiny pre-authorization TCB. IAM policy is a full and expressive authorization language with a rich grammar. They're then paired with an ecosystem of features like CloudTrail and last-use-of-credential that give meaningful controls that avoid trade-offs between availability and security.
It doesn't have to be AWS, but I do think that solutions in this space really have to be operational services, and not just technologies or standards. It has to be someone's full time job to look at this stuff and make sure it's working and secure, and that's just very hard to match. If I quit my job tomorrow and was building a startup, I'd use AWS API Gateway's SIGv4 and policy support and leave it at that.
22 comments
[ 349 ms ] story [ 1795 ms ] threadFirst: TLS (and mTLS) create secure channels. A channel bears many requests and responses. In many designs, a single channel will bear requests for many different users. The job of a textbook secure channel is to prevent an attacker with control of the underlying network from tampering with requests and responses. Binding identities and authorization claims to requests is a different job. The textbook solution to that problem is authenticated requests, not authenticated secure channels. I wouldn't have used the word "layering violation", which I would contend Is Not A Thing, but the point MacCárthaigh makes is important. So: when MacCárthaigh says SQLI and Request Smuggling are "still things", he's not saying that mTLS introduces SQLI, but rather that authenticating at the level of secure channels means that a request smuggling attack is almost automatically a game-over for your application, because you aren't authenticating the requests independently of the channel. Even Basic Auth can potentially avoid that problem, if an attacker has to know a secret to slip into a request in order to forge a request on a compromised channel.
Second: TLS was designed for the WebPKI, and that's its most important application. A consequence of this is that a lot of TLS software is designed for WebPKI threat models. Like everything else the IETF produces, TLS has features that support a lot of other applications, like using heartbeat messages to debug remote processes. But what matters in the real world is the installed base of TLS software, and that installed base does not have sane defaults and safe ergonomics for mTLS. You have to be extra careful rolling it out that way.
Third: TLS is built on X.509, and X.509 is extremely complex and error prone. This complexity is deceptive, because we've spent 20 years filing down the complexity of X.509 in the WebPKI deployment mode, and developers mostly don't have to care about anything except the filenames for their certificates and keys. That's out the window with mTLS, where peers actually have to crack open certificates and look at them to accomplish authn and authz tasks.
Fourth: Revocation is a debacle. It's a debacle everywhere, but TLS complexifies it, because it has to work at Internet scale. If you're managing a fleet for even a large, complex application, you have recourse to designs that make revocation easier to handle.
Fifth: Fleet-wide credential rotation is much harder in mTLS than it is in simpler systems, where the root of trust might just be a simple secret you can quickly load on to a given machine and tabletop how to do a fleetwide deployment in a quick meeting. You can add mechanism to automate certificate issuance, but those systems have to be resilient to the total loss of the root of trust, which is something the TLS ecosystem is not generally good at; the WebPKI could suffer a total loss of all trust roots if we broke RSA, but not so much if we just broke OpenSSL.
Sixth: The standard modern solution to revocation, short-lifetime certificates, also relies on secure clocks. Breaking clocks Internet-wide might be difficult enough to be outside a reasonable threat model, but breaking clocks in a single data center is not.
Seventh: To make short-expiry work in a setting where a compromised certificate isn't just a potential coffee shop MITM for a subset of your users but instead a game-over compromise for the entire application, you might need lifetimes so short they're hard to operationalize without outages. Part of what we like about public key authentication schemes is that they don't require continuous access to a central trusted authority just to keep the system up and running!
Eighth: Serious applications want queryable audit trails for inter-service RPCs, which means having ready, oft...
Reading it over, I think we mostly agree on the facts. It's easy to do mTLS and x509 wrong. The question, then, is what's easier / more secure: doing mTLS/x509 right or doing something else? I think that's somewhat subjective: it depends on your requirements, your environment, and your skillset.
One point that I'd like to reiterate is this: if you want a consistent cryptographic solution that works everywhere, TLS is pretty much your only choice. You could use something else for client authentication, but you probably still need TLS.
As a strawman, here's a sketch of how I'd recommend doing TLS in a microservice system. I consider this "right" for most garden-variety microservices-in-cloud scenarios and don't think it's particularly hard to do. Most of this is already implemented in https://github.com/smallstep/certificates:
I want to respond specifically to your first and final points.On your first point: I understand that in theory an attacker could slip a request across a secure channel, and binding authentication to a request could in theory prevent that. I don't understand how that's likely to happen in the context I'm thinking of here. Which may be different than the context you're thinking of. So let me clarify.
Suppose I have `<end-user> -> <service-a> -> <service-b> -> <database>`. Let's focus on `<service-a> -> <service-b>`. I don't see how using end-to-end mTLS, terminating in `<service-a>` and `<service-b>` application code, would be any more vulnerable to this variety of attack than an HTTP Basic header like `Authorization: Basic base64(service-a:password)`. Surely, the logic in `<service-a>` is simply "insert HTTP Basic...
I'll stipulate to mTLS resolving client-identity and server-identity. But the client is making requests of the server that pertain to a user. The certificate doesn't (and can't) attest to a user.
In most designs I've actually evaluated, what ends up happening is that the server just trusts the user provided by the client, and the client tries hard not to ask for things for the wrong user. But beyond whether the client has authorized every code path that generates a request, you have an additional problem here, because even after the client authorizes a path that generates a request, a bug in the client can give an attacker influence over the request itself. The server has no way of distinguishing between a corrupted request and a real request, even though you're relying on a secure channel.
Authenticated requests mitigate this problem: an attacker might be able to corrupt a request, but it's not enough to corrupt it; you need to know whatever secret authenticates the request itself.
To do this right, you now need two authentication schemes in play: one for the secure channel, and one for the requests themselves. But if you can reliably authenticate requests, why are we rigorously authenticating the secure channel? We're spending complexity chits to buy only marginal extra security.
I think it can make some sense to mesh up services with mTLS as a "you must be this high to get on the ride" mechanism. But since that's all it's doing, we don't need a lot of complicated mechanism to give precisely the right certificates to services, because even if something gets screwed up, request authentication, not secure channel authentication, should be what's protecting your application.
Especially when we think about short-lived certificates and ACME and rollover: what is this really buying us in a strong system with request authentication? A WebPKI ACME certificate will live for 90 days. If certificates matter to our design, we can't let a compromised cert live for 90 days. Our certificate lifespans have to be much shorter --- the duration in which we'd be OK having a compromised credential still viable for an attacker. Remember the differing consequences between a WebPKI certificate compromise and an internal credential compromise: one is a second-order flaw that lets attackers with some other vulnerability MITM a subset of users; the other is a game-over flaw. How long would you let a compromised developer prod SSH key live?
So now we're talking about very short-lived certificates, and you can see, this is a design that spends all the complexity chits asymmetric encryption costs, but is asymptotically approaching the operational resiliency of Kerberos. Yikes.
The WebPKI has to assume certificates might be compromised because there are 18 zillion different ways people might mishandle a certificate. But that's not at all true for a data center deployment of a microservice ensemble! Certificate keys are as secure in our system as the microservice itself is. If you lose one to an attacker, you lost the microservice too --- even if you rotate the cred, whatever flaw gave the attacker that certificate is just going to give the attacker the next one, too. If you stop trusting a certificate (and, by extension, the instance it was resident on), you zap the whole instance, revoke, and root-cause the flaw that caused the problem. Rotation just isn't winning you much.
At bottom the issue here is that the WebPKI has a very particular threat model, and it's a shitty threat model, and we have built lots of tools for that threat model, and many of them are by necessity shitty because we live in a fallen world. But your microservice ensemble wasn't born with original sin! You don&...
First, agreed. Certificates can't and shouldn't attest to the user's identity. You need another mechanism for that.
If you're securing requests, then securing the channel might be redundant. However, I do want authenticated service identity (not just user identity). The shape of policy I want to enforce is: "<service x> should be allowed to make <rpc> to <service y> in the context of responding to <request> from <user z>". I need the authenticated identity of both the service and the user to do that. Either that, or the authorization could take place higher up in the stack and I need a capability token passed through. Even with a capability token, it's nice to know the authenticated identity of the calling service for other purposes, like audit. Not to beat a dead horse, but I realize that macaroons satisfy all of these requirements :).
I also want confidentiality. Macaroons don't do this. This is an important point that I want to make more strongly this time, because I feel like it's getting lost in the noise. Authentication isn't the only thing people are using TLS for.
I appreciate your perespective. I think the best way to respond is to actually answer @halvarflake's question: what's the threat model that justifies service-to-service mutual TLS?
One answer is: that's the wrong question. Many people use mTLS for flexibility, not security.
Most microservice systems are born as a bunch of web services thrown in a VPC (or, more recently, a kubernetes). They can all talk to each other. Some of these systems are successful, and eventually outgrow their perimeter. Sub-components need to be able to communicate across untrusted channels (e.g., the web). TLS can give you a "cryptographic VPN" to do this.
There are a lot of VPN-y solutions with a lot of pros and cons. TLS isn't always the right way. The big pro for TLS, however, is that it works everywhere. Even if you don't control the host stack (e.g., functions as a service) or if you're on a constrained device, etc. TLS is also baked into most infrastructure: proxies, queues, databases, etc. You can easily terminate TLS easily at your perimeter if that's what you want. Alternatively, you can connect directly from a service in one perimeter to another service or to a piece of infrastructure like a database in another, without a proxy. Again, TLS is ubiquitous and it's flexible.
At this point I think we're at your "you must be this high to get on the ride" model. Why go further? This is where it makes sense to start talking about threat model.
First... I already mentioned this, but, once again: confidentiality. I think this is less controversial, and it doesn't require client certs. So I'll not elaborate further except to say that people need it.
The interesting topic is: service authentication. Why use client certificates for service authentication? I contend that service-to-service mTLS plus a bit of simple, coarse authorization actually does address a number of common security threats. Specifically:
* Malicious insiders: if you're using microservices and practicing DevOps and agile and have engineers on-call you probably have a lot of people and a lot of code that can get inside your perimeter. Coarse segmentation of service-to-service interactions can help here. An engineer who has the ability to `ssh` or `kubectl exec` or tunnel into your perimeter won't immediately have access to every sensitive service you run.
* Lateral movement: very similar to the above, but thi...
Another sleight of hand I see here is the ambiguity between network authentication, service authentication, and user authentication. When it's convenient for the argument, it seems, mTLS is just a way to ensure both ends of a network connection are part of the super best friends club. But later, when it's helpful to argue that mTLS is "flexible", certs carry credentials. Maybe even JWTs! There's a subtext to the "be wary of mTLS" argument, and it's that unified site-wide PKIs don't work well in practice.
Finally, you've lost me completely with this "revocation is less critical to the threat model than you think it is" stuff. The step-ca website is even explicit about this distinction you're trying to draw between "active revocation" and "passive revocation" (short-lifetime certificates). No. That doesn't work in internal application environments. step-ca defaults to 24-hour certificates. Have you considered how weird that is? 24 hours is short enough that your CA needs to be highly available --- ask around about production outages that have been caused by certificates expiring at much longer lifetimes --- but long enough that you can't possibly "wait out" a compromised cert.
I feel like something that is happening here is that WebPKI-style mTLS advocates think that a server cert and a client cert are two different instances of the same thing. They are not. They are wildly different entities. A server certificate prevents a MITM active attacker from intercepting a TLS connection. That's a rare attack, and, more importantly, a second-order attack. A client certificate is an access token for the application. Lose a prod-viable SSH key and then tell me I should wait 7 hours for it to expire.
The reality is that every application that deploys "passive revocation" is ultimately going to have to implement "active revocation", and, if they follow the advice on the step-ca website, they're going to have to do it on no notice, and in the most disruptive way possible, because when an internal access credential is lost, it has to be shut off immediately.
I'm also not arguing that some active mechanism to immediately remove attacker access is unimportant. Of course it is. I'm arguing that active certificate revocation is not the only way to achieve that. I gave two examples: remove access at the perimeter or push new policy to revoke access for the compromised entity. Most microservice systems already have tools here: they're typically adding mTLS on top of a perimeter model with decent privileged access management.
This is a bit of an asymmetric argument, in any case. I have all of my cards on the table. The opposition isn't really proposing any concrete alternative, so it's hard to make a meaningful comparison. If you want active revocation, implement active revocation. It's not easy, but OCSP and CRL exist and they do work. I even suggested a way to do it well: push a CRL to a cloud bucket, which is scalable and highly available. You can call that "cargo culting all of the idiosyncracies of the WebPKI", but I still haven't heard a concrete alternative. So where does that leave us?
The only concrete thing that's been proposed... that I had to propose myself, since no one else would even answer this very simple question, is macaroons. Which I've said repeatedly I think are a good idea. However, it's abundantly clear to me why people, in practice, are choosing client certs over macaroons. Macaroons can't layer into an existing system the way that mTLS can, and if all you want is "blast containment" you can get that with a fairly simple client certificate setup and a bit of coarse authorization.
Regarding CA availability: if you can't remediate an availability issue with a shared-nothing web service in 8 hours (the downtime you'd need for a cert to expire in a default step-ca setup) then you've got bigger problems than certificate management. The 24 hour default is long enough for people to remediate an outage and short enough to get people operationally comfortable with the idea of short-lived certificates. If I were to use shared secrets instead of asymmetric keys, and I created a system that rotated all of my service-to-service shared secrets automatically every 24 hours would you say that's a bad idea?
Secondly:
I'm also not arguing that some active mechanism to immediately remove attacker access is unimportant. Of course it is. I'm arguing that active certificate revocation is not the only way to achieve that.
An "active mechanism to remove attacker access" is revocation. That's literally what the word means.
Re: revocation... the point is that you can revoke attacker access without revoking a certificate.
Or, if you want, you can revoke certificates. I’m not totally opposed to that. CRL is finicky, but architecturally it works well with short-lived certs and it’s not very different than secret rotation. If you can push new secrets, you can push a CRL. In fact, if you can push new secrets, you can also push new roots.
There are many people who use mTLS. Along with consistent service logging, it’s one of the main reasons people use service meshes. It’s also very common in IoT.
If you find someone malicious in your network, you lock them out of your network. You don’t need to revoke a certificate to do this.
OR, you can do CRL. This isn’t jazz hands, it’s a choice. As you push mTLS harder, and need things like active revocation to satisfy your threat model, it’s worth reconsidering whether mTLS is the right choice. Sometimes it is, sometimes it’s not. Sometimes you don’t have a feasible alternative.
You can't "lock someone out of your network". That's handwaving. If you can do that reliably, you don't need any other security controls. Just lock all the bad people out!
The problem with mTLS in configurations where you rely on it for authentication is, if an attacker manages to obtain a client certificate key, it no longer matters if you nuke the machine from orbit, because they have the keypair! The keypair still works! They can use that keypair from any other place on the network they have access to (if there's no such place, you just defined away the need for mTLS).
Which is why you need active revocation if you're going to rely on mTLS: when you lose confidence in your sole custody of a client keypair, you need to actively revoke that keypair, immediately. You can't just wait 7 hours for the step-ca default key lifetime to expire!
"Or, you can do CRL" is literally active revocation. Your project has a prominent call-out about how most people don't need to do this. But that's literally the opposite of the truth. It's true in the WebPKI, but I think you've become confused about the difference between WebPKI server certs and internal client certificates. Telling people they should deploy lots of mTLS but not worry about revocation is malpractice.
What's going to happen to people in practice is that they're going to have to re-key their entire fleet any time there's any question about the integrity of a service. Or, more likely: they won't, because the mTLS deployment mode you're encouraging creates so much goddamn friction to protecting a key that people will roll the dice with the safety of their users rather than confront the fact that the correct engineering solution is an outage-inducing all-hands-on-deck rekeying.
What blows my mind about this is, at 24-hour expiry, in a medium-sized application, you're going to have machines needing to refresh keys practically every hour of the day; your CA will need to be available 24/7. At that point, you've basically reinvented Kerberos. Ops teams fucking hate Kerberos! And for all that effort, you still face fleetwide updates any time something sketchy happens anywhere.
I like mTLS for things like "we set up a Consul cluster; let's make sure just the machines that use Consul can reach it". It works fine for that. I don't think it's a good idea to take it much further.
If you have a perimeter with well-defined entry points (e.g., a bastion for HTTPS and SSH) then you can lock people out at those entry points. Mutual TLS is defense-in-depth, and how you implement least authority for services inside the perimeter.
I know that CRL is literally active revocation. If you want active certificate revocation you can literally do active certificate revocation. Whether you need it depends on your threat model. And it’s not actually that hard in many scenarios. It is harder than not doing active revocation, and if you find yourself needing it to satisfy your threat model you probably should consider other alternatives for client authentication.
I’m not telling people they should use lots of mTLS but not worry about revocation. Any content I have written that discusses short-lived certificates is very careful to point out the fact that a compromised certificate will be considered valid until it is revoked. Most people who are using our open source toolchain are actually using it for internal server auth TLS, for narrow use cases like issuing certificates for database authentication or VPNs, or for mTLS to satisfy the sort of threat model I’ve described here.
We haven’t implemented anything around active revocation yet. But it is planned, because it is necessary in many scenarios.
If you’re worried about keeping a simple web service running 24 hours a day then you shouldn’t use our stuff. If you’re far enough along to need our stuff, you’ve figured out how to keep a web service running... you have to keep your own web services running 24 hours a day otherwise your microservice system is already going to have availability issues.
You don’t need revocation because the credential is useless outside of the perimeter. If you find a malicious insider or a compromised service, you fix the service or lock the insider out of the perimeter. You’re glad that you had client certificates in place as it reduced the scope of the compromise. It would be nice if you could actively revoke any compromised credentials, but it’s not critical because you’ve locked out the offending user and/or fixed the compromised service.
I think we're probably at an impasse.
Credential rotation is good security hygiene. To suggest otherwise is malpractice. Our toolchain makes certificate rotation trivially easy. Why not rotate frequently?
Hopefully the threat model stuff made sense. It still feels like you actively want to disagree with me, and I’m still not sure why. But I agree that this is starting to feel unproductive.
I do appreciate the discussion. I understand your position on client certs better now. Your concerns are valid.
Maybe one day we can discuss over beers or something. It feels like that would be the right atmosphere.
Furthermore, it’s not arbitrary. Credentials leak and services come and go. Having active keys around that aren’t in use is worse than not having them around. If someone accidentally commits a key to a GitHub repo or something, it’s nice to know that key will only be useful for a little while.
If you still want to rotate less frequently, change the default.
This really has very little to do with the topic at hand, so I’m not sure why we’re debating it. Do you want me to change the default certificate lifetime in step-ca? What do you think it should be?
Not too long into my tenure of being an active maintainer of Apache httpd I somehow ended up being the X509 person. I'd worked on some of the crypto code in Apache-SSL, which was maintained by non-US people only at the time, and X509 sort of fell into that bucket. I also got on well with Dirk-Willem van Gulik who wrote a lot of the original X509 code, and later he ended up hiring me, but that's a different story.
As part of being an "Apache x509 person" I got a consultancy gig to audit an "Enterprise" company's mTLS setup. On the first day, based on nothing more than tailing logs, it became clear that some people had access to things they really shouldn't. The CTOs Executive Assistant, who was part of the group trying out mTLS, actually had god level access to everything. EAs are gods and should be worshiped, they make companies run, but this level of access was not intentional. A few hours of root causing and I figured out that the EA had god-level access to everything just because the string "admin" was in their job title. Access was being granted based on a greedy PCRE regex match on a string. It wasn't even in the right field in the certificate, but the string matching didn't care. This is what x.509 does to your brain.
Similar gigs would find more problems; to an approximation, no one had working revocation, more than once I found setups that didn't even require client certificates, but no one had noticed. Over time I just got the impression that x509 was commonly set up by folks who were blinded by a cryptographic luster but wouldn't follow through. There's been a general absence of professionally built solutions that get the details right.
Literally nobody likes x509. Is anyone seriously working on an alternative? I think there’s room for a more modern, simpler cert format.
Sort of relatedly, I actually really like JWK for representing keys.
Do you think a professionally built solution that gets the details right is possible?
It doesn't have to be AWS, but I do think that solutions in this space really have to be operational services, and not just technologies or standards. It has to be someone's full time job to look at this stuff and make sure it's working and secure, and that's just very hard to match. If I quit my job tomorrow and was building a startup, I'd use AWS API Gateway's SIGv4 and policy support and leave it at that.
I’ve always thought of AWS IAM as: great for my-service-to-cloud-service, not so great for my-service-to-my-service.
I haven’t looked at AWS IAM in a while. I’m gonna take another look now :)