Before the inevitable flood of the super insightful and valuable comments like "hurr durr another google product killed", let me ask:
Is there a good alternative frontend for these queries? As far I understand the underlying data is still public and available from many sources, but this was the only easy to use search I know of.
Cloudflare sells some services to notify you on changes to your domains, but what about just checking on it manually, or exploring history of other domains?
I am aware of that, but as a non profit providing certificate services, it aligns with their mission (imho) to also provide CT log storage and search functionality imho (not to denigrate crt.sh, which is good tooling, but yet another for profit org [Sectigo.com, formerly Comodo CA] benevolently providing the service for now).
Personally, I think it's good to move away from Google, Facebook, and similar providing these core internet services. Throw some money at Let's Encrypt, the Internet Archive, and whomever is going to run Elasticsearch for this corpus, call it a day. The primitives are cheap to do so (compute and storage).
Building this was exciting, I've been proud to run it and and it predated services from Facebook or Cloudflare. I'm not sure what the future should hold however, it's hard to bring it up anywhere without being told people should just use Facebook.
I know there are CT search services like crt.sh, but is it practical to download the raw data and search it locally? If the logs are append‐only, it feels like a perfect usecase for rsync.
Or purchase 50x4TB drives for $10k USD, or 100 if you want RAID-1. These can easily fit into 4-8U of rack space, at around $1,500 or less per rack U (don't buy new / expensive servers if all you need is high-IO NAS storage hosts for cephFS).
This will have minimal recurring fees. But will cost you time to manage instead of paying $CLOUDVENDER $50k/year. Please evaluate the tradeoffs for your scenario but good God don't blindly default to the cloud without first thinking carefully about it.
See also related: Today's HN top article is "Just say Yes to self-hosting"
Do you think that CT log data replication would be more secure and efficient if the CT logs were stored by a zero trust distributed application like a blockchain, instead of Merkle signatures in a database owned by one party (that's now discontinuing free indexing, at least)?
> Trillian is a centralized Merkle tree: it doesn't support native replication [...] According to the trillian README, trillian depends upon MySQL/MariaDB and thus internal/private replication is as good as the SQL replication model (which doesn't have a distributed consensus algorithm like e.g. paxos).
And what about indexing and search queries at volume, again without replication?
>> Indexers are node operators in The Graph Network that stake Graph Tokens (GRT) in order to provide indexing and query processing services. Indexers earn query fees and indexing rewards for their services. They also earn from a Rebate Pool that is shared with all network contributors proportional to their work, following the Cobbs-Douglas Rebate Function.
>> GRT that is staked in the protocol is subject to a thawing period and can be slashed if Indexers are malicious and serve incorrect data to applications or if they index incorrectly. Indexers can also be delegated stake from Delegators, to contribute to the network.
>> Indexers select subgraphs to index based on the subgraph’s curation signal, where Curators stake GRT in order to indicate which subgraphs are high-quality and should be prioritized. Consumers (eg. applications) can also set parameters for which Indexers process queries for their subgraphs and set preferences for query fee pricing.
> For convenience, these sources are aggregated and continuously exported to a GCS bucket maintained by OSV: gs://osv-vulnerabilities
> This bucket contains individual entries of the format gs://osv-vulnerabilities/<ECOSYSTEM>/<ID>.json as well as a zip containing all vulnerabilities for each ecosystem at gs://osv-vulnerabilities/<ECOSYSTEM>/all.zip
> E.g. for PyPI vulnerabilities:
# Or download over HTTP via https://osv-vulnerabilities.storage.googleapis.com/PyPI/all.zip
gsutil cp gs://osv-vulnerabilities/PyPI/all.zip
Hopefully, with an incentivized Blockchain Indexing service and/or e.g. GCS buckets that you just always `cp` and then load locally and then query locally, we can find a solution for queries of the growing CT Certificate Transparency logs.
How many people here do any sort of regular review of CT logs, and do you really think an attacker couldn't get away with whatever their attack was by the time you found out?
I did for a long time but I stopped. Real time is still super useful though. I have Cloudflare and Let's Encrypt email me now. Mostly it's been helpful for tracking down old boxes/services that are still running that have been lost/forgotten.
I would claim that I'm quite familiar with the TLS and certificate ecosystem, and up until this news item I didn't know Google runs a CT search engine.
Everyone uses either crt.sh or to a lesser extend Censys. I think this space is covered well and there's just no need for a Google-operated CT search engine nobody knows about.
I think I would claim to be similarly familiar :) but I did know Google operated this search and I've even used it in the past when Rob's site was down.
As I think I expressed in a post last year it's important that we could migrate off crt.sh but it isn't a problem if we never do.
One thing I don't like but can't attest to still being a problem since maintaining one is no longer my job, is some logs are (were) unreliable for a crawler, which is a problem if you wanted to spin up an alternative to crt.sh. Google checks things work for them as part of qualification, but that's at best a 90% solution
How many people feel they get positive value from CT services in general?
I have Cloudflare email me whenever my domain gets a cert, and that is a positive value for me. However, it is very clear to me that bots are watching these logs (in near real time) because obscure hostnames (sometimes UUIDs) that I set up for a quick test but need a real cert will start getting traffic almost immediately after the cert comes back from Let's Encrypt.
As an attacker, this log is a fantastic tool for service/host discovery, but as a defender I kind of wish I could opt out.
Note: a wildcard cert is a good way to avoid the problem, but I prefer to avoid wildcards if possible.
A lot do, but without knowing it. Chrome uses an 'expect-CT' header, which means the sites TLS cert should chain to a root CA, with both stored in a CTL.
I don't think that's what your parent post meant, but, Chrome actually mechanically requires CT logging in order to trust a site, the expect-CT header doesn't come into it. Safari has the same policy, and Mozilla intends to some day do likewise in Firefox but the politics of the exact rules are tricky for reasons we could get into if somebody cares.
When your Chrome connects to a TLS server, the server has to provide a certificate, and Chrome examines that certificate before any HTTP traffic happens (and thus before you could send an HTTP header like Expect-CT anywhere) unless you've got Group Policy or similar rules saying otherwise:
- The certificate must have proof it was logged in the form of SCTs. For most sites the SCTs are baked into the certificate when they got it, they're all that incomprehensible gibberish near the end of your certificate if you've read it.
- The certificate must have been issued relatively recently (825 days or less previously) and it must expire within 825 days of issuance or if it was issued since some time in 2020 when Apple's policy change happened, 398 days.
- The certificate says it is for TLS Servers (in most cases certificates say they're also for TLS Clients, and sometimes other things too, but Chrome checks it says specifically TLS Servers here)
- The certificate has a Subject Alternative Name matching the DNS name of the server, or, if the URL we're resolving is for a numeric IP address, a SAN matches that IP Address. SANs are typed, and are not free human text, so a DNS SAN and an IP address SAN are distinguishable even if 10.20.30.40 was a valid DNS name, which it is not. Old-fashioned "Common" names are disregarded.
- The certificate must be signed by a trusted CA (or by an intermediate which in turn was trusted, and so on recursively).
I think I hit the big ticket items, there might be some others.
Everybody indirectly gets value from CT because it allows the public to exercise their oversight over the Web PKI more effectively, and thus it acts as a guarantee over the behaviour of the CAs.
As a researcher it lets you call "bullshit" easily, many years ago I actually used the logs to show that CAs "missed" (did an inadequate job of identifying and revoking) some certificates they should have revoked months earlier, if you attempted that without logs you'd never be sure how much you missed and you'd have no fixed scope for the work.
[[ As to getting hit by bots, keep in mind that passive DNS can cause that too, if anybody (testers, third party services, anybody) does a DNS lookup for name X on most well known public DNS servers, the server operators sell the DNS answer + timestamp (but not who asked so this is not personal information), which you can then buy as a bulk service. So e.g. some-uuid-on-a.server.example gets looked up on an iPhone over somebody's ISP, and within seconds a feed has added some-uuid-on-a.server.example A 10.20.30.40 to the list of known DNS answers. ]]
You can use these monitors, (although not specifically the one operated by Google for much longer) or you can build your own. If you build your own you can choose to specialise. Maybe you only care about certificates for names under .horse, or you only care about certificates from for-profit companies, or with RSA keys.
45 comments
[ 3.3 ms ] story [ 106 ms ] threadIs there a good alternative frontend for these queries? As far I understand the underlying data is still public and available from many sources, but this was the only easy to use search I know of.
Cloudflare sells some services to notify you on changes to your domains, but what about just checking on it manually, or exploring history of other domains?
Personally, I think it's good to move away from Google, Facebook, and similar providing these core internet services. Throw some money at Let's Encrypt, the Internet Archive, and whomever is going to run Elasticsearch for this corpus, call it a day. The primitives are cheap to do so (compute and storage).
But thanks for the recommendation, I'll try it some time later.
Edit: Regarding the facebook one: "Log into Facebook to use this tool." Yeah no thanks.
* https://crt.sh/
* https://developers.facebook.com/tools/ct
* https://ui.ctsearch.entrust.com/ui/ctsearchui
Not sure I'd even list that as a reasonable alternative.
https://ctadvisor.lolware.net/ https://github.com/technion/ct_advisor
Building this was exciting, I've been proud to run it and and it predated services from Facebook or Cloudflare. I'm not sure what the future should hold however, it's hard to bring it up anywhere without being told people should just use Facebook.
And yes, it's quite challenging. It would be a nice service to the community if someone would host downloadable dumps of CT logs.
That’s not that impractical. Could easily be stored on gcs and s3 with requestor pays for the api and any associated egress charges.
205 terabytes at 2 cent per GB/month is just 50k/year.
This will have minimal recurring fees. But will cost you time to manage instead of paying $CLOUDVENDER $50k/year. Please evaluate the tradeoffs for your scenario but good God don't blindly default to the cloud without first thinking carefully about it.
See also related: Today's HN top article is "Just say Yes to self-hosting"
https://news.ycombinator.com/item?id=30781536
https://www.backblaze.com/b2/cloud-storage-pricing.html
You would need to know the index, or you could just iterate over a range
From "Oak, a Free and Open Certificate Transparency Log" (LetsEncrypt 2019) https://news.ycombinator.com/item?id=19920002 :
> Trillian is a centralized Merkle tree: it doesn't support native replication [...] According to the trillian README, trillian depends upon MySQL/MariaDB and thus internal/private replication is as good as the SQL replication model (which doesn't have a distributed consensus algorithm like e.g. paxos).
And what about indexing and search queries at volume, again without replication?
From "A future for SQL on the web" https://news.ycombinator.com/item?id=28158491 :
> https://thegraph.com/docs/indexing
>> Indexers are node operators in The Graph Network that stake Graph Tokens (GRT) in order to provide indexing and query processing services. Indexers earn query fees and indexing rewards for their services. They also earn from a Rebate Pool that is shared with all network contributors proportional to their work, following the Cobbs-Douglas Rebate Function.
>> GRT that is staked in the protocol is subject to a thawing period and can be slashed if Indexers are malicious and serve incorrect data to applications or if they index incorrectly. Indexers can also be delegated stake from Delegators, to contribute to the network.
>> Indexers select subgraphs to index based on the subgraph’s curation signal, where Curators stake GRT in order to indicate which subgraphs are high-quality and should be prioritized. Consumers (eg. applications) can also set parameters for which Indexers process queries for their subgraphs and set preferences for query fee pricing.
> For convenience, these sources are aggregated and continuously exported to a GCS bucket maintained by OSV: gs://osv-vulnerabilities
> This bucket contains individual entries of the format gs://osv-vulnerabilities/<ECOSYSTEM>/<ID>.json as well as a zip containing all vulnerabilities for each ecosystem at gs://osv-vulnerabilities/<ECOSYSTEM>/all.zip
> E.g. for PyPI vulnerabilities:
Hopefully, with an incentivized Blockchain Indexing service and/or e.g. GCS buckets that you just always `cp` and then load locally and then query locally, we can find a solution for queries of the growing CT Certificate Transparency logs.Secure-corpdomain.com etc?
Everyone uses either crt.sh or to a lesser extend Censys. I think this space is covered well and there's just no need for a Google-operated CT search engine nobody knows about.
Our use case is finding phishing sites targeting our customers, CT Logs have been amazing for detecting campaigns early.
As I think I expressed in a post last year it's important that we could migrate off crt.sh but it isn't a problem if we never do.
One thing I don't like but can't attest to still being a problem since maintaining one is no longer my job, is some logs are (were) unreliable for a crawler, which is a problem if you wanted to spin up an alternative to crt.sh. Google checks things work for them as part of qualification, but that's at best a 90% solution
I have Cloudflare email me whenever my domain gets a cert, and that is a positive value for me. However, it is very clear to me that bots are watching these logs (in near real time) because obscure hostnames (sometimes UUIDs) that I set up for a quick test but need a real cert will start getting traffic almost immediately after the cert comes back from Let's Encrypt.
As an attacker, this log is a fantastic tool for service/host discovery, but as a defender I kind of wish I could opt out.
Note: a wildcard cert is a good way to avoid the problem, but I prefer to avoid wildcards if possible.
When your Chrome connects to a TLS server, the server has to provide a certificate, and Chrome examines that certificate before any HTTP traffic happens (and thus before you could send an HTTP header like Expect-CT anywhere) unless you've got Group Policy or similar rules saying otherwise:
- The certificate must have proof it was logged in the form of SCTs. For most sites the SCTs are baked into the certificate when they got it, they're all that incomprehensible gibberish near the end of your certificate if you've read it.
- The certificate must have been issued relatively recently (825 days or less previously) and it must expire within 825 days of issuance or if it was issued since some time in 2020 when Apple's policy change happened, 398 days.
- The certificate says it is for TLS Servers (in most cases certificates say they're also for TLS Clients, and sometimes other things too, but Chrome checks it says specifically TLS Servers here)
- The certificate has a Subject Alternative Name matching the DNS name of the server, or, if the URL we're resolving is for a numeric IP address, a SAN matches that IP Address. SANs are typed, and are not free human text, so a DNS SAN and an IP address SAN are distinguishable even if 10.20.30.40 was a valid DNS name, which it is not. Old-fashioned "Common" names are disregarded.
- The certificate must be signed by a trusted CA (or by an intermediate which in turn was trusted, and so on recursively).
I think I hit the big ticket items, there might be some others.
As a researcher it lets you call "bullshit" easily, many years ago I actually used the logs to show that CAs "missed" (did an inadequate job of identifying and revoking) some certificates they should have revoked months earlier, if you attempted that without logs you'd never be sure how much you missed and you'd have no fixed scope for the work.
[[ As to getting hit by bots, keep in mind that passive DNS can cause that too, if anybody (testers, third party services, anybody) does a DNS lookup for name X on most well known public DNS servers, the server operators sell the DNS answer + timestamp (but not who asked so this is not personal information), which you can then buy as a bulk service. So e.g. some-uuid-on-a.server.example gets looked up on an iPhone over somebody's ISP, and within seconds a feed has added some-uuid-on-a.server.example A 10.20.30.40 to the list of known DNS answers. ]]
You can use these monitors, (although not specifically the one operated by Google for much longer) or you can build your own. If you build your own you can choose to specialise. Maybe you only care about certificates for names under .horse, or you only care about certificates from for-profit companies, or with RSA keys.