HN Explorer

49 comments

[ 0.20 ms ] story [ 116 ms ] thread
GOATS-
This also applies to their avatars subdomain, causing them not to load anymore.
sha-3
Are you sure? avatars.githubusercontent.com works fine for me.
guessmyname
> Are you sure? avatars.githubusercontent.com works fine for me.

That’s because they already resolved the issue → https://www.githubstatus.com/incidents/x7njwb481j9b

This is what I saw an hour ago:

        $ echo | openssl s_client -connect avatars.githubusercontent.com:443
        CONNECTED(00000005)
        depth=2 C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert Global Root CA
        verify return:1
        depth=1 C = US, O = DigiCert Inc, CN = DigiCert TLS RSA SHA256 2020 CA1
        verify return:1
        depth=0 C = US, ST = California, L = San Francisco, O = "GitHub, Inc.", CN = *.github.io
    >>> verify error:num=10:certificate has expired
        notAfter=Mar 21 23:59:59 2023 GMT
        verify return:1
        depth=0 C = US, ST = California, L = San Francisco, O = "GitHub, Inc.", CN = *.github.io
        notAfter=Mar 21 23:59:59 2023 GMT
        verify return:1
koolba
The cert for objects.githubusercontent.com has also expired:

    $ openssl s_client -connect objects.githubusercontent.com:443

    CONNECTED(00000005)
    depth=2 C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert Global Root CA
    verify return:1
    depth=1 C = US, O = DigiCert Inc, CN = DigiCert TLS RSA SHA256 2020 CA1
    verify return:1
    depth=0 C = US, ST = California, L = San Francisco, O = "GitHub, Inc.", CN = *.github.io
    verify error:num=10:certificate has expired
    notAfter=Mar 21 23:59:59 2023 GMT
    verify return:1
    depth=0 C = US, ST = California, L = San Francisco, O = "GitHub, Inc.", CN = *.github.io
    notAfter=Mar 21 23:59:59 2023 GMT
    verify return:1
    ---
    Certificate chain
     0 s:C = US, ST = California, L = San Francisco, O = "GitHub, Inc.", CN = *.github.io
       i:C = US, O = DigiCert Inc, CN = DigiCert TLS RSA SHA256 2020 CA1
     1 s:C = US, O = DigiCert Inc, CN = DigiCert TLS RSA SHA256 2020 CA1
       i:C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert Global Root CA
What are the odds this happens the same day they rotate their SSH keys?
AviationAtom
Could be a good chance. I'd venture to guess they failed to update the known_hosts file for one of their systems that handles certificate management. Strictly me taking a stab at the answer though.
ksml
They're serving the wrong cert on pkg-containers.githubusercontent.com (it's for *.githubassets.com) and their support site also expired 3/21... https://support.github.com/ What is going on over there?
gorjusborg
And today of all days I have a moment to upgrade homebrew stuff.
carrina
Not Before Fri, 18 Mar 2022 00:00:00 GMT

Not After Tue, 21 Mar 2023 23:59:59 GMT

3-day certs.

GOATS-
There's a whole year between those dates.
warent
3 day plus 365 days ;) you're missing a full year
vehementi
Short lived certs are a thing, also
tonto
got a "RequestError: certificate has expired" doing a release just now...as usual, not a good idea to release on a friday
bvogelzang
It looks as though it's back for me now. Status page is now showing the problem: https://www.githubstatus.com/
deathanatos
… well, the status page is back to green now, but AFAICT, the domains are still serving the expired cert.

  » TIMEZONE=UTC date; openssl s_client -connect support.github.com:443 2>&1 | grep 'cert.*has.*ex'
  Fri Mar 24 17:40:28 EDT 2023
  verify error:num=10:certificate has expired
      Verify return code: 10 (certificate has expired)
The previous incident seems pretty clearly to be this … so it seems like they think they fixed it…
nwatson
They laid off the wrong person.
AviationAtom
This is so comical because it's so relevant.

Do you think people architect poorly designed systems more often than not as a means of job security or just a failure to put much forethought in whilst planning it?

tomrod
Failure of forethought with the wrong deadlines in place, soup to nuts.
reaperducer
Or maybe just laziness.

I know someone who joined a company and found a dead-man's switch in the server.

He could have taken it out, but instead he just resets it every three months, just like the guy before him.

If the company ever gets rid of him and doesn't hire someone equally skilled and thorough, the production server will eat itself right about the time his unemployment benefits run out.

misnome
I’d more bet on “Warning for years that this is a point of failure but management wants to chase $shiny instead”
remram
I guess copilot can't write monitoring rules.
blooalien
> I’d more bet on “Warning for years that this is a point of failure but management wants to chase $shiny instead”

I'd almost guarantee you're right on the money with that line of thinking…

lanstin
Or the team is on a new project and after ten attempts to get new owners have an outlook rule to delete any mails about the old project.

The only way to do cert renewal at an org level is one well organized team of not creative software types. yeah yeah the team will automate but in the meantime someone has to check all the dates carefully. And usually good public certs can't be fully automated, at least in the deploy bit.

I heard about a new cert once with a longer private key that cauaed all the terminating F5s to fall over due to out of CPU

Kelamir
Previously I had the same issue, but it works for me now, as well as for a friend in another EU country.
[deleted]
(comment deleted)
ccheney
EDIT: this specific issue is resolved

Failing for us in GitHub Actions

For SEO purposes:

  npm ERR! code ERR_TLS_CERT_ALTNAME_INVALID
  npm ERR! errno ERR_TLS_CERT_ALTNAME_INVALID
  npm ERR! request to https://pkg- 
 npm.githubusercontent.com/npmregistryv2prod/blobs/\*\* failed, reason: 
  Hostname/IP does not match certificate's altnames: Host: pkg-npm.githubusercontent.com. is not in the cert's altnames: DNS:\*.githubassets.com, DNS:githubassets.com
[deleted]
(comment deleted)
apetresc
Seems to be resolved now. My `brew update` works again.
lytedev
Back up now, it looks like.
dz0ny
Still some weird stuff around (* subject: CN=apistatus.chorus.co.nz).

    curl https://www.githubstatus.com/ -vvvv -I
    \*   Trying 52.215.192.131:443...
    \* Connected to www.githubstatus.com (52.215.192.131) port 443 (#0)
    \* ALPN: offers h2
    \* ALPN: offers http/1.1
    ...
    \* SSL connection using TLSv1.3 / AEAD-AES256-GCM-SHA384
    \* ALPN: server accepted h2
    \* Server certificate:
    \*  subject: CN=apistatus.chorus.co.nz
    \*  start date: Mar  6 23:10:30 2023 GMT
    \*  expire date: Jun  4 23:10:29 2023 GMT
    \*  subjectAltName: host "www.githubstatus.com" matched cert's "www.githubstatus.com"
    \*  issuer: C=US; O=Let's Encrypt; CN=R3
    \*  SSL certificate verify ok.
    \* Using HTTP2, server supports multiplexing
jjice
Well I'm kind of just waiting on PRs for the rest of the day today and it's a Friday, so I'll consider this a modern equivalent of https://xkcd.com/303/
reedjosh
cert-manager though amiright?

Like what's going on there?

varenc
This seems to now be fixed.
jmspring
Sounds like whoever is in charge of certificates at GH must have come over from MSFT. Afterall, I think Microsoft has had 2-3 certificate expiry issues in the last several years.
jiggawatts
Azure had several global outages because of issues with certificates. One outage was caused by an incorrect date computation: the certificates last for one year, and this was computed with: "new DateTime(now.Year+1,now.Month,now.Day)".

If you do that on Feb 29th of a leap year, it'll throw an exception because the next year doesn't have a Feb 29th! Oops.

They "fixed" it and promptly had another related outage the very next day.

jmspring
That was one of them I recall.
artyom
ChatGPT, rotate my certs
pmontra
It needs a plugin for that.
bink
Did we learn nothing from Son of Anton?
slowmovintarget
Why? Don't most cloud providers have auto-renewing certs now?
gunshai
For us dumb dumbs what does this mean?