x509: certificate signed by unknown authority

[bmueller]

Early this morning I started getting "x509: certificate signed by unknown authority" error messages in my console.

After investigating, it appears that a bunch of services (including Heroku, which I use) have started removing the GeoTrust Global CA cert from their root certificates. However, Apple requires the GeoTrust Global CA cert in order to send notifications via tokens.

Is there a workaround for this? Would it be possible to bundle the GeoTrust Global CA cert (https://www.geotrust.com/resources/root_certificates/certificates/GeoTrust_Global_CA.pem) with the client so that everything just works out of the box?

[extrawurst]

for me sudo apt-get install ca-certificates does the trick

[bmueller]

@extrawurst - unfortunately the sudo command is unavailable on Heroku. I'm not sure how to work around this since I can't update the certificates on there manually.

[edmorley]

To add some more context...

The Mozilla CA program has been been progressively removing support for the Symantec/GeoTrust certificate authorities since 2017 due to serious shortcomings in the way the CA was operated:
https://wiki.mozilla.org/CA:Symantec_Issues
https://wiki.mozilla.org/CA/Additional_Trust_Changes#Symantec

Most recently the GeoTrust Global CA cert was removed from the Mozilla CA root certificates store:
https://bugzilla.mozilla.org/show_bug.cgi?id=1670769
https://hg.mozilla.org/projects/nss/rev/4c69d6d0cf210546bef1eed490712462b9296c62

On 2021-02-02, this change was released to all Ubuntu LTS releases, as part of the regular update of the ca-certificates package. For example, here's the Ubuntu 18.04 version of the update:
https://ubuntuupdates.org/package/core/bionic/main/security/ca-certificates
http://launchpad.net/ubuntu/+archive/primary/+files/ca-certificates_20201027ubuntu0.18.04.1_20210119~18.04.1.diff.gz

Debian's equivalent update occurred in:
https://salsa.debian.org/debian/ca-certificates/-/commit/315ae87762dc2edce56042cfa486eb2d92711338

Since https://api.push.apple.com uses GeoTrust Global CA, this means connections to it fail from any up to date Ubuntu and/or Debian installation.

For example:

$ docker run --rm -it ubuntu:20.04 bash
root@1b98119cc104:/# apt-get update -qq
root@1b98119cc104:/# apt-get install -yqq curl
...
root@1b98119cc104:/# curl -I https://api.push.apple.com
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

And similarly the official Go docker image (in the case of this tag, based on Debian Buster):

$ docker run --rm -it golang:1.15.8 curl -I https://api.push.apple.com
curl: (60) SSL certificate problem: unable to get local issuer certificate
...

Ideally Apple would stop using a legacy CA that most of the ecosystem no longer trusts.

However their docs say:

To establish HTTP/2-based TLS sessions with APNs, you must ensure that a GeoTrust Global CA root certificate is installed on each of your providers. If a provider is running macOS, this root certificate is in the keychain by default. On other systems, this certificate might require explicit installation. You can download this certificate from the GeoTrust Root Certificates website. Here is a direct link to the certificate.

...which suggests they are aware of the issue, but are still choosing not to update the CA used, presumably for backwards compatibility reasons (eg old clients with an outdated root cert stores or that have unfortunately hardcoded the CA).

However now that modern OSes/clients are being affected, perhaps this might nudge them into finally doing the right thing.

This issue is affecting Apple Push Notification clients in other languages too, eg:
jchambers/pushy#809
jchambers/pushy#810

In the meantime the only options for systems where the user doesn't control the root store, are:

1. For users to manually configure clients to use the GeoTrust CA for requests to https://api.push.apple.com
2. Or, for clients to bundle the GeoTrust CA cert and automatically configure it when making requests to https://api.push.apple.com

Re (1), is there a way for users of apns2 to configure the CA it uses for outbound requests?

[neilmorton]

Having tried sudo apt-get install ca-certificates as mentioned by @extrawurst above, the GeoTrust Global CA Cert wasn't re-installed.

After looking at the comments from @edmorley, I ended up looking at Debian Bug 962596 which references installing the missing certificate manually, and provides the following recipe:

wget --no-check-certificate -c https://www.geotrust.com/resources/root_certificates/certificates/GeoTrust_Global_CA.pem   \
    && mkdir /usr/local/share/ca-certificates/extra                                                                       \
    && mv GeoTrust_Global_CA.pem /usr/local/share/ca-certificates/extra/GeoTrust_Global_CA.crt                            \
    && update-ca-certificates

The certificate linked above is that which is mentioned in Apple Remote Notification Server documentation.

Once done, I can confirm that APNS2 normal processing resumed.

Putting thought to @bmueller's specific issue relating to not having sudo access or being able to update the certificates manually, whilst I haven't updated APNS2 to specifically pass in the root certificate as opposed to adding the cert to the server, looking at crypto/tls, I wonder if it may be possible to add the certificate via the config object, either using RootCAs or ClientCAs? If so, perhaps this could be passed via apns2/client.go?

[cenkbilgen]

Thanks @neilmorton. That did the trick, slightly different for Arch and maybe other Linux distros.

wget --no-check-certificate -c https://www.geotrust.com/resources/root_certificates/certificates/GeoTrust_Global_CA.pem  \
&& mv GeoTrust_Global_CA.pem /etc/ca-certificates/trust-anchor/anchors/GeoTrust_Global_CA.crt \  
&& trust extract-compat 

[neilmorton]

Good to hear it helped @cenkbilgen. Yes, I should have said that the commands were based on Ubuntu / Debian, so other distros may need tweaking!

[cenkbilgen]

I'd still rather not add this system-wide, even if Apple still trusts it to some extent. But it is works and they are aware of the problem, so more elaborate work-arounds may be unnecessary.

[adityadet]

As per the Apple documentation, they're supporting the GeoTrust certificate only till 29th March 2021 so we need to add both the GeoTrust certificate and the AAACertificateServicesRootCA (Sectigo Portal).

Tweaked @neilmorton 's command with the new certs.

wget --no-check-certificate -c https://www.geotrust.com/resources/root_certificates/certificates/GeoTrust_Global_CA.pem   \
    && wget --no-check-certificate -c -O AAACertificateServices.crt https://comodoca.my.salesforce.com/sfc/dist/version/download/?oid=00D1N000002Ljih&ids=0683l00000G9fLm&d=%2Fa%2F3l000000VbG0%2Fh70Hv.GWfGuD79pR_if0MtGjJFcUj.NRZS_RLqEyC_4&asPdf=false  \
    && mkdir -p /usr/local/share/ca-certificates/extra                                                                       \
    && mv GeoTrust_Global_CA.pem /usr/local/share/ca-certificates/extra/GeoTrust_Global_CA.crt                            \
    && mv AAACertificateServices.crt /usr/local/share/ca-certificates/extra/AAACertificateServices.crt                     \
    && update-ca-certificates

[adityadet]

As per the Apple documentation, they're supporting the GeoTrust certificate only till 29th March 2021 so we need to add both the GeoTrust certificate and the AAACertificateServicesRootCA (Sectigo Portal).

Tweaked @neilmorton 's command with the new certs.

wget --no-check-certificate -c https://www.geotrust.com/resources/root_certificates/certificates/GeoTrust_Global_CA.pem   \
    && wget --no-check-certificate -c -O AAACertificateServices.crt https://comodoca.my.salesforce.com/sfc/dist/version/download/?oid=00D1N000002Ljih&ids=0683l00000G9fLm&d=%2Fa%2F3l000000VbG0%2Fh70Hv.GWfGuD79pR_if0MtGjJFcUj.NRZS_RLqEyC_4&asPdf=false  \
    && mkdir -p /usr/local/share/ca-certificates/extra                                                                       \
    && mv GeoTrust_Global_CA.pem /usr/local/share/ca-certificates/extra/GeoTrust_Global_CA.crt                            \
    && mv AAACertificateServices.crt /usr/local/share/ca-certificates/extra/AAACertificateServices.crt                     \
    && update-ca-certificates

As it turns out, the Comodo CA will be effective March 29 2021, which is imported by default in the CA certs (/etc/ssl/certs/Comodo_AAA_Services_Root.pem). So we need to perform this workaround only till 29th March.

[neilmorton]

Thanks for the update @adityadet .

Looks like Apple got onto this later on yesterday and provided the details surrounding their change on 29th March 2021.

As you mention, the AAACertificateServicesRootCA (Comodo) is already included in the current CA Certs, so assuming root access is available, adding the GeoTrust Global CA mentioned above as a temporary measure until 29th March 2021 seems to be the only required step in order to maintain service.

On the 29th March, once Apple swap to using the AAA Certificate Services root certificate, you can remove the directory /usr/local/share/ca-certificates/extra and run the command update-ca-certificates again, which will remove the temporarily added GeoTrust_Global_CA.crt.

[nimajalali]

We needed a solution that worked on App Engine. AFAIK we can't modify the system certs on App Engine.

So we extended the RootCAs in Go by overriding the http transport.

apnsClient := apns2.NewClient(cert).Production()

// This is needed until March 29 2021 to allow the GeoTrust Root CA to be trusted.
// https://developer.apple.com/news/?id=7gx0a2lp
certpool, err := x509.SystemCertPool()
if err != nil {
    logrus.WithError(err).Panicln("unable to get system cert pool")
}
certpool.AppendCertsFromPEM(geoTrustRootCA)

tlsConfig := &tls.Config{
    Certificates: []tls.Certificate{cert},
    RootCAs: certpool,
}
transport := &http2.Transport{
    TLSClientConfig: tlsConfig,
    DialTLS:         apns2.DialTLS,
}
apnsClient.HTTPClient.Transport = transport

Based on code from https://www.tzeejay.com/blog/2021/01/fix-geotrust-apns/

[goginenigvk]

getting the below error. Can someone look into the issue
Error: Get https://100.64.0.1/api/v1/namespaces/helm/pods?labelSelector=app%3Dhelm%2Cname%3Dtiller: x509: certificate signed by unknown authority

[UAT:]> helm version
Client: &version.Version{SemVer:"v2.12.3", GitCommit:"eecf22f77df5f65c823aacd2dbd30ae6c65f186e", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.12.3", GitCommit:"eecf22f77df5f65c823aacd2dbd30ae6c65f186e", GitTreeState:"clean"}
[UAT:]> kubectl version
Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.12", GitCommit:"e2a822d9f3c2fdb5c9bfbe64313cf9f657f0a725", GitTreeState:"clean", BuildDate:"2020-05-06T05:17:59Z", GoVersion:"go1.12.17", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.12", GitCommit:"a8b52209ee172232b6db7a6e0ce2adc77458829f", GitTreeState:"clean", BuildDate:"2019-10-15T12:04:30Z", GoVersion:"go1.11.13", Compiler:"gc", Platform:"linux/amd64"}

how can we get the tiller-token back if we delete the pod
any ideas?

[neilmorton]

@goginenigvk If this issue has just occurred, have you just created new certificates? If so, it may be worth looking at this: https://developer.apple.com/support/expiration/, in particular Apple Push Notification Service SSL Certificates, so see if this is involved.