clair: unable to run updaters

[hbandi]

we are observing frequent errors on Clair logs and also Clair is not detecting any vulns its always showing 0 for all images.

{"level":"error","component":"libvuln/New","component":"libvuln/Libvuln/loopUpdaters","error":"updating errors:\n\talpine-main-v3.9-updater: alpine: error making request: Get \"https://secdb.alpinelinux.org/v3.9/main.json\": http: serve
r gave HTTP response to HTTPS client\n\talpine-community-v3.7-updater: alpine: error making request: Get \"https://secdb.alpinelinux.org/v3.7/community.json\": http: server gave HTTP response to HTTPS client\n\talpine-main-v3.8-updater:
 alpine: error making request: Get \"https://secdb.alpinelinux.org/v3.8/main.json\": http: server gave HTTP response to HTTPS client\n\talpine-community-v3.4-updater: alpine: error making request: Get \"https://secdb.alpinelinux.org/v3.
4/community.json\": http: server gave HTTP response to HTTPS client\n\talpine-main-v3.6-updater: alpine: error making request: Get \"https://secdb.alpinelinux.org/v3.6/main.json\": http: server gave HTTP response to HTTPS client\n\talpi
ne-community-v3.11-updater: alpine: error making request: Get \"https://secdb.alpinelinux.org/v3.11/community.json\": http: server gave HTTP response to HTTPS client\n\talpine-main-v3.3-updater: alpine: error making request: Get \"https
://secdb.alpinelinux.org/v3.3/main.json\": http: server gave HTTP response to HTTPS client\n\talpine-community-v3.10-updater: alpine: error making request: Get \"https://secdb.alpinelinux.org/v3.10/community.json\": http: server gave HT
TP response to HTTPS client\n\talpine-community-v3.6-updater: alpine: error making request: Get \"https://secdb.alpinelinux.org/v3.6/community.json\": http: server gave HTTP response to HTTPS client\n\tdebian-buster-updater: failed to r
etrieve OVAL database: Get \"https://www.debian.org/security/oval/oval-definitions-buster.xml\": http: server gave HTTP response to HTTPS client\n\tsuse-updater-suse.linux.enterprise.server.11: Get \"https://support.novell.com/security/
oval/suse.linux.enterprise.server.11.xml\": http: server gave HTTP response to HTTPS client\n\tsuse-updater-opensuse.leap.15.0: Get \"https://support.novell.com/security/oval/opensuse.leap.15.0.xml\": http: server gave HTTP response to
HTTPS client\n\talpine-main-v3.4-updater: alpine: error making request: Get \"https://secdb.alpinelinux.org/v3.4/main.json\": http: server gave HTTP response to HTTPS client\n\tdebian-wheezy-updater: failed to retrieve OVAL database: Ge
t \"https://www.debian.org/security/oval/oval-definitions-wheezy.xml\": http: server gave HTTP response to HTTPS client\n\tdebian-stretch-updater: failed to retrieve OVAL database: Get \"https://www.debian.org/security/oval/oval-definit
ions-stretch.xml\": http: server gave HTTP response to HTTPS client\n\tsuse-updater-opensuse.leap.15.1: Get \"https://support.novell.com/security/oval/opensuse.leap.15.1.xml\": http: server gave HTTP response to HTTPS client\n\talpine-c
ommunity-v3.9-updater: alpine: error making request: Get \"https://secdb.alpinelinux.org/v3.9/community.json\": http: server gave HTTP response to HTTPS client\n\tsuse-updater-suse.linux.enterprise.server.12: Get \"https://support.novel
l.com/security/oval/suse.linux.enterprise.server.12.xml\": http: server gave HTTP response to HTTPS client\n\talpine-main-v3.10-updater: alpine: error making request: Get \"https://secdb.alpinelinux.org/v3.10/main.json\": http: server g
ave HTTP response to HTTPS client\n\taws-linux2-updater: failed to create client: failed to make request for linux2 mirrors: Get \"https://cdn.amazonlinux.com/2/core/latest/x86_64/mirror.list\": http: server gave HTTP response to HTTPS
client\n\tdebian-jessie-updater: failed to retrieve OVAL database: Get \"https://www.debian.org/security/oval/oval-definitions-jessie.xml\": http: server gave HTTP response to HTTPS client\n\tphoton-updater-photon3: Get \"https://packag
es.vmware.com/photon/photon_oval_definitions/com.vmware.phsa-photon3.xml\": http: server gave HTTP response to HTTPS client\n\talpine-main-v3.7-updater: alpine: error making request: Get \"https://secdb.alpinelinux.org/v3.7/main.json\":
 http: server gave HTTP response to HTTPS client\n\talpine-community-v3.3-updater: alpine: error making request: Get \"https://secdb.alpinelinux.org/v3.3/community.json\": http: server gave HTTP response to HTTPS client\n\talpine-main-v
3.12-updater: alpine: error making request: Get \"https://secdb.alpinelinux.org/v3.12/main.json\": http: server gave HTTP response to HTTPS client\n\talpine-community-v3.5-updater: alpine: error making request: Get \"https://secdb.alpin
elinux.org/v3.5/community.json\": http: server gave HTTP response to HTTPS client\n\tphoton-updater-photon1: Get \"https://packages.vmware.com/photon/photon_oval_definitions/com.vmware.phsa-photon1.xml\": http: server gave HTTP response
 to HTTPS client\n\tphoton-updater-photon2: Get \"https://packages.vmware.com/photon/photon_oval_definitions/com.vmware.phsa-photon2.xml\": http: server gave HTTP response to HTTPS client\n\talpine-community-v3.8-updater: alpine: error
making request: Get \"https://secdb.alpinelinux.org/v3.8/community.json\": http: server gave HTTP response to HTTPS client\n\talpine-community-v3.12-updater: alpine: error making request: Get \"https://secdb.alpinelinux.org/v3.12/commun
ity.json\": http: server gave HTTP response to HTTPS client\n\talpine-main-v3.5-updater: alpine: error making request: Get \"https://secdb.alpinelinux.org/v3.5/main.json\": http: server gave HTTP response to HTTPS client\n\talpine-main-
v3.11-updater: alpine: error making request: Get \"https://secdb.alpinelinux.org/v3.11/main.json\": http: server gave HTTP response to HTTPS client\n\tsuse-updater-suse.linux.enterprise.server.15: Get \"https://support.novell.com/securi
ty/oval/suse.linux.enterprise.server.15.xml\": http: server gave HTTP response to HTTPS client\n","time":"2021-01-07T14:44:39Z","time":"2021-01-07T14:44:39Z","message":"unable to run updaters"}

• Clair version/image: clairV4.0.0
• Clair client name/version: clairctl
• Host OS: centos
• Kernel (e.g. uname -a):
• Kubernetes version (use kubectl version):
• Network/Firewall setup:

[hdonnay]

It looks like none of the updaters were able to run, as signified by

http: server gave HTTP response to HTTPS client

and

"message":"unable to run updaters"

Sounds like some sort of proxy to me. The standard http_proxy environment variable should be supported, but it's possible an unconfigured client is in use somewhere.

[hbandi]

Thanks, @hdonnay
It's a proxy issue, we need to white list all the updater URLs at our end.
From logs, we found these 3 URLs.
www.debian.org
secdb.alpinelinux.org
support.novell.com
packages.vmware.com

where can we find the list of all updater URLs?

[hbandi]

From the code we found this super set
secdb.alpinelinux.org
www.debian.org
linux.oracle.com
packages.vmware.com
people.canonical.com
www.redhat.com
please let us know anything we missed.

[thomasmckay]

https://issues.redhat.com/browse/PROJQUAY-1444

[hbandi]

@hdonnay @thomasmckay
Thanks for your help

After adding the above URLs to proxy config,looks like it worked .
But now we are 403 error, please help us on this.

{"level":"error","component":"libvuln/New","component":"libvuln/Libvuln/loopUpdaters","error":"updating errors:\n\tRHEL8-openstack-16.0: ovalutil: fetcher got unexpected HTTP response: 403 (403 Forbidden)\n\tRHEL6-satellite-tools-6.3: o
valutil: fetcher got unexpected HTTP response: 403 (403 Forbidden)\n\tRHEL7-satellite-6.5: ovalutil: fetcher got unexpected HTTP response: 403 (403 Forbidden)\n\tRHEL7-satellite-6.8: ovalutil: fetcher got unexpected HTTP response: 403 (
403 Forbidden)\n\tRHEL7-rhel-7.2-e4s: ovalutil: fetcher got unexpected HTTP response: 403 (403 Forbidden)\n\tRHEL8-openshift-service-mesh-1.0: ovalutil: fetcher got unexpected HTTP response: 403 (403 Forbidden)\n\tRHEL8-satellite-tools-
6.8: ovalutil: fetcher got unexpected HTTP response: 403 (403 Forbidden)\n\tRHEL8-openshift-4.1: ovalutil: fetcher got unexpected HTTP response: 403 (403 Forbidden)\n\tRHEL6-satellite-proxy-5.8: ovalutil: fetcher got unexpected HTTP res
ponse: 403 (403 Forbidden)\n\talpine-main-v3.5-updater: alpine: error making request: Get "https://secdb.alpinelinux.org/v3.5/main.json\": http: server gave HTTP response to HTTPS client\n\tRHEL7-rhel-7.3-tus: ovalutil: fetcher got une
xpected HTTP response: 403 (403 Forbidden)\n\tRHEL8-rhel-8.4-eus: ovalutil: fetcher got unexpected HTTP response: 403 (403 Forbidden)\n\tRHEL8-satellite-tools-6.7: ovalutil: fetcher got unexpected HTTP response: 403 (403 Forbidden)\n\ta
lpine-community-v3.5-updater: alpine: error making request: Get "https://secdb.alpinelinux.org/v3.5/community.json\": http: server gave HTTP response to HTTPS client\n\tRHEL6-rhvh-4-including-unpatched: ovalutil: fetcher got unexpected
HTTP response: 403 (403 Forbidden)\n\tRHEL7-openshift-3.5-including-unpatched: ovalutil: fetcher got unexpected HTTP response: 403 (403 Forbidden)\n\tRHEL7-jboss-cs-including-unpatched: ovalutil: fetcher got unexpected HTTP r

[hdonnay]

That's an issue with the server hosting the file, not Clair. We're trying to hunt it down, but it's an intermittent issue with another Red Hat team's server.

[hbandi]

Thank you very much for your response.
But we are also observing Clair is not able to detect vulnerabilities for any image, getting always empty vulns.
Can you help us to debug this issue?
Thanks

[hdonnay]

I'll take a look if you provide any details to reproduce with. To start, what's the image?

[ldelossa]

@hbandi is this still an issue?

[hbandi]

Thanks for asking @ldelossa
have not tried again, will try this week and update you guys.
Thanks,
Hanumanth

[brtkwr]

Yes I think it is, my Debian based image was only showing 23 unknown CVEs which on Quay.io was reporting 200+ of different kinds. I waiting a few days for this to change but no change. Not sure what I need to do to get the updater to run.