Since yesterday we've started noticing issues when installing new packages and in some cases with running the
remote_file resource on some of the systems running slightly older Chef versions.
The errors are all related to commands being unable to verify server certificates:
curl -I https://pkg.jenkins.io
curl: (60) server certificate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none More details here: http://curl.haxx.se/docs/sslcerts.html
OpenSSL::SSL::SSLError ---------------------- SSL Error connecting to https://getcomposer.org/composer-1.phar - SSL_connect returned=1 errno=0 state=error: certificate verify failed
apt-get updatewith nginx in the apt sources
E: Failed to fetch https://nginx.org/packages/ubuntu/dists/bionic/nginx/source/Sources server certificate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none
Because this started happening suddenly on several completely separate legacy servers, this meant that it wasn't an isolated issue (i.e. network issues) but more of an 'outside' issue affecting more than one server.
We started looking out for any 'broken ssl' related articles/tweets/etc and found a bunch of articles online related to an expired root certificate that Let's Encrypt use:
Having one of root certificates expired made a lot of sense as to why all these separate servers were affected all of a sudden.
We've tried a lot of the solutions mentioned in the previous articles, such as:
- making sure
opensslis >= 1.0.2g,
However nothing seemed to work.
Starting to mess about with removing root certificates from the
/etc/ssl/certs/ca-certificates.crt file seemed more like a workaround and not a proper solution, and would've been too difficult to easily replicate and deploy on other servers affected by this issue.
As a final attempt (which should've been run probably earlier) we decided to run 'dist-upgrade' to see if there were any packages updated that might contain a fix.
sudo apt-get dist-upgrade finally fixed the issue on that server, so we had a look at all the packages that have been upgraded that could've fixed it, and we isolated the following package to fixing it:
libgnutls-openssl27 (this also upgrades one of its dependencies
sudo apt-get install libgnutls-openssl27
Once we narrowed it down to these packages we were able to upgrade them on other servers where we experienced the issue and could confirm they were all fixed.
Extra Chef solution
This however didn't fully fix the Chef recipes that used the
remote_file resource that were also affected by this issue. After doing some digging it looks like in our case Chef uses its own set of CA certificates bundle found at:
In order to solve this we've created a very simple chef recipe which, first of all, updates the packages mentioned above, and then copies the Ubuntu CA bundle from /etc/ssl/certs/ca-certificates.crt to the path that Chef uses and that fixed all the issues we were experiencing.
cp /etc/ssl/certs/ca-certificates.crt /opt/chef/embedded/ssl/certs/cacert.pem
(Maybe backup your Chef certificate just in case it starts running into any issues afterwards, though we've not experienced anything since)