As some community members noticed earlier this week, there was a problem with the GPG Keys and signature validation associated with 1.x TICK stack components.
There have been a number of digital supply chain attacks as of late and we have been discussing a wide range of security related topics to improve our overall security posture for our community. One of these is ensuring that we periodically rotate our signing keys.
In attempting to perform a periodic key rotation, we encountered some unanticipated downstream effects and this is the reason for the temporary issue. We have rolled back the change for now as we look to make this process smoother and ensure that we do a better job of communicating this kind of change to our community members.
We deeply apologize for any inconvenience, and lack of prior notification. We appreciate all of those community members who reached out to check in, confirm and alert us to the situation. You are doing the right thing – and we are happy to see that our community is leveraging the security measures that we have in place.
Once we have remediated the issues we encountered, we will perform the key rotation and communicate that change as a part of our improved process.
Thank you again for being part of the InfluxData Community.
Tim
2 Likes
On the Debian side at least, the problems were minor and easy to work around.
Glad to see this work, even if an egg or two gets broken.
thanks for your understanding.
Hrm - revisiting this now. Unfortunately the rollback method ended up penalizing those of us who responded quickly to the change and augmented our configuration management code to cope with the change.
Now that you’ve rolled back in-place, we have to shuffle our Ansible code again to respond to the roll-back.
That’s understandable enough in light of the actions taken upstream both in the original change, then in the roll-back. But this could be done more cleanly in the future by not attempting to shuffle the APT keys using the same name for the current key ‘influxdb.key’ while cycling the old key to ‘influxdb.key.old’.
I’d like to recommend that you adopt a different strategy for subsequent key change cycles and use a scheme similar to this:
influxdb.2021.key
influxdb.2023.key
This is more predictable and easier to cope with changes since we don’t have to shuffle our current keys and fingerprints around, but rather just have to accept the new key as it becomes available.
wow…super great and specific feedback. Thank you for sharing. We’ll get this reviewed.
Much appreciated.
It looks like this issue is still present for versions 1.7.7 through 1.8.4 (those are the ones I’ve tested). The GPG check only appears to work for version 1.8.5.
Hi @JanekLehr. What package(s) are you targeting and what part of your workflow is failing? Can you describe the steps you’ve taken, what you expected to have happen, and what did happen? We’d like to get to the bottom of this.
Thanks!
Hi @codyshepherd,
I followed the instructions here for installing on Red Hat & CentOS. To reproduce the error, I spun up a brand new CentOS 7 Docker image and ran a yum install -y influxdb-1.<version> after adding the influx yum repo, per the instructions. I tried all versions from 1.7.7 through 1.8.5, except for 1.8.1 because of the memory bug.
Here’s an example failure:
Downloading packages:
warning: /var/cache/yum/x86_64/7/influxdb/packages/influxdb-1.7.7.x86_64.rpm: Header V4 RSA/SHA256 Signature, key ID e8ee6fd0: NOKEY45 MB 00:00:00 ETA
Public key for influxdb-1.7.7.x86_64.rpm is not installed
influxdb-1.7.7.x86_64.rpm | 48 MB 00:00:03
Retrieving key from https://repos.influxdata.com/influxdb.key
Importing GPG key 0x2582E0C5:
Userid : "InfluxDB Packaging Service <support@influxdb.com>"
Fingerprint: 05ce 1508 5fc0 9d18 e99e fb22 684a 14cf 2582 e0c5
From : https://repos.influxdata.com/influxdb.key
Public key for influxdb-1.7.7.x86_64.rpm is not installed
Failing package is: influxdb-1.7.7-1.x86_64
GPG Keys are configured as: https://repos.influxdata.com/influxdb.key
After that error, attempting a later version install fails with:
warning: /var/cache/yum/x86_64/7/influxdb/packages/influxdb-1.8.3.x86_64.rpm: Header V4 RSA/SHA256 Signature, key ID e8ee6fd0: NOKEY57 MB 00:00:00 ETA
Public key for influxdb-1.8.3.x86_64.rpm is not installed
influxdb-1.8.3.x86_64.rpm | 61 MB 00:00:04
Retrieving key from https://repos.influxdata.com/influxdb.key
The GPG keys listed for the "InfluxDB Repository - RHEL 7" repository are already installed but they are not correct for this package.
Check that the correct key URLs are configured for this repository.
Still on the same container, running the install for 1.8.5 succeeds with:
Downloading packages:
influxdb-1.8.5.x86_64.rpm | 61 MB 00:00:02
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Installing : influxdb-1.8.5-1.x86_64 1/1
Created symlink /etc/systemd/system/influxd.service, pointing to /usr/lib/systemd/system/influxdb.service.
Created symlink /etc/systemd/system/multi-user.target.wants/influxdb.service, pointing to /usr/lib/systemd/system/influxdb.service.
Verifying : influxdb-1.8.5-1.x86_64 1/1
Installed:
influxdb.x86_64 0:1.8.5-1
Complete!
Hi @JanekLehr, I’ve reproduced the behavior you’ve described and will provide an update as I investigate further. Thanks for bringing this to our attention.
1 Like
Hi @JanekLehr, the behavior you’ve identified appears to have been due to the fact that the old rpms were still signed with the key we created as part of our key rotation initiative earlier this month. Since we’ve postponed that initiative for the moment, the rpms have been re-signed using the original key. The installation process for 1.7.6 and forward should work without the key error now. Packages older than 1.7.6 are in the process of being similarly re-signed.
Thanks again for bringing this up!
Thank you @codyshepherd, I’ve confirmed that it’s back to working on my end as well.
Hi @codyshepherd I think version 1.7.10 was missed in the latest fix. I’ve successfully installed influxdb-1.7.9 and influxdb-1.7.11 in a centos7 container, but influxdb-1.7.10 is still failing.
1.7.10 should be working now. Thanks for pointing it out!
Thank you. Just checked, and I can confirm it’s working for me!
This issue seems to have arisen again. I see that the GPG key was updated yesterday, and it looks like packages may have been re-signed based on timestamps I see in the repo.
But:
“warning: /var/cache/yum/x86_64/7/influxdb/packages/telegraf-1.9.1-1.x86_64.rpm: Header V4 RSA/SHA512 Signature, key ID 7df8b07e: NOKEY\n\n\nThe GPG keys listed for the "InfluxDB Repository - RHEL 7" repository are already installed but they are not correct for this package.\nCheck that the correct key URLs are configured for this repository.\n\n\n Failing package is: telegraf-1.9.1-1.x86_64\n GPG Keys are configured as: https://repos.influxdata.com/influxdb.key\n\n”
