[MODULES3307] PUPPETLABSAPT DOESNT DETECT OR ALLOW UPDATING EXPIRED KEYS

[MODULES-3307] puppetlabs-apt doesn't detect or allow updating expired keys Created: 2016/04/28  Updated: 2018/09/27  Resolved: 2018/09/27

Status:

Resolved

Project:

Modules

Component/s:

apt

Affects Version/s:

puppet_agent 1.1.0

Fix Version/s:

None

Type:

Bug

Priority:

Normal

Reporter:

Bill Broadley

Assignee:

Eimhin Laverty

Resolution:

Fixed

Votes:

12

Labels:

None

Remaining Estimate:

Not Specified

Time Spent:

Not Specified

Original Estimate:

Not Specified

Environment:

Ubuntu 14.04, puppet version 3.8.7 with puppetlabs-apt version 2.2.2

Template:

MODULES Bug Template

Epic Link:

apt Overhaul

Team:

Modules

I'm using puppet to manager CRAN's R. Their APT signing key is installed with:

apt::key { 'crankey':

          id      => 'E298A3A825C0D65DFD57CBB651716619E084DAB9',

          server  => 'keyserver.ubuntu.com',

}

This works great on new systems, but on older systems they have an expired key, with the same fingerprint.

There is an ensure => present, but unlike packages there's no ensure => latest. Nor is there any way I could find documented to check if the key is valid, or refresh the key from the keyserver.

So the result is apt fails with:

W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: http://cran.us.r-project.org trusty/ Release: The following signatures were invalid: KEYEXPIRED 1445181253 KEYEXPIRED 1445181253 KEYEXPIRED 1445181253

If I view the key:

pub   2048R/E084DAB9 2010-10-19 [expired: 2015-10-18]

      Key fingerprint = E298 A3A8 25C0 D65D FD57  CBB6 5171 6619 E084 DAB9

uid                  Michael Rutter <[email protected]>

On a newer system:

pub   2048R/E084DAB9 2010-10-19 [expires: 2020-10-16]

      Key fingerprint = E298 A3A8 25C0 D65D FD57  CBB6 5171 6619 E084 DAB9

uid                  Michael Rutter <[email protected]>

sub   2048R/1CFF3E8F 2010-10-19 [expires: 2020-10-16]

Note the fingerprint is the same, but the expiration is different. So the problem is the puppetlabs-apt seems to have no way for me to ask for an up to date key.

Comment by Nick Peelman [ 2016/07/11 ]

Seconded. Just got bit by the apt.puppetlabs.com repo key expiring today.

Even manually setting content via apt-key doesn't seem to take effect since the IDs match...

Short of managing the keys inside of puppet, and having to manually update puppet each time they change, or running via exec and touching a file for idempotency, neither of which is the best way forward, I'm not sure what the action should be here. Touching all 25 of my machines to run `sudo apt-key adv --recv-keys --keyserver keys.gnupg.net 4BD6EC30` isn't a big deal, but if I had 200 machines, there has to be a better way...

Comment by Tom Downes [ 2016/07/11 ]

Hear, hear! Puppet, too!

Comment by Nick Peelman [ 2016/07/11 ]

My hack, for anybody interested:

  # Update puppetlabs apt key

  exec { "update-puppet-labs-key" :

    command => "apt-key adv --recv-keys --keyserver keys.gnupg.net 4BD6EC30 && touch /var/lib/apt/keyrings/2016-apt.puppetlabs.com-key-updated",

    cwd => '/',

    user => root,

    creates => "/var/lib/apt/keyrings/2016-apt.puppetlabs.com-key-updated",

    path => ['/bin', '/sbin', '/usr/bin', '/usr/sbin/', '/usr/local/sbin'],

    require => Apt::Source['puppetlabs'],

  }

Comment by Nick Peelman [ 2016/07/11 ]

After a little more digging I unearthed [this gem](https://gist.github.com/garthk/3726289):

      $key = "4BD6EC30"

      exec { 'apt-key puppetlabs':

        path    => "/bin:/usr/bin",

        unless  => "apt-key list | grep '${key}' | grep -v expired",

        command => "apt-key adv --keyserver keyserver.ubuntu.com --recv-keys ${key}",      

      }

Kudos to garthk. That is a lot cleaner and much more future-proof than a touch file. Lesson learned; hadn't considered doing a chained walk of commands to test for expired keys...that kind of path will come in handy...

All that said, something similar to GarthK's solution could be baked into the apt module rather easily.

Comment by Clayton O'Neill [ 2016/07/11 ]

Couldn't this be fixed by just treating expired keys as absent in the apt_key resource?

Comment by Nick Peelman [ 2016/07/11 ]

I don't think so, because the ID doesn't change. They aren't expiring the old key and replacing it with a new one. They are renewing the old public key by extending its expiration date (insofar as I can tell). Looks like the R maintainers in the OP are doing the same thing.

Comment by Arney [ 2016/08/29 ]

apt_key can already detect whether a key has expired, but this is a read-only property. I guess the main question here is how it should act on this?

• It should check, whether there is a key available at the source

• Is that key actually newer or just the expired key installed already?

• If newer, then replace.

Sure, an expired key is pretty useless, but deleting it without adequate replacement seems risky.

If anyone comes up with the commands to do this, I'd volunteer to implement them.

Comment by Clayton O'Neill [ 2016/08/29 ]

GPG tells you in the listkeys output if the key is expired. One fix would be to treat expired keys as if they weren't present. This would allow replacing keys with unexpired versions, but would cause idempotence issues when the key expires and there is no replacement. I don't know if there is a way to check an existing key to see if it has expired, especially when you may have it as a file on disk, as a string, or it may need to be retrieved from a remote key server.

Comment by Coen de Meijer [ 2016/08/31 ]

Nick Peelman thanks for the gem. I needed to modify it a bit to get it to work for me but it looks like a nice addition. Looking forward to january fifth next year.

Changes I made:

• Removed single quotes surrounding ${key}

• Changed 'grep -v expired' to 'grep expires'

Comment by Christoph Tavan [ 2016/09/03 ]

Nick Peelman thanks for the snippet. I had to change it slightly:

  $key = '4BD6EC30'

  exec { 'apt-key puppetlabs':

    path    => '/bin:/usr/bin',

    onlyif  => "apt-key list | grep ${key} | grep expired",

    command => "apt-key adv --keyserver keyserver.ubuntu.com --recv-keys ${key}",

  }

The reason was that I also had a revoked version of the key on some hosts, so the unless check was not evaluating correctly before:

# apt-key list | grep 4BD6EC30 | grep -v expired

pub   4096R/4BD6EC30 2010-07-10 [revoked: 2014-09-11]

Coen de Meijer I think that's probably what you would also wanted to do, since your change towards grep expires will always fetch the keys with each puppet run since you are checking for keys which will expire in the future instead of for keys which have expired.

Comment by Nick Peelman [ 2016/09/03 ]

That works. I might have just added `-v revoked` (you can keep adding strings to grep and it will concat matches as needed). I don't think inverting the logic will cause problems, but its a holiday weekend and my brain has checked out so I might not be thinking through all the edge cases well enough.

Comment by Coen de Meijer [ 2016/09/29 ]

Christoph Tavan I had to let go of the grep -v expired because it would not produce the desired result in every case.

It would return with 0 exit code (which is what we want) if the previous command produced multiple lines.
But... It returns 1 if the previous command only produces a single line of output.

Grep -v returns 0 (as desired):

cat << __EOT | grep -v expired;echo $?

yadda

expired

__EOT

Grep -v returns 1 (bummer):

cat << __EOT | grep -v expired;echo $?

expired

__EOT

As I understand it, when I have one line that expires in the future, there is no need to run the command. Which I believe is what my code does now (according to the documentation of the unless attribute for exec: "If this parameter is set, then this exec will run unless the command has an exit code of 0."

My puppet code right now (which does not fetch the keys at every run):

  # Handle any expired keys. The $update_expired_keys array should just contain

  # a list of names that occur in the $::apt::keys array in the configuration.

  #

  # The indicated apt keys are updated unless a key exists that has an expiry

  # date in the future (status: expires).

  # An apt-get update is triggered if any key is actually updated.

  #

  $update_expired_keys.each |$k| {

    if($::apt::keys.has_key($k)) {

      $values=$::apt::keys[$k]

      $key = $values['id'][-8,8]

      notify{"Checking whether apt key has expired [name=${k}, key=${key}]":}

      exec { "update expired apt-key ${k}":

        before  => Exec['apt_update'],

        command => "apt-key adv --keyserver keyserver.ubuntu.com --recv-keys ${key}",

        notify  => Exec['apt_update'],

        path    => '/bin:/usr/bin',

        unless  => "apt-key list | grep ${key} | grep expires",

      }

    }

  }

In my hiera config I simply have an array that lists the names of the keys that need to be checked.

Comment by Christoph Tavan [ 2016/09/29 ]

Coen de Meijer I'm still not sure about your suggestion... What about keys that never expire? I understand that you will probably not add those keys to your hiera config, but nonetheless it duplicates configuration.

I also don't really understand what the problem is with:

onlyif  => "apt-key list | grep ${key} | grep expired",

Can you elaborate? That's why I'm using onlyif in combination with grep expired and not unless in combination with grep -v expired (which would exhibit the problem you describe above...

Thanks!

Comment by Coen de Meijer [ 2016/09/29 ]

I'm not sure what you mean by duplication here. I have the puppetlabs key in hiera and then a separate array with the names of the keys that may need to be updated. The puppetlabs key is the only one with this issue so far. Feels like a minimum of additional code (additional array and a tiny bit of puppet script).

To be honest, I don't know why I stuck with the unless. I actually like your suggestion better. I vaguely remember there being some reason, but I'm not sure. Maybe I just forgot the first grep?

Comment by Herwig Bogaert [ 2016/10/28 ]

I solved this problem by adding a refresh parameter to the apt_key resource.
When refresh => true, puppet will try to recreate a key that is expired.

https://github.com/hbog/puppetlabs-apt/tree/refresh

Comment by Mithil Patel [ 2017/03/30 ]

Inspite of running the command manually, this doesn't seem to be fixed. Output as below:

root@netlogin-test-01:~# apt-key list | grep expired
pub 4096R/4BD6EC30 2010-07-10 [expired: 2017-01-05]

root@netlogin-test-01:~# apt-key adv --recv-keys --keyserver keyserver.ubuntu.com 4BD6EC30
Executing: gpg --ignore-time-conflict --no-options --no-default-keyring --homedir /tmp/tmp.DPryggE0R0 --no-auto-check-trustdb --trust-model always --keyring /etc/apt/trusted.gpg --primary-keyring /etc/apt/trusted.gpg --keyring /etc/apt/trusted.gpg.d/debian-archive-squeeze-automatic.gpg --keyring /etc/apt/trusted.gpg.d/debian-archive-squeeze-stable.gpg --keyring /etc/apt/trusted.gpg.d/debian-archive-wheezy-automatic.gpg --keyring /etc/apt/trusted.gpg.d/debian-archive-wheezy-stable.gpg --keyring /etc/apt/trusted.gpg.d/puppetlabs-keyring.gpg --keyring /etc/apt/trusted.gpg.d/puppetlabs-nightly-keyring.gpg --recv-keys --keyserver keyserver.ubuntu.com 4BD6EC30
gpg: requesting key 4BD6EC30 from hkp server keyserver.ubuntu.com
gpg: key 4BD6EC30: "Puppet Labs Release Key (Puppet Labs Release Key) <[email protected]>" not changed
gpg: key 4BD6EC30: "Puppet Labs Release Key (Puppet Labs Release Key) <[email protected]>" not changed
gpg: key 4BD6EC30: "Puppet Labs Release Key (Puppet Labs Release Key) <[email protected]>" not changed
gpg: Total number processed: 3
gpg: unchanged: 3

root@netlogin-test-01:~# apt-key list | grep expired
pub 4096R/4BD6EC30 2010-07-10 [expired: 2017-01-05]

If this command doesn't even work manually then no point in creating an exec resource. Has anyone faced this issue as well?

Comment by Eimhin Laverty [ 2018/09/05 ]

Hey Herwig Bogaert I'm just following up to see if the solution you have specified was able to remediate the issue of updating an expired key? 

Comment by Herwig Bogaert [ 2018/09/05 ]

Hi Eimhin Laverty. the solution in https://github.com/hbog/puppetlabs-apt/tree/refresh worked and has been running for a while. In it latest version, I used 'ensure => refreshed' to make it update expired keys.  I stopped using the branch a few months ago, because it was no longer needed in our setup.  Note that we were running Puppet 3.8 at the time it was in use.

Comment by Eimhin Laverty [ 2018/09/05 ]

Hey Herwig Bogaert I'll use the additions you have added to the repo to figure out a solution for the current version of the module if you don't mind. Thanks!

Comment by Herwig Bogaert [ 2018/09/06 ]

Feel free Eimhin Laverty.

Comment by Eimhin Laverty [ 2018/09/24 ]

Just updating this ticket to let you all know that I have merged a pull request which introduces functionality to allow you to have your keys automatically update when expired. This can be achieved by simply setting the `ensure` property of your apt::key resources to `refreshed`. Many thanks to Herwig Bogaert for his solution which was a big help. I hope this helps solve your issues but please feel free to provide feedback (or a PR  ) if you feel there is any way it can be improved. 

Comment by Eimhin Laverty [ 2018/09/27 ]

I'm going to go ahead and resolve this ticket now but if there are issues that you feel need re-addressed feel free to re-open the ticket. Thanks!