ca_extend

Table of Contents

1. Overview
2. Description - What the module does and why it is useful
3. Setup - The basics of getting started with this module
4. Usage - Configuration options and additional functionality
5. Reference - An under-the-hood peek at what the module is doing
6. Development - Guide for contributing to the module

Overview

This module can extend a certificate authority (CA) that's about to expire or has already expired.

A Puppet CA certificate is only valid for a finite time (a new installation of PE 2019.x / Puppet 6.x will create a 15 year CA, while earlier versions will create a 5 year CA; and upgrading does not extend the CA.), after which it expires. When a CA certificate expires, Puppet services will no longer accept any certificates signed by that CA, and your Puppet infrastructure will immediately stop working.

If your CA certificate is expiring soon (or it's already expired), you need to:

• Generate a new CA certificate using the existing CA keypair.
• Distribute the new CA certificate to agents.

This module can automate those tasks.

Description

This module is composed of Plans and Tasks to extend the expiration date of the CA certificate in Puppet Enterprise (and Puppet Open Source) and distribute that CA certificate to agents.

Note that, with Puppet Open Source, if the CA certificate is only used by the Puppet CA and no other integrations, there is no further action to take after using the two Plans. However, if it is used for other integrations (such as SSL encrypted PuppetDB traffic) then those integrations will need to have their copy of the CA certificate updated. If the CA certificate is stored in any keystores, those will also need to be updated.

The functionality of this module is composed into two Plans:

• ca_extend::extend_ca_cert
  • Extend the CA certificate and configure the primary Puppet server, Replica, Compilers, and Postgres nodes to use that extended certificate.
• ca_extend::upload_ca_cert
  • Distribute the CA certificate to agents using transport supported by Puppet Bolt, such as ssh and winrm.

Regardless of whether the CA certificate is expired, the extend_ca_cert plan may be used to extend its expiration date in-place and configure the primary Puppet server and any Compilers to use it.

After the CA certificate has been extended, there are three methods for distributing it to agents:

1. Using the ca_extend::upload_ca_cert plan or another method to copy the CA certificate to agents.
2. Manually deleting ca.pem on agents and letting them download that file as part of the next Puppet agent run. The agent will download that file only if it is absent, so it must be deleted to use this method.
3. Using a Puppet file resource to manage ca.pem. Note: This method is only possible if the CA certificate has not yet expired because Puppet communications depend upon a valid CA certificate.

There are also complementary tasks to check the expiration date of the CA certificate, agent certificates, and the CA CRL.

• ca_extend::check_ca_expiry
  • Checks if the CA certificate expires by a certain date. Defaults to three months from today.
• ca_extend::check_agent_expiry
  • Checks if any agent certificate expires by a certain date. Defaults to three months from today.
• ca_extend::check_crl_expiry
  • Checks if the CA crl on the primary server has expired
• ca_extend::crl_truncate
  • Will truncate and regenerate the CA CRL, this should only be run if the CRL is expired

** If the CA certificate is expiring or expired, you must extend it as soon as possible. **

Setup

This module requires Puppet Bolt >= 1.38.0 on either on the primary Puppet server or a workstation with connectivity to the primary.

The installation procedure will differ depending on the version of Bolt. If possible, using Bolt >= 3.0.0 is recommended. For example, this will install the latest Bolt version on EL 7.

sudo rpm -Uvh https://yum.puppet.com/puppet-tools-release-el-7.noarch.rpm
sudo yum install puppet-bolt

The following two sections show how to install the module dependencies depending on the installed version of Bolt.

Bolt >= 1.38.0 < 3.0.0

The recommended procedure for these versions is to use a Bolt Puppetfile. From within a Boltdir, specify this module and puppetlabs-stdlib as dependencies and run bolt puppetfile install. For example:

mkdir -p ~/Boltdir
cd ~/Boltdir

cat >>Puppetfile <<EOF
mod 'puppetlabs-stdlib'

mod 'puppetlabs-ca_extend'
EOF

bolt puppetfile install

Bolt >= 3.0.0

The recommended procedure for these versions is to use a Bolt Project. When creating a Bolt project, specify this module and puppetlabs-stdlib as dependencies and initialize the project. For example:

sudo rpm -Uvh https://yum.puppet.com/puppet-tools-release-el-7.noarch.rpm
sudo yum install puppet-bolt

If your primary Puppet server or workstation has internet access, the project can be initialized with the needed dependencies with the following:

mkdir ca_extend
cd ca_extend

bolt project init expiry --modules puppetlabs-stdlib,puppetlabs-ca_extend

Otherwise, if your primary Puppet server or workstation operates behind a proxy, initialize the project without the --modules option:

mkdir ca_extend
cd ca_extend

bolt project init expiry

Then edit your bolt-project.yaml to use the proxy according to the documentation. Next, add the module dependencies to bolt-project.yaml:

---
name: expiry
modules:
  - name: puppetlabs-stdlib
  - name: puppetlabs-ca_extend

Finally, install the modules.

bolt module install

See the "Usage" section for how to run the tasks and plans remotely or locally on the primary Puppet server.

Dependencies

• A Puppet Bolt >= 1.38.0
• puppetlabs-stdlib
• A base64 binary on the primary Puppet server which supports the -w flag
• bash >= 4.0 on the primary Puppet server

Usage

Extend the CA using the ca_extend::extend_ca_cert plan

First, check the expiration of the Puppet agent certificate by running the following command as root on the primary Puppet server:

/opt/puppetlabs/puppet/bin/openssl x509 -in "$(/opt/puppetlabs/bin/puppet config print hostcert)" -enddate -noout

If, and only if, the notAfter date printed has already passed, then the primary Puppet server certificate has expired and must be cleaned up before the CA can be regenerated. This can be accomplished by passing regen_primary_cert=true to the ca_extend::extend_ca_cert plan.

Note: This plan will also run the ca_extend::check_crl_cert task and if the crl is expired, will automatically resolve the issue by running the ca_extend::crl_truncate task.

bolt plan run ca_extend::extend_ca_cert regen_primary_cert=true --targets <primary_fqdn> replica=<replica_fqdn> compilers=<comma_separated_compiler_fqdns> --run-as root

Note that if you are running extend_ca_cert locally on the primary Puppet server, you can avoid potential Bolt transport issues by specifying --targets local://hostname, e.g.

bolt plan run ca_extend::extend_ca_cert --targets local://hostname --run-as root

Distribute ca.pem to agents

Next, distribute ca.pem to agents using one of the three methods:

1. Using the ca_extend::upload_ca_cert Plan

Using the ca_extend::upload_ca_cert plan relies on using ssh and/or winrm transport methods. Use the cert parameter to specify the location of the updated CA cert on the primary server. For example, you may use cert=$(puppet config print localcacert). Distribute the CA certificate to agent nodes specified in the targets parameter. Bolt defaults to using ssh transport, which in turn will use ~/.ssh/config for options such as username and private-key. However, the ca_extend::upload_ca_cert plan works best with a Bolt inventory file to specify targets; this allows for simultaneous uploads to *nix and Windows agents. See the Bolt documentation for more information on configuring an inventory file and the targets parameter.

bolt plan run ca_extend::upload_ca_cert cert=<path_to_cert> --targets <TargetSpec>

As an alternative to using the targets parameter, you may specify targets for the ca_extend::upload_ca_cert plan by connecting Bolt to PuppetDB, after which the --query parameter can be used.

Example query for all agent nodes excluding puppetserver nodes because the ca_extend::extend_ca_cert plan already updates the primary's and compilers' copies of the CA certificate:

bolt plan run ca_extend::upload_ca_cert cert=<path_to_cert> --query "nodes[certname]{! certname in ['primaryfqdn', 'compiler1fqdn', 'compiler2fqdn']}"

2. Manually deleting ca.pem on agents and letting them download that file as part of the next Puppet agent run

The agent will download ca.pem only if it is absent, so it must be deleted to use this method.

For example, on an *nix agent node delete ca.pem by running:

rm $(puppet config print localcacert)

Next, run puppet so the agent will retreive ca.pem:

puppet agent -t

Note: If you are depending on agent nodes downloading ca.pem during a scheduled Puppet run rather than manually initiating a Puppet run with puppet agent -t, you may need to restart the puppet service on *nix nodes. This is because the Puppet agent daemon on *nix nodes could have previous CA content loaded into memory.

3. Using a Puppet file resource to manage ca.pem

You may add this code to the catalog received by your agent nodes; the code manages ca.pem on Windows and *nix nodes with the contents of ca.pem on the compiling server (primary server or compiler). The code will not work with a serverless approach such as puppet apply. Note: This method is only possible if the CA certificate has not yet expired because Puppet communications depend upon a valid CA certificate.

  $localcacert = $facts['os']['family'] ? {
    'windows' => 'C:\ProgramData\PuppetLabs\puppet\etc\ssl\certs\ca.pem',
    default   => '/etc/puppetlabs/puppet/ssl/certs/ca.pem'
  }
  file {$localcacert:
    ensure  => file,
    content => file($settings::localcacert),
  }

ca_extend::check_ca_expiry Task

You can use this task to check the CA cert expiry on the primary mainly but you can also use it to check that a remote *nix node's CA cert has been updated after using any means to distribute the new CA certificate.

bolt task run ca_extend::check_ca_expiry --targets <TargetSpec>

ca_extend::check_agent_expiry Task

You can use this task to categorize all PE certs in a PE environment as part of a valid or expiring section based on a customizable date in the future (default 3 months from now). This task runs against a primary server and checks all certs under /etc/puppetlabs/puppet/ssl/ca/signed as the single source of truth for the PE environment and splits the certs between a valid section or expiring section.

bolt task run ca_extend::check_agent_expiry --targets local://hostname

As such, the following output illustrates that all available certs in /etc/puppetlabs/puppet/ssl/ca/signed are valid and nothing is expiring in the next 3 months.

[root@pe-server-7a5b76-0 ca_extend]# bolt task run ca_extend::check_agent_expiry --targets local://hostname
Started on local://pe-server-7a5b76-0.us-west1-c.internal...
Finished on local://pe-server-7a5b76-0.us-west1-c.internal:
  {
    "valid": [
      {
        "console-cert.pem": "Jan 14 19:55:34 2024 GMT"
      },
      {
        "critical-boom.delivery.puppetlabs.net.pem": "Apr 21 17:57:20 2027 GMT"
      },
      {
        "irate-maple.delivery.puppetlabs.net.pem": "Apr 21 19:25:35 2027 GMT"
      }
    ],
    "expired": [

    ]
  }

Successful on 1 target: local://pe-server-7a5b76-0.us-west1-c.internal
Ran on 1 target in 1.32 sec

See REFERENCE.md for more detailed examples.

Reference

Puppet's security is based on a PKI using X.509 certificates.

This module's ca_extend::extend_ca_cert plan creates a new self-signed CA certificate using the same keypair as the prior self-signed CA. The new CA has the same:

• Keypair.
• Subject.
• Issuer.
• X509v3 Subject Key Identifier (the fingerprint of the public key).

The new CA has a different:

• Authority Key Identifier (just the serial number, since it's self-signed).
• Validity period (the point of the whole exercise).
• Signature (since we changed the serial number and validity period).

Since Puppet's services (and other services that use Puppet's PKI) validate certificates by trusting a self-signed CA and comparing its public key to the Signatures and Authority Key Identifiers of the certificates it has issued, it's possible to issue a new self-signed CA certificate based on a prior keypair without invalidating any certificates issued by the old CA. Once you've done that, it's just a matter of delivering the new CA certificate to every participant in the PKI.

How to Report an issue or contribute to the module

If you are a PE user and need support using this module or are encountering issues, our Support team would be happy to help you resolve your issue and help reproduce any bugs. Just raise a ticket on the support portal.

If you have a reproducible bug or are a community user you can raise it directly on the Github issues page of the module here. We also welcome PR contributions to improve the module. Please see further details about contributing here

---

Supporting Content

Articles

The Support Knowledge base is a searchable repository for technical information and how-to guides for all Puppet products.

This Module has the following specific Article(s) available:

1. Check and fix the expiry date for your CA certificate in Puppet Enterprise

Videos

The Support Video Playlist is a resource of content generated by the support team

---