Migrating from Certbot to acme.sh on an Old Zimbra Server

Published 8 February 2020

Historical article: This article documents a solution used on an old CentOS 6.6 Zimbra server. CentOS 6, certbot-auto, certificate chains and ACME clients have changed substantially since this was written. Do not use these instructions as a current installation guide without checking the present documentation.

What to do when a Zimbra server can no longer renew its Let's Encrypt certificate because the old certificate chain has expired and Certbot no longer supports the operating system.

The symptoms

The Zimbra server certificate would no longer update from Let's Encrypt. There turned out to be two separate problems:

The old IdenTrust DST Root CA X3 certificate had expired.
certbot-auto no longer supported the old operating system.

The server in this case was running CentOS 6.6. It was an old, well-isolated server that was still performing a specific task, so replacing the complete system was not immediately practical.

The expired IdenTrust root certificate

In the old Zimbra configuration, the root certificate had been appended to the Let's Encrypt chain:

cat /etc/letsencrypt/live/[ServerName]/chain.pem identrust.pem > zimbrachain.pem

Here identrust.pem was the previously downloaded DST Root CA X3 certificate. The resulting zimbrachain.pem was then passed to zmcertmgr for verification and deployment.

Checking the old root certificate showed the problem:

# openssl verify identrust.pem
identrust.pem: O = Digital Signature Trust Co., CN = DST Root CA X3
error 10 at 0 depth lookup: certificate has expired
OK

At the time, Let's Encrypt was moving to its own ISRG Root X1 certificate. Simply replacing the old IdenTrust certificate was not sufficient to make the existing Certbot installation work, however, because there was another problem waiting.

Certbot no longer supported the operating system

Running Certbot manually produced:

/usr/local/bin/certbot-auto renew

WARNING: couldn't find Python 3.5+ to check for updates.
Bootstrapping dependencies for Legacy RedHat-based OSes that will use Python3...
yum is hashed (/usr/bin/yum)

To use Certbot on this operating system, packages from the SCL
repository need to be installed.

Enable the SCL repository and try running Certbot again.

The installed versions were:

# python3 --version
Python 3.4.8

# cat /etc/system-release
CentOS release 6.6 (Final)

Since Certbot wanted Python 3.5 or later, I first attempted to update the old CentOS installation.

# yum install centos-release-scl

YumRepo Error: All mirror URLs are not using ftp, http[s] or file.
Eg. Invalid release/repo/arch combination/
removing mirrorlist with no valid mirrors:
/var/cache/yum/x86_64/6/base/mirrorlist.txt

Error: Cannot find a valid baseurl for repo: base

CentOS 6 had reached end of life, so its normal repositories were no longer available in their previous locations. I replaced the repository configuration with an EOL repository configuration and updated the system.

# cp centos6-eol.repo /etc/yum.repos.d/CentOS-Base.repo
# yum update

Do not copy this procedure onto a current system. CentOS 6 has been unsupported for many years. These repository changes document what was done to this particular legacy server.

After that:

# yum install centos-release-scl

The SCL repositories also needed to be redirected to the CentOS vault. The relevant repository entries were changed to:

[centos-sclo-rh]
baseurl=http://vault.centos.org/centos/6.10/sclo/$basearch/rh/

[centos-sclo-sclo]
baseurl=http://vault.centos.org/centos/6.10/sclo/$basearch/sclo/

Python 3.6 could then be installed:

# yum install rh-python36

# scl enable rh-python36 bash
# python3 --version
Python 3.6.12

Unfortunately, after all that work Certbot finally revealed the actual showstopper:

# /usr/local/bin/certbot-auto renew

Bootstrapping dependencies for Legacy RedHat-based OSes that will use Python3...
yum is /usr/bin/yum
...
Upgrading certbot-auto 1.3.0 to 1.20.0...
Replacing certbot-auto...

Your system is not supported by certbot-auto anymore.
Certbot cannot be installed.
Please visit https://certbot.eff.org/ to check for other alternatives.

So upgrading Python did not solve the problem. I also tried downgrading Certbot, without success.

Installing acme.sh instead

I therefore moved certificate handling to acme.sh. Unlike the old certbot-auto installation, it did not require the same Python bootstrap process.

The original installation was:

# sudo -s
# cd ~

# curl https://get.acme.sh | sh -s email="root@my.email.domain"

# cd ~/.acme.sh

# ./acme.sh --set-default-ca --server letsencrypt
# ./acme.sh --set-default-chain --preferred-chain ISRG --server letsencrypt

# yum install socat

socat was required for the standalone ACME mode used in this installation.

The installation command above is preserved from the original 2020 procedure. For a new installation, use the current acme.sh installation instructions rather than assuming this historical command is still the preferred method.

Deploying the certificate to Zimbra

Obtaining a certificate was only the first part of the problem. Zimbra also needed the certificate copied, verified and deployed using zmcertmgr.

I therefore wrote an acme.sh deployment hook. It was stored as:

~/.acme.sh/deploy/zimbra.sh

The following is the original deployment script, with the actual server name replaced by my.server.domain.

#!/bin/bash

#
# acme.sh, run as root, standalone.
# Zimbra/Jetty must release port 80 for standalone validation.
#

# Installation used at the time:
#
# yum install socat
#
# wget -O /root/.acme.sh/isrgrootx1.pem \
#     https://letsencrypt.org/certs/isrgrootx1.pem.txt
#
# curl https://get.acme.sh | sh -s email="root@my.server.domain"
#
# cd ~/.acme.sh
#
# ./acme.sh --set-default-ca --server letsencrypt
#
# ./acme.sh --set-default-chain \
#     --preferred-chain ISRG \
#     --server letsencrypt
#
#
# Manual certificate issue:
#
# ./acme.sh --issue \
#     --pre-hook /root/.acme.sh/zimbra-stop.sh \
#     --standalone \
#     -d my.server.domain \
#     --force
#
#
# Deploy:
#
# ./acme.sh --deploy \
#     --deploy-hook zimbra \
#     -d my.server.domain
#


function mail_ok ()
{
    /opt/zimbra/postfix/sbin/sendmail \
        -froot@my.server.domain \
        -F'root@my.server.domain' -t <<EOF

To: root@my.server.domain
Subject: Zimbra mail server certificate update $(date +%B)

Zimbra mail server certificate has been renewed.

$(cat lasterr.txt)

EOF
}


function mail_err ()
{
    /opt/zimbra/postfix/sbin/sendmail \
        -froot@my.server.domain \
        -F'root@my.server.domain' -t <<EOF

To: root@my.server.domain
Subject: Zimbra mail server certificate update $(date +%B) failed

Zimbra mail server certificate update error:
$*

$(cat lasterr.txt)

EOF
}


# Arguments supplied by acme.sh:
#
# domain keyfile certfile cafile fullchain

zimbra_deploy()
{
    _cdomain="$1"
    _ckey="$2"
    _ccert="$3"
    _cca="$4"
    _cfullchain="$5"

    # ISRG Root X1 was downloaded separately.
    ISG_X1="$(dirname "$_cca")/../isrgrootx1.pem"

    #
    # Zimbra Jetty is stopped before entering this function,
    # because acme.sh standalone mode requires port 80.
    #

    # Append root certificate so zmcertmgr can verify the chain.
    cat "$_cfullchain" "$ISG_X1" > "${_cca}.real"

    chown zimbra:zimbra \
        "$_ckey" \
        "$_ccert" \
        "${_cca}.real"

    _debug "Verifying..."

    /opt/zimbra/bin/zmcertmgr \
        verifycrt comm \
        "$_ckey" \
        "$_ccert" \
        "${_cca}.real" > lasterr.txt || {

        echo "Verification of the issued certificate failed."

        su - zimbra -c "zmcontrol restart" >> lasterr.txt

        mail_err \
            "/opt/zimbra/bin/zmcertmgr verifycrt comm \
$_ckey $_ccert ${_cca}.real failed."

        return 1
    }


    # ----- Backup

    cp -a \
        /opt/zimbra/ssl/zimbra \
        /opt/zimbra/ssl/zimbra.$(date "+%Y%m%d")


    # ----- Copy private key

    cp -f \
        "$_ckey" \
        /opt/zimbra/ssl/zimbra/commercial/commercial.key


    # ----- Deploy

    rm -f lasterr.txt

    _debug "Deploy..."

    /opt/zimbra/bin/zmcertmgr \
        deploycrt comm \
        "$_ccert" \
        "${_cca}.real" > lasterr.txt || {

        echo "Installation of the issued certificate failed."

        su - zimbra -c "zmcontrol restart" >> lasterr.txt

        mail_err \
            "/opt/zimbra/bin/zmcertmgr deploycrt comm \
$_ckey $_ccert ${_cca}.real failed."

        return 1
    }


    # ----- Restart the server

    _debug "Restarting Zimbra..."

    /usr/bin/openssl x509 \
        -enddate \
        -noout \
        -in "$_ccert" >> lasterr.txt

    echo "" >> lasterr.txt

    time su - zimbra -c "zmcontrol restart" >> lasterr.txt

    sleep 5

    su - zimbra -c "zmcontrol status" >> lasterr.txt

    mail_ok

    _debug "Done."

    return 0
}

Stopping Jetty before validation

Standalone ACME validation needed port 80. Zimbra's Jetty process was using that port, so I added a pre-hook named ~/.acme.sh/zimbra-stop.sh:

#!/bin/bash

# Stop Zimbra mailboxd/Jetty so acme.sh can use port 80.

su - zimbra -c "zmmailboxdctl stop"

Only Jetty was stopped rather than the complete Zimbra system. LDAP needed to remain available when the certificate was subsequently deployed.

ISRG Root X1

In this historical setup, ISRG Root X1 was downloaded separately:

# wget -O /root/.acme.sh/isrgrootx1.pem \
    https://letsencrypt.org/certs/isrgrootx1.pem.txt

The deployment hook appended this certificate to the chain supplied to Zimbra's zmcertmgr.

Testing certificate issuance

acme.sh stored the pre-hook when a certificate was actually issued. I therefore used --force for the initial test:

# ./acme.sh --issue \
    --pre-hook /root/.acme.sh/zimbra-stop.sh \
    --standalone \
    -d my.server.domain \
    --force

For testing certificate retrieval without the pre-hook, it could be omitted, but Zimbra/Jetty still had to be stopped manually so that acme.sh could listen on port 80.

Testing deployment

# ./acme.sh --deploy \
    --deploy-hook zimbra \
    -d my.server.domain

LDAP needed to be running during deployment. Otherwise zmcertmgr could report:

** Saving server config key zimbraSSLCertificate...failed.
** Saving server config key zimbraSSLPrivateKey...failed.

This was the reason that the pre-hook stopped only Jetty rather than shutting down the complete Zimbra installation.

Automating renewal

acme.sh initially installed a cron entry for the root user:

# crontab -l

21 0 * * * "/root/.acme.sh"/acme.sh --cron \
    --home "/root/.acme.sh" > /dev/null

I preferred keeping system jobs in /etc/cron.d, so I created a dedicated entry:

#
# Regular cron job for Zimbra mail-server certificate renewal
#

0 4 1,7,14,21 * * root \
    [ -x "/root/.acme.sh"/acme.sh ] && \
    "/root/.acme.sh"/acme.sh \
    --cron \
    --home "/root/.acme.sh" > /dev/null

I then removed the acme.sh-generated root crontab:

# crontab -r

crontab -r removes the entire crontab for that user. Always check crontab -l first and do not use this command if unrelated jobs are present.

Result

The final setup allowed the legacy Zimbra server to obtain Let's Encrypt certificates without relying on certbot-auto. acme.sh handled ACME renewal, while custom pre- and deployment hooks integrated the process with Zimbra.

This solved a specific legacy-system problem at the time. Today, replacing or upgrading an unsupported operating system should normally be preferred over extending its lifetime in this way.