Get HTTPS for free

Let's Encrypt (website) is an official CA that allows you to create trusted certificates for free. You are not asked to share any personal information. The only thing that you need to do is proof that you have control over the domain or the website contents. Let's Encrypt is trusted by most browsers and allows your website to be encrypted without all the warnings you will have with self-signed certificates.

The procedure is based on the ACME protocol, made into an IETF standard. More info on the protocol here:

• https://en.wikipedia.org/wiki/Automated_Certificate_Management_Environment
• https://tools.ietf.org/html/rfc8555

The Service of Let's Encrypt is subject to rate limits, this seems to be 5 requests per 7 days. That is why you should use the provided staging servers while configuring your client. The certificates are valid for a limited number of 90 days. There are several client tools available to help you to use the service. You can find them via the Let's Encrypt website. As I want to know exactly what is happening I made my own script, based on these other scripts.

The steps

In short the steps of the ACME protocol to get a new certificate:

1. You register an 'account' by creating and using a private/public key pair, this is then used to 'sign' all subsequent requests. This is not an online form or anything, it can - and is meant to be fully automated.
2. For every domain that will be part of the certificate you request a unique token that you will then receive.
3. You will use the token to create a token-file on a pre-defined location on your webserver. The token must be accessable over port 80 via the url:
   • http://YOURDOMAIN/.well-known/acme-challenge/TOKENFILENAME
4. Next you request to check that the token is there, the service will access your webserver over port 80 to verify
5. After all the domains have been verified you create a Certificate Request, based on a new private/public key pair. The private key is never communicated and needs to be kept very private!
6. If everything is fine you can now request and receive the certificate and make it available to your webserver together with the private key.

Introduction

Here is my bash script. This version:

• supports API version 2
• implements http-01 method only
• is only about 300 lines of bash code

This script is based on several of the available scripts out there, it supports only part of API version 2. But, that is also the reason it is short and easier to read and understand.

More on certificates for Apache and configuration:

• Apache2 configuration for SOGo and MediaWiki
• Create certificates for Apache2 ('self-signed and' and 'how it works')

Install and run

#this script uses the JSON query tool
# other used tools are commonly available on Linux distributions 
sudo apt install jq

#run:
sudo acme-update.sh 

#force new certificate (beware of the rate limits):
sudo acme-update.sh --force

#force new account:
sudo acme-update.sh --forceaccount

Cron example, execute every day

sudo nano /etc/crontab

     45 0 * * * root /../acme-update.sh

Apache webserver config example (partly)

sudo nano /etc/apache2/sites-available/website.conf

    Listen 80
    <VirtualHost _default_:80>
      Alias "/.well-known/acme-challenge" "/var/www/.well-known/acme-challenge"
      <Directory /var/www/.well-known/acme-challenge>
        ...

     <VirtualHost _default_:443>
      SSLEngine On
      SSLCertificateFile $CERTDIR/server.cer
      SSLCertificateKeyFile $CERTDIR/server.key
      ...

The Script

The $CERTDIR folder should only be accessible by root. The keys are only read when the webserver is starting, not while it is running. This is why the webserver does not need to run as root.

CERTDIR="/etc/apache2/ssl"
sudo mkdir $CERTDIR
sudo chown root:root $CERTDIR
sudo chmod 700 $CERTDIR

Put the following settings, functions and the main program in this order into a file 'acme-update.sh' or any other name of your choosing.

Script: settings

#!/bin/bash

DOMAIN="example.com"
ALLDOMAINS="$DOMAIN,www.$DOMAIN"

#Use the staging server for testing, the main server has rate limits
CA="acme-staging-v02.api.letsencrypt.org"
#CA="acme-v02.api.letsencrypt.org"

CERTDIR="/etc/apache2/ssl"
WORKDIR="$CERTDIR/workdir"
TOKENDIR="/var/www/.well-known/acme-challenge"
CERT_PUB_FILE="server.cer"
CERT_KEY_FILE="server.key"
logfile="/var/log/wjv/main.log"
min_days_valid=80

Script: token and webserver functions

These functions should be modified to accommodate your own webserver.

deploy_token_to_acme_challenge_location() {
  #The token-file will need to be available to the CA at:
  #  http://$DOMAIN/.well-known/acme-challenge/${tokenValue}
  echo -n "$tokenValue.$thumbprint" > "$TOKENDIR/$tokenValue"
  error=$(( error + $? ))
  chmod 644 "$TOKENDIR/$tokenValue"
  echo "$(date '+%Y-%m-%d %H.%M.%S') (err${error}) acme: token deployed" | tee -a "$logfile"
}

remove_token() {
  rm "$TOKENDIR/$tokenValue"
  echo "$(date '+%Y-%m-%d %H.%M.%S') (err${error}) acme: token removed" | tee -a "$logfile"
}

deploy_new_certificates() {
  echo "$(date '+%Y-%m-%d %H.%M.%S') (err${error}) acme: deploy new certificates" | tee -a "$logfile"

  #backup
  cp "$CERTDIR/$CERT_KEY_FILE" "$WORKDIR/$(date '+%Y-%m-%d-%H.%M.%S')-$CERT_KEY_FILE"
  error=$(( error + $? ))
  cp "$CERTDIR/$CERT_PUB_FILE" "$WORKDIR/$(date '+%Y-%m-%d-%H.%M.%S')-$CERT_PUB_FILE"
  error=$(( error + $? ))

  #deploy
  if [[ $error -eq 0 ]] ; then
    cp "$tmp_cert_key_file" "$CERTDIR/$CERT_KEY_FILE"
    cp "$tmp_cert_pub_file" "$CERTDIR/$CERT_PUB_FILE"
    
    systemctl restart apache2
    error=$(( error + $? ))
    echo "$(date '+%Y-%m-%d %H.%M.%S') (err${error}) acme: webserver restarted"| tee -a "$logfile"
  fi
}

Script: other functions

hex2bin() { # Remove spaces, add leading zero, escape as hex string ensuring no trailing new line char
  echo -e -n "$(cat | sed -r -e 's/[[:space:]]//g' -e 's/^(.(.{2})*)$/0\1/' -e 's/(.{2})/\\x\1/g')"
}

urlbase64() { # urlbase64: base64 encoded string with '+' replaced with '-' and '/' replaced with '_'
  openssl base64 -e | tr -d '\n\r' | sed -r -e 's:=*$::g' -e 'y:+/:-_:'
}

signed_request() { # Sends a request to the ACME server, signed with your accountkey.
  # $1=url, $2=data, $jwk=JSON Web Key, $kid=Key ID, $CA=CA

  # convert payload to url base 64
  payload64="$(printf '%s' "$2" | urlbase64)"
  
  # get nonce from ACME server 
  # Note: getting CR+SPACE as line-endings (unix=LF, windows=CR+LF)
  # Note: header uses lowercase keywords
  newNonceHeader=$(curl --silent -I "$URL_newNonce")
  nonce=$(echo "$newNonceHeader" | tr '\r' '\n' | grep -i "^Replay-Nonce:" | awk '{print $2}' | tr -d '\r\n ')
  
  # Build header with nonce and encode it as urlbase64  
  if [[ -z "$kid" ]]; then
    #jwk: only account login and creation
    protected64="$(printf '%s' '{"alg": "RS256", "jwk": '$jwk', "nonce": "'$nonce'", "url": "'"$1"'"}' | urlbase64)"
  else
    #kid: other
    protected64="$(printf '%s' '{"alg": "RS256", "kid": "'"$kid"'", "nonce": "'$nonce'", "url": "'"$1"'"}' | urlbase64)"
  fi
  
  # Sign header with nonce and our payload with our private key and encode signature as urlbase64
  signed64="$(printf '%s' "${protected64}.${payload64}" | openssl dgst -sha256 -sign "$WORKDIR/$CA-account.key" | urlbase64)"
  
  # Send data + signature to the acme-server
  body='{"protected": "'"$protected64"'","payload": "'"$payload64"'","signature": "'"$signed64"'"}'
  
  # Note: bash vars do not accept binary 0-characters, these are deleted to prevent errors, the remaining content should be ignored
  response=$(curl --silent --dump-header "$WORKDIR/curl.header" -X POST --header "Content-Type: application/jose+json" --data "$body" "$1" | tr '\0' '0')
  responseHeader=$(cat "$WORKDIR/curl.header")

  code=$(awk ' $1 ~ "^HTTP" {print $2}' "$WORKDIR/curl.header" | tail -1)
 
  if jq -e . >/dev/null 2>&1 <<<"$response"; then 
    responseStatus=$( echo "$response" | jq -r '.status' )
  else
    responseStatus="non-json-response"      
  fi
}

log() {
  # $1=message
  echo "$(date '+%Y-%m-%d %H.%M.%S') (err${error}) acme: $1" | tee -a "$logfile"
  # report failure
  if [ $error -ge 1 ] ; then 
    echo "$(date '+%d %b %Y, %H:%M') acme: failed!" >> "$failfile"
  fi
}

Script: main program

# check current cert
error=0
force=0
if [[ -f "$CERTDIR/$CERT_PUB_FILE" ]]; then
  date=$(openssl x509 -in $CERTDIR/$CERT_PUB_FILE -enddate -noout | sed "s/.*=\(.*\)/\1/")
  date_s=$(date -d "$date" +%s)
  now_s=$(date -d now +%s)
  date_diff=$(( (date_s - now_s) / 86400 ))
else
  date_diff=0
fi

log "certificate expires after $date_diff days"

mkdir -p "$WORKDIR"

# force
if [[ -f "$WORKDIR/force.new" || "$1" == "--force" || "$1" == "--forceaccount" ]] ; then
  log "force renewal"
  echo "" > "$WORKDIR/force.new"
  force=1
fi

if [[ $date_diff -gt $min_days_valid && $force -eq 0 ]] ; then
  log "automatic renewal after $((date_diff-min_days_valid)) days"
else
  log "automatic renewal, CA=$CA"
  
  ca_all_loc=$(curl "https://$CA/directory" 2>/dev/null)
  error=$(( error + $? ))

  URL_newAccount=$(echo "$ca_all_loc" | grep "newAccount" | awk -F'"' '{print $4}')
  URL_newNonce=$(echo "$ca_all_loc" | grep "newNonce" | awk -F'"' '{print $4}')
  URL_newOrder=$(echo "$ca_all_loc" | grep "newOrder" | awk -F'"' '{print $4}')

  # optionally create new account
  if [[ $error -eq 0 ]] ; then 
    if [[ ! -f "$WORKDIR/$CA-account.key" ]] || [[ "$1" == "--forceaccount" ]] ; then
    
      log "create new account"

      # backup account key (if any)
      if [[ -f "$WORKDIR/$CA-account.key" ]] ; then
        cp "$WORKDIR/$CA-account.key" "$WORKDIR/$(date '+%Y-%m-%d-%H.%M.%S')-$CA-account.key"
      fi
      
      # create new account key 
      openssl genrsa 4096 > "$WORKDIR/$CA-account.key"
    fi
  fi
  
  # public part from account key
  # jwk (JSON Web Key) for registering account only
  pub_exp64=$(openssl rsa -in "$WORKDIR/$CA-account.key" -noout -text | grep publicExponent | grep -oE "0x[a-f0-9]+" | cut -d'x' -f2 | hex2bin | urlbase64)
  pub_mod64=$(openssl rsa -in "$WORKDIR/$CA-account.key" -noout -modulus | cut -d'=' -f2 | hex2bin | urlbase64)
  jwk='{"e":"'"$pub_exp64"'","kty":"RSA","n":"'"$pub_mod64"'"}'
  kid=""
  thumbprint="$(printf "%s" "$jwk" | openssl dgst -sha256 -binary | urlbase64)"
 
  # agreement CA / register
  signed_request "$URL_newAccount" '{"termsOfServiceAgreed": true}'

  if [[ "$code" == "" ]] || [[ "$code" == '200' ]] || [[ "$code" == '201' ]] ; then
    log "account registration accepted"
  elif [[ "$code" == '409' ]] ; then
    log "account already registered"
  else
    error=$(( error + 2 ))
    log "account ERROR: code=$code, status=$responseStatus"
  fi

  if [[ $error -eq 0 ]] ; then
  
    # kid (Key ID) for all non-registering requests
    kid=$(echo "$responseHeader" | grep -i location | awk '{print $2}'| tr -d '\r\n ')

    request='{"identifiers": [{"type":"dns","value":"'$( echo "${ALLDOMAINS}" | sed "s/,/\"},{\"type\":\"dns\",\"value\":\"/g" )'"}]}'
    signed_request "$URL_newOrder" "$request"

    orderLink=$(echo "$responseHeader" | grep -i location | awk '{print $2}'| tr -d '\r\n ')
    finalizeLink=$( echo "$response" | jq -r '.finalize' )

    tmpalldomains=$( echo "$response" | jq -r '.identifiers[].value' )
    tmpallauth=$( echo "$response" | jq -r '.authorizations[]' )

    i=1
    for d in $tmpalldomains; do
      authLink=$(echo $tmpallauth | cut -d ' ' -f $i) 
      signed_request "$authLink" ""

      log "token $i requested for $d"
  
      # get the token from the http-01 component
      tokenValue=$( echo "$response" | jq -r '.challenges[] | select(.type=="http-01").token' )

      # get the uri from the http component
      uri=$( echo "$response" | jq -r '.challenges[] | select(.type=="http-01").url' )
      
      if [[ -z "$tokenValue" ]] || [[ -z "$code" ]] || [[ $code -gt 202 ]] ; then
        error=$(( error + 3 ))
        log "token ERROR: code=$code, status=$responseStatus, $(echo "$response" | jq -r '.detail')"
        break
      fi
      
      # copy token to acme challenge location
      deploy_token_to_acme_challenge_location

      # request the challenge
      signed_request "$uri" '{"resource": "challenge", "keyAuthorization": "'"$tokenValue.$thumbprint"'"}'

      if [[ -z "$code" ]] || [[ $code -gt 202 ]] ; then
        error=$(( error + 5 ))
        log "challenge ERROR: code=$code, status=$responseStatus"
        break
      else
        log "challenge pending"
      fi
      
      # check the challenge result
      retries=5
      while [[ $retries -gt 0 ]]; do
        sleep 2;
      
        signed_request "$uri" ""
        error=$(( error + $? ))
        status=$(echo "$response" | jq -r '.status')

        if [[ "$status" == "valid" ]] ; then
          log "challenge accepted"
          break;
        elif [[ "$status" == "pending" ]] ; then
          log "challenge pending ($retries)"
        else
          error=$(( error + 7 ))
          log "challenge ERROR: status=$status, $(echo "$response" | jq -r '.detail')"
          break;
        fi
    
        ((retries--))
      done
      
      remove_token

      if [[ $error -gt 0 ]] ; then
        break
      fi

      ((i++))
    done  ## next domain d
  fi

  # create private key and CSR - certificate request
  if [[ $error -eq 0 ]] ; then

    # create new domain key
    tmp_cert_key_file="$WORKDIR/$DOMAIN.key"
    openssl genrsa 4096 > "$tmp_cert_key_file"

    # create CSR - certificate request
    CONF="$WORKDIR/$DOMAIN.csr.conf"
    cat "$(openssl version -d 2>/dev/null| cut -d\" -f2)/openssl.cnf" | grep -v "RANDFILE" > "$CONF"
    echo "[SAN]" >> "$CONF"
    echo "subjectAltName=DNS:${ALLDOMAINS//,/,DNS:}" >> "$CONF"
    
    openssl req -new -sha256 -key "$tmp_cert_key_file" -subj "/" -reqexts SAN -config "$CONF" > "$WORKDIR/$DOMAIN.csr"
    error=$(( error + $? ))
  fi

  if [[ $error -eq 0 ]] ; then   
    # obtain certificate info
    # Note: ca public certificate is included
    tmp_cert_pub_file="$WORKDIR/$DOMAIN.cer"
    der=$(openssl req -in "$WORKDIR/$DOMAIN.csr" -outform DER | urlbase64)
    signed_request "$finalizeLink" '{"csr": "'$der'"}'
    certDataLink=$(curl --silent "$orderLink" | jq -r '.certificate')
    curl --silent "$certDataLink" > "$tmp_cert_pub_file"
    
    if [[ ! "$code" == '200' ]] ; then
      error=$(( error + 8 ))
      log "obtain certificate ERROR: code=$code, status=$responseStatus"
    else
      log "certificate received"

      deploy_new_certificates
    fi
  fi
fi

if [[ -f "$WORKDIR/force.new" ]] ; then
  if [[ $error -eq 0 ]] ; then
    rm "$WORKDIR/force.new"
  else
    log "failure, renewal will be forced with the next run"
  fi
fi

log "done."