Get HTTPS for free

Let's Encrypt (website) is an official CA that allows you to create trusted certificates for free. You are not asked to share any personal information. The only thing that you need to do is proof that you have control over the domain or the website contents. Let's Encrypt is trusted by most browsers and allows your website to be encrypted without all the warnings you will have with self-signed certificates.

The procedure is based on the ACME protocol, made into an IETF standard. More info on the protocol here:

• https://en.wikipedia.org/wiki/Automated_Certificate_Management_Environment
• https://tools.ietf.org/html/rfc8555

The Service of Let's Encrypt is subject to rate limits, this seems to be 5 requests per 7 days. That is why you should use the provided staging servers while configuring your client. The certificates are valid for a limited number of 90 days. There are several client tools available to help you to use the service. You can find them via the Let's Encrypt website. As I want to know exactly what is happening I made my own script, based on these other scripts.

The steps

In short the steps of the ACME protocol to get a new certificate:

1. You register an 'account' by creating and using a private/public key pair, this is then used to 'sign' all subsequent requests. This is not an online form or anything, it can - and is meant to be fully automated.
2. For every domain that will be part of the certificate you request a unique token that you will then receive.
3. You will use the token to create a token-file on a pre-defined location on your webserver. The token must be accessable over port 80 via the url:
   • http://YOURDOMAIN/.well-known/acme-challenge/TOKENFILENAME
4. Next you request to check that the token is there, the service will access your webserver over port 80 to verify
5. After all the domains have been verified you create a Certificate Request, based on a new private/public key pair. The private key is never communicated and needs to be kept very private!
6. If everything is fine you can now request and receive the certificate and make it available to your webserver together with the private key.

Introduction

Here is my bash script. I made it to learn to understand the proces, this version:

• supports API version 2
• is only http-01 method
• is only about 300 lines of bash code

This script is based on several of the available scripts out there, it supports only part of API version 2. But, that is also the reason it is short and easier to read and understand.

More on certificates for Apache and configuration:

• Apache2 configuration for SOGo and MediaWiki
• Create certificates for Apache2 ('self-signed and' and 'how it works')

Install and run

#this script uses the JSON query tool
# other used tools are commonly available on Linux distributions 
sudo apt install jq

#run:
sudo acme-update.sh 

#force new certificate (beware of the rate limits):
sudo acme-update.sh --force

#force new account:
sudo acme-update.sh --forceaccount

Cron example, execute every day

sudo nano /etc/crontab

     45 0 * * * root /../acme-update.sh

Apache webserver config example (partly)

sudo nano /etc/apache2/sites-available/website.conf

    Listen 80
    <VirtualHost _default_:80>
      Alias "/.well-known/acme-challenge" "/var/www/.well-known/acme-challenge"
      <Directory /var/www/.well-known/acme-challenge>
        ...

     <VirtualHost _default_:443>
      SSLEngine On
      SSLCertificateFile $CERTDIR/server.cer
      SSLCertificateKeyFile $CERTDIR/server.key
      ...

The Script

The $CERTDIR folder should only be accessible by root. The keys are only read when the webserver is starting, not while it is running.

CERTDIR="/etc/apache2/ssl"
sudo mkdir $CERTDIR
sudo chown root:root $CERTDIR
sudo chmod 700 $CERTDIR

Put the following settings, functions and the main program in this order into a file 'acme-update.sh' or any other name of your choosing.

Script: settings

#!/bin/bash

DOMAIN="example.com"
ALLDOMAINS="$DOMAIN,www.$DOMAIN"

#Use the staging server for testing, the main server has rate limits
CA="acme-staging-v02.api.letsencrypt.org"
#CA="acme-v02.api.letsencrypt.org"

CERTDIR="/etc/apache2/ssl"
WORKDIR="$CERTDIR/workdir"
TOKENDIR="/var/www/.well-known/acme-challenge"
CERT_PUB_FILE="server.cer"
CERT_KEY_FILE="server.key"
logfile="/var/log/wjv/main.log"
min_days_valid=7

Script: token and webserver functions

These functions should be modified to accomodate your own webserver.

deploy_token_to_acme_challenge_location() {
  #The token-file will need to be available to the CA at:
  #  http://$DOMAIN/.well-known/acme-challenge/${tokenValue}
  echo -n "$tokenValue.$thumbprint" > "$TOKENDIR/$tokenValue"
  error=$(( error + $? ))
  chmod 644 "$TOKENDIR/$tokenValue"
  echo "$(date '+%Y-%m-%d %H.%M.%S') (err${error}) acme: token deployed" | tee -a "$logfile"
}

remove_token() {
  rm "$TOKENDIR/$tokenValue"
  echo "$(date '+%Y-%m-%d %H.%M.%S') (err${error}) acme: token removed" | tee -a "$logfile"
}

deploy_new_certificates() {
  echo "$(date '+%Y-%m-%d %H.%M.%S') (err${error}) acme: deploy new certificates" | tee -a "$logfile"

  #backup
  cp "$CERTDIR/$CERT_KEY_FILE" "$WORKDIR/$(date '+%Y-%m-%d-%H.%M.%S')-$CERT_KEY_FILE"
  error=$(( error + $? ))
  cp "$CERTDIR/$CERT_PUB_FILE" "$WORKDIR/$(date '+%Y-%m-%d-%H.%M.%S')-$CERT_PUB_FILE"
  error=$(( error + $? ))

  #deploy
  if [[ $error -eq 0 ]] ; then
    cp "$tmp_cert_key_file" "$CERTDIR/$CERT_KEY_FILE"
    cp "$tmp_cert_pub_file" "$CERTDIR/$CERT_PUB_FILE"
    
    systemctl restart apache2
    error=$(( error + $? ))
    echo "$(date '+%Y-%m-%d %H.%M.%S') (err${error}) acme: webserver restarted"| tee -a "$logfile"
  fi
}

Script: other functions

hex2bin() { # Remove spaces, add leading zero, escape as hex string ensuring no trailing new line char
  echo -e -n "$(cat | sed -r -e 's/[[:space:]]//g' -e 's/^(.(.{2})*)$/0\1/' -e 's/(.{2})/\\x\1/g')"
}

urlbase64() { # urlbase64: base64 encoded string with '+' replaced with '-' and '/' replaced with '_'
  openssl base64 -e | tr -d '\n\r' | sed -r -e 's:=*$::g' -e 'y:+/:-_:'
}

signed_request() { # Sends a request to the ACME server, signed with your account private key.
  # $1=url, $2=data, $jwk=JSON Web Key, $kid=Key ID, $CA=CA

  # convert payload to url base 64
  payload64="$(printf '%s' "$2" | urlbase64)"
  
  # get nonce from ACME server 
  # Note: getting CR+SPACE as line-endings (unix=LF, windows=CR+LF)
  nonce=$(curl --silent -I "$URL_newNonce" | tr '\r' '\n' | grep "^Replay-Nonce:" | awk '{print $2}' | tr -d '\r\n ')

  # Build header with nonce and encode it as urlbase64  
  if [[ -z "$kid" ]]; then
    protected64="$(printf '%s' '{"alg": "RS256", "jwk": '$jwk', "nonce": "'$nonce'", "url": "'"$1"'"}' | urlbase64)"
  else
    protected64="$(printf '%s' '{"alg": "RS256", "kid": "'"$kid"'", "nonce": "'$nonce'", "url": "'"$1"'"}' | urlbase64)"
  fi
  
  # Sign header with nonce and our payload with our private key and encode signature as urlbase64
  signed64="$(printf '%s' "${protected64}.${payload64}" | openssl dgst -sha256 -sign "$WORKDIR/$CA-account.key" | urlbase64)"
 
  # Send data + signature to the acme-server
  body='{"protected": "'"$protected64"'","payload": "'"$payload64"'","signature": "'"$signed64"'"}'
 
  # Note: bash vars do not accept 0-characters, these are deleted
  response=$(curl --silent --dump-header "$WORKDIR/curl.header" -X POST --header "Content-Type: application/jose+json" --data "$body" "$1" | tr '\0' '0')
  responseHeader=$(cat "$WORKDIR/curl.header")
  code=$(awk ' $1 ~ "^HTTP" {print $2}' "$WORKDIR/curl.header" | tail -1)
 
  if jq -e . >/dev/null 2>&1 <<<"$response"; then 
    responseStatus=$( echo "$response" | jq -r '.status' )
  else
    responseStatus="non-json-response"      
  fi
}

Script: main program

# check current certificate
error=0
force=0
if [[ -f "$CERTDIR/$CERT_PUB_FILE" ]]; then
  date=$(openssl x509 -in $CERTDIR/$CERT_PUB_FILE -enddate -noout | sed "s/.*=\(.*\)/\1/")
  date_s=$(date -d "$date" +%s)
  now_s=$(date -d now +%s)
  date_diff=$(( (date_s - now_s) / 86400 ))
else
  date_diff=0
fi

echo "$(date '+%Y-%m-%d %H.%M.%S') (err${error}) acme: certificate expires after $date_diff days" | tee -a "$logfile"

mkdir -p "$WORKDIR"

# force
if [[ -f "$WORKDIR/force.new" || "$1" == "--force" || "$1" == "--forceaccount" ]] ; then
  echo "$(date '+%Y-%m-%d %H.%M.%S') (err${error}) acme: force renewal" | tee -a "$logfile"
  force=1
fi

if [[ $date_diff -gt $min_days_valid && $force -eq 0 ]] ; then
  echo "$(date '+%Y-%m-%d %H.%M.%S') (err${error}) acme: automatic renewal after $((date_diff-min_days_valid)) days" | tee -a "$logfile"
else
  echo "$(date '+%Y-%m-%d %H.%M.%S') (err${error}) acme: renewal initiated" | tee -a "$logfile"
  echo "$(date '+%Y-%m-%d %H.%M.%S') (err${error}) acme: CA=$CA" | tee -a "$logfile"

  ca_all_loc=$(curl "https://$CA/directory" 2>/dev/null)
  error=$(( error + $? ))

  URL_newAccount=$(echo "$ca_all_loc" | grep "newAccount" | awk -F'"' '{print $4}')
  URL_newNonce=$(echo "$ca_all_loc" | grep "newNonce" | awk -F'"' '{print $4}')
  URL_newOrder=$(echo "$ca_all_loc" | grep "newOrder" | awk -F'"' '{print $4}')
  
  # optionally create new account
  if [[ $error -eq 0 ]] ; then 
    if [[ ! -f "$WORKDIR/$CA-account.key" ]] || [[ "$1" == "--forceaccount" ]] ; then
    
      echo "$(date '+%Y-%m-%d %H.%M.%S') (err${error}) acme: create new account" | tee -a "$logfile"

      # backup account key (if any)
      if [[ -f "$WORKDIR/$CA-account.key" ]] ; then
        cp "$WORKDIR/$CA-account.key" "$WORKDIR/$(date '+%Y-%m-%d-%H.%M.%S')-$CA-account.key"
      fi
      
      # create new account key 
      openssl genrsa 4096 > "$WORKDIR/$CA-account.key"
    fi
  fi
  
  # public part from account key
  # jwk (JSON Web Key) only for registering account
  pub_exp64=$(openssl rsa -in "$WORKDIR/$CA-account.key" -noout -text | grep publicExponent | grep -oE "0x[a-f0-9]+" | cut -d'x' -f2 | hex2bin | urlbase64)
  pub_mod64=$(openssl rsa -in "$WORKDIR/$CA-account.key" -noout -modulus | cut -d'=' -f2 | hex2bin | urlbase64)
  jwk='{"e":"'"$pub_exp64"'","kty":"RSA","n":"'"$pub_mod64"'"}'
  kid=""
  thumbprint="$(printf "%s" "$jwk" | openssl dgst -sha256 -binary | urlbase64)"
 
  # agreement CA, register
  signed_request "$URL_newAccount" '{"termsOfServiceAgreed": true}'

  if [[ "$code" == "" ]] || [[ "$code" == '200' ]] || [[ "$code" == '201' ]] ; then
    echo "$(date '+%Y-%m-%d %H.%M.%S') (err${error}) acme: account registration accepted" | tee -a "$logfile"
  elif [[ "$code" == '409' ]] ; then
    echo "$(date '+%Y-%m-%d %H.%M.%S') (err${error}) acme: account already registered" | tee -a "$logfile"
  else
    error=$(( error + 2 ))
    echo "$(date '+%Y-%m-%d %H.%M.%S') (err${error}) acme: account ERROR: code=$code, status=$responseStatus" | tee -a "$logfile"
  fi

  if [[ $error -eq 0 ]] ; then
  
    # kid (Key ID) for all non-registering requests
    kid=$(echo "$responseHeader" | grep -i location | awk '{print $2}'| tr -d '\r\n ')

    request='{"identifiers": [{"type":"dns","value":"'$( echo "${ALLDOMAINS}" | sed "s/,/\"},{\"type\":\"dns\",\"value\":\"/g" )'"}]}'
    signed_request "$URL_newOrder" "$request"

    orderLink=$(echo "$responseHeader" | grep -i location | awk '{print $2}'| tr -d '\r\n ')
    finalizeLink=$( echo "$response" | jq -r '.finalize' )

    tmpalldomains=$( echo "$response" | jq -r '.identifiers[].value' )
    tmpallauth=$( echo "$response" | jq -r '.authorizations[]' )

    i=1
    for d in $tmpalldomains; do
      authLink=$(echo $tmpallauth | cut -d ' ' -f $i) 
      response=$(curl --silent "$authLink" 2>/dev/null)

      echo "$(date '+%Y-%m-%d %H.%M.%S') (err${error}) acme: token requested for $d" | tee -a "$logfile"
  
      # get the token from the http-01 component
      tokenValue=$( echo "$response" | jq -r '.challenges[] | select(.type=="http-01").token' )

      # get the uri from the http component
      uri=$( echo "$response" | jq -r '.challenges[] | select(.type=="http-01").url' )
      
      if [[ -z "$tokenValue" ]] || [[ -z "$code" ]] || [[ $code -gt 202 ]] ; then
        error=$(( error + 3 ))
        echo "$(date '+%Y-%m-%d %H.%M.%S') (err${error}) acme: token: code=$code, status=$responseStatus" | tee -a "$logfile"
        echo "$(date '+%Y-%m-%d %H.%M.%S') (err${error}) acme: token: ERROR: $(echo "$response" | jq -r '.detail')" | tee -a "$logfile"
        break
      fi
      
      # copy token to acme challenge location
      deploy_token_to_acme_challenge_location

      # request the challenge
      signed_request "$uri" '{"resource": "challenge", "keyAuthorization": "'"$tokenValue.$thumbprint"'"}'

      if [[ -z "$code" ]] || [[ $code -gt 202 ]] ; then
        error=$(( error + 5 ))
        echo "$(date '+%Y-%m-%d %H.%M.%S') (err${error}) acme: challenge ERROR: code=$code, status=$responseStatus" | tee -a "$logfile"
        break
      else
        echo "$(date '+%Y-%m-%d %H.%M.%S') (err${error}) acme: challenge pending" | tee -a "$logfile"
      fi
      
      # check the challenge result
      retries=5
      while [[ $retries -gt 0 ]]; do
        sleep 1;
      
        response=$(curl --silent "$uri")
        error=$(( error + $? ))
        status=$(echo "$response" | jq -r '.status')

        if [[ "$status" == "valid" ]] ; then
          echo "$(date '+%Y-%m-%d %H.%M.%S') (err${error}) acme: challenge accepted" | tee -a "$logfile"
          break;
        elif [[ "$status" == "pending" ]] ; then
          echo "$(date '+%Y-%m-%d %H.%M.%S') (err${error}) acme: challenge pending ($retries)" | tee -a "$logfile"
        else
          error=$(( error + 7 ))
          echo "$(date '+%Y-%m-%d %H.%M.%S') (err${error}) acme: challenge ERROR status=$status, $(echo "$response" | jq -r '.detail')" | tee -a "$logfile"
          break;
        fi
        ((retries--))
      done
      
      remove_token

      if [[ $error -gt 0 ]] ; then
        break
      fi

      ((i++))
    done  ## next domain d
  fi

  # create private key and certificate request (CSR)
  if [[ $error -eq 0 ]] ; then

    # create new domain key
    tmp_cert_key_file="$WORKDIR/$DOMAIN.key"
    openssl genrsa 4096 > "$tmp_cert_key_file"

    # create CSR
    CONF="$WORKDIR/$DOMAIN.csr.conf"
    cat "$(openssl version -d 2>/dev/null| cut -d\" -f2)/openssl.cnf" | grep -v "RANDFILE" > "$CONF"
    echo "[SAN]" >> "$CONF"
    echo "subjectAltName=DNS:${ALLDOMAINS//,/,DNS:}" >> "$CONF"
    
    openssl req -new -sha256 -key "$tmp_cert_key_file" -subj "/" -reqexts SAN -config "$CONF" > "$WORKDIR/$DOMAIN.csr"
    error=$(( error + $? ))
  fi

  if [[ $error -eq 0 ]] ; then   
    # obtain certificate info, note: the ca public certificate is included in this file
    tmp_cert_pub_file="$WORKDIR/$DOMAIN.cer"
    der=$(openssl req -in "$WORKDIR/$DOMAIN.csr" -outform DER | urlbase64)
    signed_request "$finalizeLink" '{"csr": "'$der'"}'
    certDataLink=$(curl --silent "$orderLink" | jq -r '.certificate')
    curl --silent "$certDataLink" > "$tmp_cert_pub_file"
    
    if [[ ! "$code" == '200' ]] ; then
      error=$(( error + 8 ))
      echo "$(date '+%Y-%m-%d %H.%M.%S') (err${error}) acme: obtain certificate: ERROR: code=$code, status=$responseStatus" | tee -a "$logfile"
    else
      echo "$(date '+%Y-%m-%d %H.%M.%S') (err${error}) acme: certificate received" | tee -a "$logfile"

      deploy_new_certificates
    fi
  fi
fi

echo "$(date '+%Y-%m-%d %H.%M.%S') acme: done."| tee -a "$logfile"