Wildcard DNS in Let’s Encrypt with Go.CD, Ansible, FreeIPA and S3

When I started working on my own home-cloud (a weird term for a small self-sustained, bare-metal paid cloud on Hetzner) I needed a way to have trusted SSL certificates. I really, really hate the warning messages of the browsers when entering an self-signed site. One of my goals was to use Let’s Encrypt, put HAproxy in front of any and all services and have HAproxy do the SSL termination (and even internally, to have all services use Let’s Encrypt signed certificates).

As part of this small architecture (based on Proxmox in a cluster configuration) it was chosen also to deploy a 5-node FreeIPA cluster to manage DNS mostly but also I took advantage of other IdM features. Another goal was to implement the wildcard DNS challenge so that I wouldn’t have to configure each and every sub-domain I required (there were a couple of TLDs and a miriad of sub-domains which I already forgot their names).

After searching and getting inspiration from a few articles around the Internet (and I’m sorry I forgot the original authors so I would quote them here) I had the courage to go and implement a small declarative YAML plugin in Go.CD that runs an Ansible playbook, triggering an “Let’s Encrypt” role which takes care generating the DNS challenge in Let’s Encrypt, populating the FreeIPA DNS resources, cleaning them up after the validation and storing the certificates in S3.

The Let’s Encrypt pipeline in Go.CD runs every 10 days to go above the rate limits imposed by the production ACME service. On the other hand, my HAproxy pipeline can suffer changes maybe on a daily basis. So as far as I was concerned, I needed a link between them that would otherwise save and maintain the “latest” batch of certificates for all my TLDs. In this case, a cheap AWS account with an S3 bucket did the trick. Of course, you’re not obliged to use S3, you have other options such as Minio or Swift, but for my personal need, this was sufficient.

So, first, let’s start with the Go.CD declarative pipeline. As you can see from below, it has a timer running every 10 days, based on the timer specification inspired from the Java-based Quartz scheduler. Secondly, the “secret” is retrieved using the newly available Secrets mechanism in Go.CD which allows to keep encrypted secrets on the master machine (which can be generated stateless as needed/at deployment time and refreshed). Afterwards, it’s just a simple call to the Ansible playbook to run the pipeline.

pipelines:
  Let.Us.Encrypt:
    timer:
      spec: "0 0 12 */10 * ?"
    group: Crons
    label_template: "${upstream[:8]}"
    materials:
      upstream:
        git: git@domain.com:/ansible-playbooks/let-us-encrypt.git
        branch: master
    environment_variables:
      FREEIPA_ADMIN_PASSWORD: "{{SECRET:[secrets][freeipa-admin-password]}}"
    stages:
      - Let.Us.Encrypt:
          clean_workspace: true
          jobs:
            Run:
              tasks:
                - exec:
                    command: ansible-galaxy
                    arguments:
                      - "install"
                      - "-r"
                      - "requirements.yml"
                      - "-p"
                      - "roles"
                - exec:
                    command: ansible-playbook
                    arguments:
                      - "-vvv"
                      - "playbook.yml"

I will skip forward and won’t detail the playbook, as it’s a simple call over one single host (the local Go.CD agent) that does the refresh of certificates. However, in the role, for each TLD I iterate with a perdomain.yml task file for each domain defined in “encrypt_domains” as so:

---
- name: Create folder to store private key of this agent
  file:
    path: /var/go/.ssh/letsencrypt
    state: directory
    mode: 0700

- name: Create Letsencrypt Account private key
  openssl_privatekey:
    path: /var/go/.ssh/letsencrypt/letsencrypt.pem

- name: Create folder to store certificates
  file:
    path: ./letsencrypt
    state: directory
    mode: 0700

- name: Run for each defined container the conditioned tasks
  include_tasks: perdomain.yml
  with_items: "{{ encrypt_domains }}"
  when: encrypt_domains is defined
  loop_control:
    loop_var: current

Now on to the juicy stuff. Since we have the folders ready, the private key generated for the signing, we can easily start making the calls to the Let’s Encrypt ACME service using the acme_certificate Ansible module and the ipa_dnsrecord module, just passing the needed variables between them:

---
- name: Create Domain Private Key
  openssl_privatekey:
    path: ./letsencrypt/{{ current.domain }}.privatekey.pem

- name: Create Certificate Signing Request
  openssl_csr:
    path: ./letsencrypt/{{ current.domain }}.csr
    privatekey_path: ./letsencrypt/{{ current.domain }}.privatekey.pem
    common_name: "{{ current.domain }}"
    subject_alt_name: "{{ item.value | map('regex_replace', '^', 'DNS:') | list }}"
  with_dict:
    dns_server:
      - "{{ current.domain }}"

- name: Create ACME Challenge
  acme_certificate:
    account_key_src: /var/go/.ssh/letsencrypt/letsencrypt.pem
    acme_directory: "{{ acme_uri }}"
    acme_version: 2
    challenge: dns-01
    csr: ./letsencrypt/{{ current.domain }}.csr
    fullchain_dest: ./letsencrypt/{{ current.domain }}.fullchain.pem
    terms_agreed: yes
    remaining_days: 60
  register: le_challenge
  retries: 5
  delay: 30
  until: le_challenge is not failed

- name: Update DNS record
  ipa_dnsrecord:
    ipa_host: "{{ ipa_secure_host }}"
    ipa_pass: "{{ ipa_secure_pass }}"
    state: present
    zone_name: "{{ current.zone }}"
    record_name: "{{ item['value']['dns-01']['resource'] }}"
    record_type: "TXT"
    record_value: "{{ item['value']['dns-01']['resource_value'] }}"
    record_ttl: 300
    validate_certs: no
  when:
    - le_challenge is changed
    - item.key == current.domain
  with_dict: "{{ le_challenge['challenge_data'] }}"

- name: Wait a bit to let DNS update
  wait_for:
    timeout: 60
  when: le_challenge is changed

- name: Validate ACME Challenge
  acme_certificate:
    account_key_src: /var/go/.ssh/letsencrypt/letsencrypt.pem
    acme_directory: "{{ acme_uri }}"
    acme_version: 2
    challenge: dns-01
    csr: ./letsencrypt/{{ current.domain }}.csr
    data: "{{ le_challenge }}"
    fullchain_dest: ./letsencrypt/{{ current.domain }}.fullchain.pem
    terms_agreed: yes
    remaining_days: 60
  when: le_challenge is changed

- name: Remove DNS record
  ipa_dnsrecord:
    ipa_host: "{{ ipa_secure_host }}"
    ipa_pass: "{{ ipa_secure_pass }}"
    state: absent
    zone_name: "{{ current.zone }}"
    record_name: "{{ item['value']['dns-01']['resource'] }}"
    record_type: "TXT"
    record_value: "{{ item['value']['dns-01']['resource_value'] }}"
    record_ttl: 300
    validate_certs: no
  when: le_challenge is changed
  with_dict: "{{ le_challenge['challenge_data'] }}"

- name: Concatenate (raw)
  raw: cat ./letsencrypt/{{ current.domain }}.fullchain.pem ./letsencrypt/{{ current.domain }}.privatekey.pem > ./letsencrypt/{{ current.domain }}.haproxy.pem

- name: Copy to s3 (chain)
  aws_s3:
    aws_access_key: "{{ s3_access_key }}"
    aws_secret_key: "{{ s3_secret_key }}"
    bucket: "{{ s3_bucket }}"
    src: "./letsencrypt/{{ current.domain }}.fullchain.pem"
    object: "{{ current.domain }}.fullchain.pem"
    mode: put
    overwrite: yes

- name: Copy to s3 (prv)
  aws_s3:
    aws_access_key: "{{ s3_access_key }}"
    aws_secret_key: "{{ s3_secret_key }}"
    bucket: "{{ s3_bucket }}"
    src: "./letsencrypt/{{ current.domain }}.privatekey.pem"
    object: "{{ current.domain }}.privatekey.pem"
    mode: put
    overwrite: yes

- name: Copy to s3 (cert))
  aws_s3:
    aws_access_key: "{{ s3_access_key }}"
    aws_secret_key: "{{ s3_secret_key }}"
    bucket: "{{ s3_bucket }}"
    src: "./letsencrypt/{{ current.domain }}.haproxy.pem"
    object: "{{ current.domain }}.haproxy.pem"
    mode: put
    overwrite: yes

Since putting this to work it’s been a couple of months since it works perfectly every 10 days. It also keeps me happy as I know for a certain that the communication leaving any container, VM either internally or over the Internet is secured by default. For the internal part, I have a public sub-domain that I map internally to hosts with private/local addresses in the sense that the main sub-domain is visible to Let’s Encrypt for validation but internally any hosts under it resolve to private IPs.

On the other side of the spectrum, the HAproxy pipeline whenever run (which can be also tied to the Let’s Encrypt pipeline as a dependency) will start to download the certificates from the S3 bucket whenever needed to a pre-defined folder on the HAproxy dedicated LXC containers I’m using whenever running the pipeline (also ensuring an idempotent execution). Hope this helps you in your endeavors to stay encrypted and protected to a certain degree.