Installing an HA VPN using OpenVPN 2.4.9 on CentOS 8.2.2004 via Ansible 2.9.10 (Part 2)

2020-07-20 · computing

In Part 1, I showed how to install an HA VPN, using the community variant of OpenVPN 2.4.9, running on the latest CentOS 8.2.2004, via Ansible 2.9.10. This setup allowed me to reroute my VPN connection simply by restarting it, despite one of my dedicated hypervisors having unexpectedly reset. Here in Part 2, I’ll show how to configure the VPN servers, create CSRs and issue certificates from a private CA, and setup clients using an OVPN config file.

Configure

OpenVPN Servers

I prefer to write out most configs deterministically from YAML or JSON data structures, rather than interpolate variables within a template. Here, we write out the config file used for each VPN server. Each VPN server in the HA cluster has a config file that is almost identical, except for the server certificates and client subnets.

-
  name: OPENVPN-SERVER config
  template:
    src: common/templates/k-v.conf
    dest: /etc/openvpn/server/server.conf
    mode: 0644
    owner: root
    group: openvpn
  vars:
    config: "{{ openvpn_config.server }}"
  notify:
    - OPENVPN-SERVER restart

Here follows the actual server config in YAML format; this gets templated into a conf file:

openvpn_config:
  server:
    ca: ca.crt
    cert: "{{ inventory_hostname_short }}.crt"
    cipher: AES-256-CBC
    client-cert-not-required:
    crl-verify: crl.pem
    dev: tun
    dh: /etc/ssl/dhparam.pem
    explicit-exit-notify: 1
    fragment: 0
    group: nobody
    ifconfig-pool-persist: ipp.txt
    keepalive: 10 120
    key: "{{ inventory_hostname_short }}.key"
    mssfix: 0
    persist-key:
    persist-tun:
    plugin: /usr/lib64/openvpn/plugins/openvpn-plugin-auth-pam.so login
    port: 1194
    proto: udp6
    status: /var/log/openvpn-status.log
    tls-auth: ta.key 0
    topology: subnet
    tun-mtu: 1400
    user: nobody
    verb: 4
    push.0: "\"route-ipv6 {{ network.ipv6.subnets.pub | ipaddr('network') }}/64\""
    push.1: "\"route {{ network.ipv4.subnets.pub | ipaddr('network') }} {{ network.ipv4.subnets.pub | ipaddr('netmask') }}\""
    server-ipv6: "{{ network_.ipv6.subnets.vpn }}"
    server: "{{ network_.ipv4.subnets.vpn | ipaddr('network') }} {{ network_.ipv4.subnets.vpn | ipaddr('netmask') }}"

There are two options when it comes to client authentication: certificates, or allowing alternative methods such as username and password authentication via PAM. Using client certificates is more secure, so if this is your intention, then omit client-cert-not-required and plugin. If, however, you’re needing to support non-technical users, username and password authentication might be much easier, so this is possible if the security is sufficient; this is, in fact, what many of the commercial VPN services use. crl-verify allows you to use a CRL to revoke client certificates; otherwise, there would be no way to revoke access from a client once granted (other than wait for the certificate to expire).

Here, we use topology: subnet and allocate a private IPv4 and IPv6 subnet for clients using server and server-ipv6. This is not the default, but it is the current recommended setting. I use IPv6 within my infrastructure, so support not only dual-stack connections to the VPN servers themselves, but also within the network.

You will almost certainly need to adjust tun-mtu, fragment, and mssfix; my servers are connected by a VLAN with a 1400 MTU, meaning almost all settings I found online wouldn’t work. Even once I found settings which worked, I frequently had dropped packets and disconnects. After much debugging and tuning by sniffing the network settings, I finally settled on settings which give me a stable connection for my setup. Optimizing OpenVPN Throughput is a very good post which goes into this and more in detail.

Finally, we push routes for using the VPN. This particular example is a split-tunnel VPN, but it can easily to customised as required, including for a full-tunnel. The digit suffixes in push.0 and push.1 get flattened into an array by my template; the actual config key is called push, which can be specified multiple times. route pushes an IPv4 route, and route-ipv6 pushes an IPv6 route. I have the subnets within my infrastructure defined centrally, so here those are selected and manipulated using the ipaddr Jinja filters. For compliant clients, once they connect, the server will push the routes, and the route table will be updated accordingly.

We add an Ansible handler to restart the server whenever the config changes:

-
  name: OPENVPN-SERVER restart
  service:
    name: openvpn-server@server
    state: restarted

OpenVPN Clients

Similar to the server config, we now write out a client config. This isn’t actually used directly by the server, but I deploy it so it’s in an easy place to download and supply to clients connecting using an OVPN file.

-
  name: OPENVPN-CLIENT config
  template:
    src: common/templates/k-v.conf
    dest: /etc/openvpn/client/client.conf
    mode: 0644
    owner: root
    group: openvpn
  vars:
    config: "{{ openvpn_config.client }}"
    post: |
      <ca>
      {{ openvpn.ssl_key.ca -}}
      </ca>
      <tls-auth>
      {{ openvpn.ta_key -}}
      </tls-auth>

Here, we use <ca> and <tls-auth> to embed the CA and static key for ease of distribution. I’ve found this particularly helpful when supplying it to non-technical users of my VPN, since with up-to-date OpenVPN clients, they don’t need to store and link these separate files.

openvpn_config:
  client:
    auth-user-pass:
    cipher: AES-256-CBC
    client:
    dev: tun
    key-direction: 1
    nobind:
    persist-key:
    persist-tun:
    remote-cert-tls: server
    remote-random:
    resolv-retry: infinite
    server-poll-timeout: 4
    tun-mtu: 1346
    verb: 3
    remote: example.com 1194

As noted for the server config, if using the higher-security client certificates only method, omit auth-user-pass.

I initially used an Ansible group to set remote to all the VPN servers in the HA cluster. For the OpenVPN clients I’ve tried, those are tried either randomly or sequentially, depending on remote-random, until a working connection is found. However, later I changed it to put multiple VPN servers within a single DNS record, which accomplishes the same thing whilst allowing me to replace VPN nodes without issuing new client OVPN config files. Again, this is particularly convenient for maintainability, especially with non-technical users.

SSL

Here, we write out the SSL certificate authority, certificate revocation list, and static key. These are generated when creating the CA, noted in a later step.

-
  name: SSL key ca write
  copy:
    content: "{{ openvpn.ssl_key.ca }}"
    dest: /etc/openvpn/server/ca.crt
    mode: 0600
    owner: root
    group: openvpn
-
  name: SSL key crl write
  copy:
    content: "{{ openvpn.ssl_key.crl }}"
    dest: /etc/openvpn/server/crl.pem
    mode: 0600
    owner: root
    group: openvpn
-
  name: TA key write
  copy:
    content: "{{ openvpn.ta_key }}"
    dest: /etc/openvpn/server/ta.key
    mode: 0600
    owner: root
    group: openvpn

Start

All that remains is to flush the handlers to ensure SELinux modules are compiled and installed, and start the OpenVPN server. Until the necessary SSL files are in place, this might fail, depending on which order you do things.

-
  meta: flush_handlers
-
  name: OPENVPN-SERVER start
  service:
    name: openvpn-server@server
    enabled: true
    state: started

Initialisation

CA and Server Certificates

To initialise the server, run openvpn-init, and generate a CSR using easyrsa-gen-req or similar.

On the system which will contain the CA keys, which should be separate, use Easy-RSA 3 to generate the CA. Note that it is rather different to using Easy-RSA 2, so pay careful attention to the version. This only needs to be done once, regardless of how many servers or clients you generate certificates for. The CA certificate is to be used for openvpn.ssl_key.ca above.

easyrsa init-pki
easyrsa build-ca

Generate a CRL, so you can revoke certificates as needed. The CRL is to be used for openvpn.ssl_key.crl above. This only needs to be done once and is shared between the servers, but you will need to update this whenever you revoke another certificate.

easyrsa gen-crl

Since we’re using hardened security, generate a static key. This only needs to be done once, and is shared between the servers. The static key is to be used for openvpn.ta_key above.

openvpn --genkey --secret ta.key

Deploy the openvpn.ssl_key.ca, openvpn.ssl_key.crl, and openvpn.ta_key files to the servers.

Import and sign the CSR for each VPN server in the HA cluster, generated by the openvpn-gen-req helper script. This is done once per server, so each server gets its own certificate. These do not need to be synchronised with the clients, since those will accept any server certificate signed by the CA. For this reason, it is critical that a private CA is used, and not a third-party one, since that would allow compromise of the VPN simply by gaining a certificate—even for another domain name!

easyrsa import-req server-name.req SERVER
easyrsa sign-req server SERVER

Transfer the certificate to each VPN server, run the openvpn-init helper script, and start the OpenVPN server services.

Easy-RSA v3 OpenVPN Howto goes into more detail about using Easy-RSA 3 to manage the CA and generate certificates.

Client Certificates

If you are using client certificates, you can use a similar process for each client. If you are only using username and password authentication, you don’t need to do this.

easyrsa gen-req CLIENT
easyrsa sign-req client CLIENT

Note that in contrast to generating server certificates, you probably don’t want to use nopass. Ideally, the client should generate their own keys, sending only the CSR to you. If non-technical people will be using the VPN, however, this might not be practical; it might make sense for you to generate the keys and CSRs yourself, and supply these to them. If you choose to do so, you can also add embed the key itself into the client OVPN config file, although this doesn’t work with all OpenVPN client versions:

<cert>
</cert>

<key>
</key>

Clients without Certificates

If you are not using client certificates, with the client auth-user-pass option and the server client-cert-not-required and plugin options above, you can use PAM to authenticate via username and password authentication. There are a number of ways to do this, but simplest is to create a normal user account on each server, setting the password. Since this is an HA VPN, this can be deployed via Ansible, perhaps with the passwords in an Ansible Vault, ensuring that the VPN servers are in sync. This doesn’t let the user change their own passwords, but that’s arguably a whole different problem.