Introduction

Consul is a key value store, and service discovery service provided by Hasicorp. Consul has gained a presence within the Cloud community and can even be used as an alternative to a load balencer. I highly watching the following video posted by one of the founders of Hashicorp - Armon Dadgar which gives you a fantastic overview of what is posisble:https://www.youtube.com/watch?v=mxeMdl0KvBIWith that said, you have probably found this post as you already know what Consul is and just want to get started.

Firstly, provision at least 3 nodes to be used for consul on your favorite cloud / virtualisation platform and get CentOS 7 deployed. You can use more than three nodes, but you need to consider quorum and the consensus protocol (https://www.consul.io/docs/internals/consensus.html)

Throughout this example, we are assuming the following architecture:

Hostname IP
HOST1 192.168.1.10
HOST2 192.168.1.11
HOST3 192.168.1.12

Once up and running, you need to ensure you have A records in your DNS provider of choice, Or edit local hosts of each of the 3 nodes to resolve the IP Address of itself, and all other nodes in the cluser.

All Nodes

Firstly, update the CentOS packages and install unzip, vim (missing from CentOS minimal) and wget:

yum -y update
yum -y install wget unzip vim

Then create a working directory to download consul, and to extract the binaries into:

mkdir consul
cd consul
wget https://releases.hashicorp.com/consul/1.4.2/consul_1.4.2_linux_amd64.zip
unzip consul_1.4.2_linux_amd64.zip

With consul now extracted, we need to copy consul to /usr/bin (sudo / root needed)

cp consul /usr/local/bin

And check all is working correctly:

consul --version

You can even able “autocomplete” for shell commands:

consul -autocomplete-install

Next we create a user for consul to run as, a home directory, disable shell access for the user and set up the consul working directories;

useradd --system --home /etc/consul.d --shell /bin/false consul
mkdir --parents /opt/consul
chown --recursive consul:consul /opt/consul

Next we need to create a systemd service definition file to manage the consul service:

sudo touch /etc/systemd/system/consul.service

Now it’s time to get to the consul configuration files. Firstly, we touch out the files and set the approproate permissions giving our consul service user access to them:

mkdir --parents /etc/consul.d
touch /etc/consul.d/consul.hcl
chown --recursive consul:consul /etc/consul.d
chmod 640 /etc/consul.d/consul.hcl

We’re heading back into VIM to edit the service configuration file.

vim /etc/consul.d/consul.hcl

And paste the following, replacing “secret” for the output from consul keygen. The double quotes are needed!

datacenter = "dc1"
data_dir = "/opt/consul"
encrypt = "secret"

Node 1

Now we will create a systemd statup configuration

vim /etc/systemd/system/consul.service
[Unit]
Description="HashiCorp Consul - A service mesh solution"
Documentation=https://www.consul.io/
Requires=network-online.target
After=network-online.target
ConditionFileNotEmpty=/etc/consul.d/consul.hcl

[Service]
User=consul
Group=consul
ExecStart=/usr/local/bin/consul agent -node=HOST1 -config-dir=/etc/consul.d/
ExecReload=/usr/local/bin/consul reload
KillMode=process
Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

Now it’s time to get to the consul configuration files. Firstly, we touch out the files and set the approproate permissions giving our consul service user access to them:

mkdir --parents  /var/lib/consul /etc/consul.d
touch /etc/consul.d/consul.hcl
chown --recursive consul:consul  /var/lib/consul /etc/consul.d
chmod 640 /etc/consul.d/consul.hcl

With that complete, we need to generate a secret key to autorise cluster members. This is achieved using the “consul” binary with the “keygen” argument. You’ll need to copy the generated output for the step that follows for use later:

consul keygen

With that done, we need to edit the main configuration file - this file defines the variables you wish to use in your cluster. It is worth reviewing the documentation on Hashicorp’s site to ensure you adjust this to your specific requirements. You will need to replace “HOST1”, “HOST2” and “HOST3” with the DNS A record for your hosts.

Also note, bootstrap_expect is set the “3” - adjust this to reflect the actual number of consul servers you are running in your environment. And finally, “client_addr” defines the CIDR notation for the subnet from which you are expecting clients to connect from. Don’t be that person that uses 0.0.0.0/0 in production!

vim /etc/consul.d/server.json

And past the following:

{
     "advertise_addr": "192.168.1.10",
     "bind_addr": "192.168.1.10",
     "bootstrap_expect": 3,
     "client_addr": "0.0.0.0",
     "datacenter": "DC1",
     "data_dir": "/var/lib/consul",
     "domain": "consul",
     "enable_script_checks": true,
     "dns_config": {
         "enable_truncate": true,
         "only_passing": true
     },
     "enable_syslog": true,
     "encrypt": "<SECRET>",
     "leave_on_terminate": true,
     "log_level": "INFO",
     "verify_incoming": false,
     "verify_incoming_rpc": true,
     "verify_outgoing": true,
     "verify_server_hostname": true,
     "ca_file": "/etc/consul.d/consul-agent-ca.pem",
     "cert_file": "/etc/consul.d/dc1-server-consul-0.pem",
     "key_file": "/etc/consul.d/dc1-server-consul-0-key.pem",
     "rejoin_after_leave": true,
     "retry_join": [
         "HOST1:8301",
         "HOST2:8301",
         "HOST3:8301"
     ],
     "server": true,
     "start_join": [
         "HOST1:8301",
         "HOST2:8301",
         "HOST3:8301"
     ],
     "ui": true,
     "ports": {
     "http": -1,
     "https": 8501
     }
}

With the service definition file copmleted, we now need to generate the ca certificate, and associated public and private keys.

Generate CA Cert

consul tls ca create

Generate Server Certs and Keys

consul tls cert create -server

Generate Client Keys

consul tls cert create -client

Generate Consul CLI certs and keys

consul tls cert create -cli

With that complete, we now need to ensure the generated keys are copied to the other servers in the cluster, ensuring that they are published to the correct directories (with permissioning in tact). SCP is outside the scope of this post, but I recommend that for ease

Node 2

vim /etc/systemd/system/consul.service
[Unit]
Description="HashiCorp Consul - A service mesh solution"
Documentation=https://www.consul.io/
Requires=network-online.target
After=network-online.target
ConditionFileNotEmpty=/etc/consul.d/consul.hcl

[Service]
User=consul
Group=consul
ExecStart=/usr/local/bin/consul agent -node=HOST2 -config-dir=/etc/consul.d/
ExecReload=/usr/local/bin/consul reload
KillMode=process
Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
mkdir --parents /etc/consul.d
touch /etc/consul.d/consul.hcl
chown --recursive consul:consul /etc/consul.d
chmod 640 /etc/consul.d/consul.hcl
vim /etc/consul.d/server.json

And past the following:

{
     "advertise_addr": "192.168.1.11",
     "bind_addr": "192.168.1.11",
     "bootstrap_expect": 3,
     "client_addr": "0.0.0.0",
     "datacenter": "DC1",
     "data_dir": "/var/lib/consul",
     "domain": "consul",
     "enable_script_checks": true,
     "dns_config": {
         "enable_truncate": true,
         "only_passing": true
     },
     "enable_syslog": true,
     "encrypt": "<SECRET>",
     "leave_on_terminate": true,
     "log_level": "INFO",
     "verify_incoming": false,
     "verify_incoming_rpc": true,
     "verify_outgoing": true,
     "verify_server_hostname": true,
     "ca_file": "/etc/consul.d/consul-agent-ca.pem",
     "cert_file": "/etc/consul.d/dc1-server-consul-0.pem",
     "key_file": "/etc/consul.d/dc1-server-consul-0-key.pem",
     "rejoin_after_leave": true,
     "retry_join": [
         "HOST1:8301",
         "HOST2:8301",
         "HOST3:8301"
     ],
     "server": true,
     "start_join": [
         "HOST1:8301",
         "HOST2:8301",
         "HOST3:8301"
     ],
     "ui": true,
     "ports": {
     "http": -1,
     "https": 8501
     }
}

Node 3

vim /etc/systemd/system/consul.service
[Unit]
Description="HashiCorp Consul - A service mesh solution"
Documentation=https://www.consul.io/
Requires=network-online.target
After=network-online.target
ConditionFileNotEmpty=/etc/consul.d/consul.hcl

[Service]
User=consul
Group=consul
ExecStart=/usr/local/bin/consul agent -node=HOST3 -config-dir=/etc/consul.d/
ExecReload=/usr/local/bin/consul reload
KillMode=process
Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
mkdir --parents /etc/consul.d
touch /etc/consul.d/consul.hcl
chown --recursive consul:consul /etc/consul.d
chmod 640 /etc/consul.d/consul.hcl
vim /etc/consul.d/server.json

And past the following:

{
     "advertise_addr": "192.168.1.12",
     "bind_addr": "192.168.1.12",
     "bootstrap_expect": 3,
     "client_addr": "0.0.0.0",
     "datacenter": "DC1",
     "data_dir": "/var/lib/consul",
     "domain": "consul",
     "enable_script_checks": true,
     "dns_config": {
         "enable_truncate": true,
         "only_passing": true
     },
     "enable_syslog": true,
     "encrypt": "<SECRET>",
     "leave_on_terminate": true,
     "log_level": "INFO",
     "verify_incoming": false,
     "verify_incoming_rpc": true,
     "verify_outgoing": true,
     "verify_server_hostname": true,
     "ca_file": "/etc/consul.d/consul-agent-ca.pem",
     "cert_file": "/etc/consul.d/dc1-server-consul-0.pem",
     "key_file": "/etc/consul.d/dc1-server-consul-0-key.pem",
     "rejoin_after_leave": true,
     "retry_join": [
         "HOST1:8301",
         "HOST2:8301",
         "HOST3:8301"
     ],
     "server": true,
     "start_join": [
         "HOST1:8301",
         "HOST2:8301",
         "HOST3:8301"
     ],
     "ui": true,
     "ports": {
     "http": -1,
     "https": 8501
     }
}

With the certs copied to the other member servers, it’s finally time to enable the services. I recommend enabling synchronise panes in tmux again for the next parts:

sudo firewall-cmd  --add-port={8300,8301,8302,8400,8500,8501,8600}/tcp --permanent
sudo firewall-cmd  --add-port={8301,8302,8600}/udp --permanent
sudo firewall-cmd --reload
sudo systemctl enable consul
sudo systemctl start consul
sudo systemctl status consul

And to ensurethat we are populating the correct environment variables for the consul cli, we’ll edit the current user’s bash profile to export those variables upon relaunching the shell environment:

vim ~/.bashrc

And paste:

export CONSUL_HTTP_ADDR=https://localhost:8501
export CONSUL_CACERT=/etc/consul.d/consul-agent-ca.pem
export CONSUL_CLIENT_CERT=/etc/consul.d/dc1-cli-consul-0.pem
export CONSUL_CLIENT_KEY=/etc/consul.d/dc1-cli-consul-0-key.pem

Assuming all went well, we can now check the status of the cluster:

consul members

and:

consul monitor

Should produce output confirming the cluster is live.

And finally, you’ll have noticed ““ui”: true,” in the server.json configuration earlier. Consul has a web ui which we have enabled. As we are only allowing local HTTPS connections (by default, consul only accepts local web connections which can be modified) we’ll need to pipe from our client machine, to one of the consul members using ssh and create a socks proxy:

ssh -N -f -L 8501:localhost:8501 root@HOST1

A quick curl should show a valid connection, but with a HTTPS error:

curl https://localhost:8501/ui/ -k -I

In order to resolve any HTTPs errors, you will need to trust the client certificate we generated earlier, which is outside the scope of this post.To use the command line CLI from your work statation / laptop, you’ll need to copy the client certificates we generated earlier from one of the servers to your local machine, and run the following (replace the cert names with the ones appropriate for your environment)

consul members -ca-file=consul-agent-ca.pem -client-cert=dc1-client-consul-0.pem -client-key=dc1-client-consul-0-key.pem -http-addr="https://localhost:8501"