Reading time ~29 minutes
Offensive Infrastructure:
the HashiStack
This post is Part 2 of the “Offensive Infrastructure with Modern Technologies” series, and is going to focus on an automated deployment of the HashiCorp stack (i.e., the HashiStack).
Part 1 explained how to
configure Consul in both single and multi node deployments using docker-compose
,
while here I’m going to provide a step-by-step walkthrough that will allow you to automatically deploy the full stack with Ansible.
High Level Design
Before jumping to the core of this post I want to quickly introduce the final setup, and, in particular, both the physical and logical deployments of the different components of the HashiStack.
Multi Node (Physical) Deployment
For this post we are going to use two nodes: one playing the role of the “master” (node-1
) which will run the the server counterparts of the different components, and the other (node-2
) playing the role of the “client” on which we will run our applications.

We are going to use Vagrant to quickly spin up these nodes. But more on this below.
Logical Deployment of the HashiStack
You have probably noticed I’ve been mentioning the term “HashiStack”. But what does actually compose this stack? Well, for this setup we are going to deploy the following components:
Component | Use Case |
---|---|
![]() |
Service Discovery. |
![]() |
Secrets Management. |
![]() |
Applications Management. |
![]() |
Applications Containerization. |
![]() |
Internal Hostnames Resolution. |
![]() |
Reverse Proxy. |
[If you need a refresher of all the different components, please refer to Part 1].
We will also need to ensure that these components are installed in the correct order, as they have interdependencies between each other. I have to thanks the team at Mockingbird Consulting for tackling this problem and producing a graph (here modified by myself to adapt it to our needs) showing an ordered deployment:

We are starting by provisioning the two hosts, followed by docker
and consul
, so to provide the service discovery on top of which all other components are going to interact with.
Once Consul is installed, we are going to deploy dnsmasq
and vault
. With Vault setup properly, we will add nomad
.
Finally, we are going to add traefik
to simplify access to the applications.
As mentioned previously, we are going to use node-1
as a server and node-2
as a client,
so this is how we are going to distribute the different components:

Environment Setup
Before jumping to the core of this post, we will have to configure a couple of requirements, namely Vagrant and Ansible.
Code Structure
To provide some reference (in case you’ll get lost in the walkthrough), the one below is the tree of the final folder structure (yes, we will reconstruct it file by file):
host:~/hashistack ❯ tree .
.
├── ansible
│ ├── ansible.cfg
│ ├── Dockerfile
│ ├── requirements.yml
│ └── roles
│ ├── brianshumate.consul
│ ├── brianshumate.nomad
│ ├── brianshumate.vault
│ ├── geerlingguy.docker
│ └── kibatic.traefik
├── playbooks
│ ├── apps
│ │ ├── redis.nomad
│ │ ├── simpleserver.nomad
│ │ └── traefik
│ │ └── traefik.toml
│ ├── apps.yml
│ ├── hashistack.yml
│ ├── inventory
│ │ ├── group_vars
│ │ │ └── docker_instances
│ │ └── hosts
│ └── restart.yml
└── Vagrantfile
Vagrant Setup
We are using Vagrant in order to have an easy-to-use development environment. This will also allow you to follow along, in case you want to replicate this setup locally on your laptop.
Let’s analyze the Vagrantfile
:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
host:~/hashistack ❯ cat Vagrantfile
# -*- mode: ruby -*-
# vi: set ft=ruby :
Vagrant.configure("2") do |config|
(1..2).each do |i|
config.vm.define "node-#{i}" do |node|
node.vm.box = "ubuntu/trusty64"
node.vm.hostname = "node-#{i}"
# Bridged network
node.vm.network "public_network", bridge: "en0: Wi-Fi (AirPort)", ip:"192.168.0.11#{i}"
# Provider-specific configuration
node.vm.provider "virtualbox" do |vb|
# Customize the amount of memory on the VM
vb.memory = "2048"
# Specify machine name
vb.name = "hashistack_node-#{i}"
end
end
end
end
- Line
6
: we are using a loop to create 2 nodes (node-1
andnode-2
) with similar characteristics. - Line
8
: both are based on Ubuntu Server 14.04 (ubuntu/trusty64
). I concur that this is not super recent, but Vagrant doesn’t play nice with most recent versions of Ubuntu Server. For local development Trusty will do, but in the following post we will switch to the most recent Ubuntu LTS. - Line
9
: as hostname, we are going to use the machine name (node-1
,node-2
). - Line
11
: we are assigning a bridged network and static IP address (192.168.0.111
and192.168.0.112
, respectively). If you want to use this Vagrantfile, remember to change these IP addresses accordingly with your own local network. - By default, Vagrant will share the current directory to the
/vagrant
folder on the guest. If you prefer to mount it to a different location, you can manually share a folder with the guest by adding this line:node.vm.synced_folder ".", "/custom/path/hashistack/"
Let’s run Vagrant with this config file:
host:~/hashistack ❯ vagrant up
Bringing machine 'node-1' up with 'virtualbox' provider...
Bringing machine 'node-2' up with 'virtualbox' provider...
==> node-1: Importing base box 'ubuntu/trusty64'...
==> node-1: Matching MAC address for NAT networking...
==> node-1: Checking if box 'ubuntu/trusty64' is up to date...
==> node-1: Setting the name of the VM: hashistack_node-1
==> node-1: Clearing any previously set forwarded ports...
==> node-1: Clearing any previously set network interfaces...
==> node-1: Preparing network interfaces based on configuration...
node-1: Adapter 1: nat
node-1: Adapter 2: bridged
==> node-1: Forwarding ports...
node-1: 22 (guest) => 2222 (host) (adapter 1)
==> node-1: Running 'pre-boot' VM customizations...
==> node-1: Booting VM...
==> node-1: Waiting for machine to boot. This may take a few minutes...
node-1: SSH address: 127.0.0.1:2222
node-1: SSH username: vagrant
node-1: SSH auth method: private key
node-1:
node-1: Vagrant insecure key detected. Vagrant will automatically replace
node-1: this with a newly generated keypair for better security.
node-1:
node-1: Inserting generated public key within guest...
node-1: Removing insecure key from the guest if it is present...
node-1: Key inserted! Disconnecting and reconnecting using new SSH key...
==> node-1: Machine booted and ready!
==> node-1: Checking for guest additions in VM...
==> node-1: Setting hostname...
==> node-1: Configuring and enabling network interfaces...
==> node-1: Mounting shared folders...
node-1: /vagrant => /Users/.../hashistack
==> node-2: Importing base box 'ubuntu/trusty64'...
==> node-2: Matching MAC address for NAT networking...
==> node-2: Checking if box 'ubuntu/trusty64' is up to date...
==> node-2: Setting the name of the VM: hashistack_node-2
==> node-2: Clearing any previously set forwarded ports...
==> node-2: Fixed port collision for 22 => 2222. Now on port 2201.
==> node-2: Clearing any previously set network interfaces...
==> node-2: Preparing network interfaces based on configuration...
node-2: Adapter 1: nat
node-2: Adapter 2: bridged
==> node-2: Forwarding ports...
node-2: 22 (guest) => 2201 (host) (adapter 1)
==> node-2: Running 'pre-boot' VM customizations...
==> node-2: Booting VM...
==> node-2: Waiting for machine to boot. This may take a few minutes...
node-2: SSH address: 127.0.0.1:2201
node-2: SSH username: vagrant
node-2: SSH auth method: private key
node-2:
node-2: Vagrant insecure key detected. Vagrant will automatically replace
node-2: this with a newly generated keypair for better security.
node-2:
node-2: Inserting generated public key within guest...
node-2: Removing insecure key from the guest if it is present...
node-2: Key inserted! Disconnecting and reconnecting using new SSH key...
==> node-2: Machine booted and ready!
==> node-2: Checking for guest additions in VM...
==> node-2: Setting hostname...
==> node-2: Configuring and enabling network interfaces...
==> node-2: Mounting shared folders...
node-2: /vagrant => /Users/.../hashistack
Now you can access your newly provisioned VMs straight from Vagrant itself:
host:~/hashistack ❯ vagrant ssh node-1
Welcome to Ubuntu 14.04.5 LTS (GNU/Linux 3.13.0-164-generic x86_64)
* Documentation: https://help.ubuntu.com/
System information disabled due to load higher than 1.0
Get cloud support with Ubuntu Advantage Cloud Guest:
http://www.ubuntu.com/business/services/cloud
0 packages can be updated.
0 updates are security updates.
New release '16.04.5 LTS' available.
Run 'do-release-upgrade' to upgrade to it.
[email protected]:~$ ifconfig
eth0 Link encap:Ethernet
inet addr:10.0.2.15 Bcast:10.0.2.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:1148 errors:0 dropped:0 overruns:0 frame:0
TX packets:870 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:127518 (127.5 KB) TX bytes:111188 (111.1 KB)
eth1 Link encap:Ethernet
inet addr:192.168.0.111 Bcast:192.168.0.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:253 errors:0 dropped:50 overruns:0 frame:0
TX packets:112 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:20003 (20.0 KB) TX bytes:11371 (11.3 KB)
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:16 errors:0 dropped:0 overruns:0 frame:0
TX packets:16 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:1368 (1.3 KB) TX bytes:1368 (1.3 KB)
Once both virtual machines are created, we will have our multi node setup up and running, as outlined in the “Multi Node (Physical) Deployment” section:

In a future post I will show how to replace Vagrant and automate the deployment on a proper infrastructure (bare metal or cloud) using Terraform. For the time being, I’ve found this cheatsheet useful to remember all commands available from Vagrant.
Ansible Setup
With our hosts up and running, we need a mean to quickly deploy our components in an automated fashion. That’s where we introduce Ansible to automate provisioning.
Ansible Worker
Since I don’t want to run Ansible straight from my host, I created a docker container (hereinafter called the “worker”) to play the role of the control machine.
First, let’s start by preparing the Ansible configuration file (a fully commented template can be found on the Ansible GitHub page):
1
2
3
4
5
6
7
8
9
10
11
host:~/hashistack ❯ cat ansible/ansible.cfg
[defaults]
host_key_checking = False
system_warnings = False
retry_files_enabled = False
remote_user = vagrant
roles_path = /etc/ansible/roles
inventory = /playbooks/inventory
ask_pass = True
# ask_vault_pass = True
# vault_password_file = /playbooks/inventory/ansible_vault_pwd.txt
- Line
6
:vagrant
is the default user configured by Vagrant while setting up the virtual machines. Remember to modify this if you are not using Vagrant. - Line
7
: we are telling Ansible that we will store our roles in the/etc/ansible/roles
folder of the worker. - Line
8
: similar to the line above, we are telling Ansible that we will place inventory-related files in the/playbooks/inventory
folder of the worker. - Line
9
: we are requesting Ansible to ask for thevagrant
user’s password upon running a playbook. - Line
11
: this is the location of a file containing the Vault password used to encrypt thegroup_vars
files. More on this in the Nomad Section below.
Next, we are going to create a Dockerfile for this worker: using alpine
as a base image, we are adding a few packages and folders needed by Ansible to work properly:
host:~/hashistack ❯ cat ansible/Dockerfile
FROM alpine:latest
# Install dependencies
RUN apk update && \
apk upgrade && \
apk add --no-cache --update ansible libffi-dev py-netaddr openssh sshpass zip
# Initialize
RUN mkdir -p /etc/ansible
RUN mkdir -p /playbooks
WORKDIR /playbooks
# RUN
CMD ["/bin/sh"]
Let’s build this image and let’s call it ansibleworker
:
host:~/hashistack ❯ docker build -t ansibleworker:1.0 ansible/
Now, we can run the container by sharing the ansible
folder on the host straight to /etc/ansible
on the worker (so to have the config files already in place to be picked up by Ansible), and the playbooks
folder to /playbooks
(where we are going to store our main playbooks):
host:~/hashistack ❯ docker run -ti --rm -v $(pwd)/ansible:/etc/ansible -v $(pwd)/playbooks:/playbooks ansibleworker:1.0
/playbooks $ ls /etc/ansible/
Dockerfile ansible.cfg requirements.yml roles
Ansible Roles
My original plan was to go the really hard way by manually configuring and deploying every single component, but luckily I’ve recently came across a nice post from Mockingbird Consulting in which they suggested to use Ansible roles pre-made and customised for the main HashiCorp’s products. I’ve considered it for a while, but I ended up agreeing on this solution, which definitely saved me a considerable amount of time.
As suggested in that post, we can use Ansible Galaxy to download some pre-configured roles:
This is reflected in the requirements.yml
file:
host:~/hashistack ❯ cat ansible/requirements.yml
- src: geerlingguy.docker
- src: brianshumate.consul
- src: griggheo.consul-template
- src: brianshumate.vault
- src: brianshumate.nomad
- src: kibatic.traefik
Once ready, let’s use our docker container to pull these roles into the /ansible/roles/
folder:
/playbooks $ ansible-galaxy install --roles-path /etc/ansible/roles -r /etc/ansible/requirements.yml
- downloading role 'docker', owned by geerlingguy
- downloading role from https://github.com/geerlingguy/ansible-role-docker/archive/2.5.2.tar.gz
- extracting geerlingguy.docker to /etc/ansible/roles/geerlingguy.docker
- geerlingguy.docker (2.5.2) was installed successfully
- downloading role 'consul', owned by brianshumate
- downloading role from https://github.com/brianshumate/ansible-consul/archive/v2.3.1.tar.gz
- extracting brianshumate.consul to /etc/ansible/roles/brianshumate.consul
- brianshumate.consul (v2.3.1) was installed successfully
- downloading role 'consul-template', owned by griggheo
- downloading role from https://github.com/griggheo/ansible-consul-template/archive/1.2.1.tar.gz
- extracting griggheo.consul-template to /etc/ansible/roles/griggheo.consul-template
- griggheo.consul-template (1.2.1) was installed successfully
- downloading role 'vault', owned by brianshumate
- downloading role from https://github.com/brianshumate/ansible-vault/archive/v2.1.0.tar.gz
- extracting brianshumate.vault to /etc/ansible/roles/brianshumate.vault
- brianshumate.vault (v2.1.0) was installed successfully
- downloading role 'nomad', owned by brianshumate
- downloading role from https://github.com/brianshumate/ansible-nomad/archive/v1.8.0.tar.gz
- extracting brianshumate.nomad to /etc/ansible/roles/brianshumate.nomad
- brianshumate.nomad (v1.8.0) was installed successfully
- downloading role 'traefik', owned by kibatic
- downloading role from https://github.com/kibatic/ansible-traefik/archive/1.7.6.tar.gz
- extracting kibatic.traefik to /etc/ansible/roles/kibatic.traefik
- kibatic.traefik (1.7.6) was installed successfully
/etc/ansible $ ls -l roles/
total 0
drwxr-xr-x 19 root root 646 Jan 21 19:32 brianshumate.consul
drwxr-xr-x 19 root root 646 Jan 21 19:32 brianshumate.nomad
drwxr-xr-x 19 root root 646 Jan 21 19:32 brianshumate.vault
drwxr-xr-x 12 root root 408 Jan 21 19:31 geerlingguy.docker
drwxr-xr-x 23 root root 782 Jan 21 19:32 griggheo.consul-template
drwxr-xr-x 15 root root 510 Jan 21 19:32 kibatic.traefik
With this step completed, we now have everything we need to finally start provisioning the HashiCorp stack.
Core Components
Core Component 1: Consul (+ dnsmasq)
The glue of the entire infrastructure is provided by Consul and its service discovery capabilities, so we are going to deploy it first. [Please refer to Part 1 for an in-depth description of Consul].
The hosts
file of our inventory is the place where we can instruct Ansible on which service needs to be placed on which host.
We are going to reflect the distribution defined in the “Logical Deployment” section, but please ensure to change the IP addresses accordingly with the ones set in your Vagrantfile
:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
host:~/hashistack ❯ cat playbooks/inventory/hosts
server01.consul ansible_host=192.168.0.111
server02.consul ansible_host=192.168.0.112
[consul_instances]
server01.consul consul_node_role=bootstrap
server02.consul consul_node_role=client
[vault_instances]
server01.consul
[docker_instances]
server01.consul nomad_node_role=both
server02.consul nomad_node_role=client
[traefik_instances]
server01.consul
[nomad_deployer]
server02.consul
Now that we have defined the inventory, we can start deploying the different pieces of our stack.
We are going to use a custom Ansible playbook for this (playbooks/hashistack.yml
), starting with Consul, and adding components as we go:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
host:~/hashistack ❯ cat playbooks/hashistack.yml
- hosts: consul_instances
become: true
become_user: root
roles:
- {role: brianshumate.consul, tags: ['consul']}
vars:
# VARIABLES TO EDIT
consul_iface: eth1
consul_datacenter: "lab"
consul_acl_datacenter: "lab"
consul_domain: "consul"
consul_recursors: ['1.1.1.1', '1.0.0.1']
# CONSTANTS
consul_client_address: "0.0.0.0"
consul_dnsmasq_enable: yes
consul_acl_master_token_display: yes
- Line
2
: the role is going to be applied only to the hosts listed in theconsul_instances
group of thehosts
file. - Line
6
: we are using the Consul role downloaded from Ansible Galaxy. - Line
9
: to be able to interact with the services directly from the host, you’ll need to specify the interface of your Vagrant boxes bridged to your local network. - Line
10
-11
: this is the name of the datacenter. The usual default isdc1
, but here I changed it tolab
. - Line
12
: the name of the Consul domain we are about to create. If you decide to change this variable, please ensure all variables ending in_domain
are set to the same value. - Line
13
: the DNS servers you want to use for external lookups. Here I set them to use the Cloudflare ones.
If you are using Vagrant, you shouldn’t need to change any variable, so we can use the Ansible worker to run the playbook against our hosts:
/playbooks $ ansible-playbook hashistack.yml
SSH password:
PLAY [consul_instances] ****************************************************************
TASK [Gathering Facts] *****************************************************************
[... omitted for brevity ...]
PLAY RECAP *****************************************************************************
server01.consul : ok=38 changed=15 unreachable=0 failed=0
server02.consul : ok=28 changed=11 unreachable=0 failed=0
If everything goes well, you should now have both Consul and Dnsmasq installed.
Now, the way the Ansible playbook spawns the Consul servers (unordered), means that sometimes the clients might have problems joining the cluster.
To solve this, I created a playbook (playbooks/restart.yml
) just to restart the services in the same order defined in the “Logical Deployment” section:
- Consul Servers
- Consul Clients
- Dnsmasq
- Vault
- Nomad Servers
- Nomad Clients
- Traefik
I’m not pasting its entire content here, but you can find it in the GitHub repository of this blogpost. Let’s run this playbook to restart the Consul servers/clients. You might get some errors as we haven’t deployed neither Vault nor Nomad yet. You can ignore them for now:
/playbooks $ ansible-playbook restart.yml
SSH password:
PLAY [consul_instances] *******************************************
TASK [Gathering Facts] *******************************************
ok: [server01.consul]
ok: [server02.consul]
TASK [Restart Consul Servers] *******************************************
skipping: [server02.consul]
changed: [server01.consul]
TASK [Restart Consul Clients] *******************************************
skipping: [server01.consul]
changed: [server02.consul]
TASK [Restart dnsmasq] *******************************************
changed: [server01.consul]
changed: [server02.consul]
[... omitted for brevity ...]
PLAY RECAP *******************************************
server01.consul : ok=7 changed=2 unreachable=0 failed=0
server02.consul : ok=5 changed=2 unreachable=0 failed=0
You can now browse to http://192.168.0.111:8500 (or http://consul.service.lab.consul:8500) to access the Consul’s web UI:

At the same time, you can verify the setup by inspecting Consul’s logs and the DNS resolution:
host:~/hashistack ❯ vagrant ssh node-1
[email protected]:~$ consul monitor
2019/01/22 18:55:13 [INFO] raft: Initial configuration (index=1): [{Suffrage:Voter ID:299c7c71-ec50-4eec-b50a-09dfbff11914 Address:192.168.0.111:8300}]
2019/01/22 18:55:13 [INFO] serf: EventMemberJoin: server01.lab 192.168.0.111
2019/01/22 18:55:13 [INFO] serf: EventMemberJoin: server01 192.168.0.111
2019/01/22 18:55:13 [INFO] agent: Started DNS server 0.0.0.0:8600 (udp)
2019/01/22 18:55:13 [INFO] raft: Node at 192.168.0.111:8300 [Follower] entering Follower state (Leader: "")
2019/01/22 18:55:13 [INFO] consul: Adding LAN server server01 (Addr: tcp/192.168.0.111:8300) (DC: lab)
2019/01/22 18:55:13 [INFO] consul: Handled member-join event for server "server01.lab" in area "wan"
2019/01/22 18:55:13 [INFO] agent: Started DNS server 0.0.0.0:8600 (tcp)
2019/01/22 18:55:13 [INFO] agent: Started HTTP server on [::]:8500 (tcp)
2019/01/22 18:55:13 [INFO] agent: started state syncer
2019/01/22 18:55:13 [INFO] agent: Retry join LAN is supported for: aliyun aws azure digitalocean gce k8s os packet scaleway softlayer triton vsphere
2019/01/22 18:55:13 [INFO] agent: Joining LAN cluster...
2019/01/22 18:55:13 [INFO] agent: (LAN) joining: [192.168.0.111]
2019/01/22 18:55:13 [INFO] agent: (LAN) joined: 1 Err: <nil>
2019/01/22 18:55:13 [INFO] agent: Join LAN completed. Synced with 1 initial agents
2019/01/22 18:55:14 [WARN] raft: Heartbeat timeout from "" reached, starting election
2019/01/22 18:55:14 [INFO] raft: Node at 192.168.0.111:8300 [Candidate] entering Candidate state in term 13
2019/01/22 18:55:14 [INFO] raft: Election won. Tally: 1
2019/01/22 18:55:14 [INFO] raft: Node at 192.168.0.111:8300 [Leader] entering Leader state
2019/01/22 18:55:14 [INFO] consul: cluster leadership acquired
2019/01/22 18:55:14 [INFO] consul: New leader elected: server01
2019/01/22 18:55:14 [INFO] consul: member 'server02' reaped, deregistering
2019/01/22 18:55:14 [INFO] agent: Synced node info
2019/01/22 18:55:18 [INFO] serf: EventMemberJoin: server02 192.168.0.112
2019/01/22 18:55:18 [INFO] consul: member 'server02' joined, marking health alive
[email protected]:/home/vagrant# dig +short server01.node.lab.consul
192.168.0.111
[email protected]:/home/vagrant# dig +short consul.service.lab.consul
192.168.0.111
Core Component 2: Vault
Next in our list is Vault. Vault is a tool that provides a unified interface for securely accessing secrets (e.g., API keys, passwords, or certificates), while providing tight access control and recording a detailed audit log.
We are going to update the main playbook (playbooks/hashistack.yml
) to add this new component:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
host:~/hashistack ❯ cat playbooks/hashistack.yml
- hosts: consul_instances
become: true
...
- hosts: vault_instances
become: true
become_user: root
roles:
- {role: brianshumate.vault, tags: ['vault']}
vars:
# VARIABLES TO EDIT
vault_datacenter: "lab"
vault_domain: "consul"
# CONSTANTS
vault_ui: yes
vault_cluster_disable: yes
- Line
13
-14
: ensure these variables have the same value as those specified for theconsul_instances
Run the playbook again to deploy Vault:
/playbooks $ ansible-playbook hashistack.yml
[... omitted for brevity ...]
TASK [brianshumate.vault : extract systemd version] ****************************************************************
skipping: [server01.consul]
TASK [brianshumate.vault : systemd unit] ***************************************************************************
skipping: [server01.consul]
TASK [brianshumate.vault : Start Vault] ****************************************************************************
changed: [server01.consul]
RUNNING HANDLER [brianshumate.vault : Restart vault] ***************************************************************
changed: [server01.consul]
TASK [brianshumate.vault : Vault API reachable?] *******************************************************************
FAILED - RETRYING: Vault API reachable? (30 retries left).
FAILED - RETRYING: Vault API reachable? (29 retries left).
At some point you will see the task failing while trying to reach Vault. This is happening because, at this point, Vault is still sealed. Head to http://192.168.0.111:8200/ (or http://active.vault.service.lab.consul:8200) to unseal it:
- First, setup the master keys: for this deployment
1
is a “good enough” value for both “Key Shares” and “Key Threshold”, but in a later post we will deploy a hardened configuration with multiple shares. Once done, click on “Initialize”:

- Next, download the master key and the root token:

- You are going to obtain something like the below (remember to store these values!):
{
"keys": [
"9bba87f37cdac08e1fd393b35911fd5e40ee7b5f174dbf3d08c2c528e9951c04"
],
"keys_base64": [
"m7qH83zawI4f05OzWRH9XkDue18XTb89CMLFKOmVHAQ="
],
"root_token": "s.4OH2DO2shJLNnG79pmpVwriA"
}
- Click on “Continue to Unseal”, and enter your master key (
keys
in the snippet above) to finally unseal Vault:

- You will be redirected to the web UI login page, where you can use your root token (
root_token
in the snippet above) as your authentication token:


- Once Vault is unsealed, the Ansible task will resume and complete successfully:
TASK [brianshumate.vault : Vault API reachable?] *******************************************************************
FAILED - RETRYING: Vault API reachable? (30 retries left).
FAILED - RETRYING: Vault API reachable? (29 retries left).
FAILED - RETRYING: Vault API reachable? (28 retries left).
FAILED - RETRYING: Vault API reachable? (27 retries left).
FAILED - RETRYING: Vault API reachable? (26 retries left).
FAILED - RETRYING: Vault API reachable? (25 retries left).
ok: [server01.consul]
PLAY RECAP *********************************************************************************************************
server01.consul : ok=57 changed=30 unreachable=0 failed=0
server02.consul
Just remember you will have to unseal Vault every time you restart it.
Core Component 3: Nomad (+ docker)
The third core component we are going to deploy is Nomad. Nomad is a tool designed for managing a cluster of machines and running applications on them. It abstracts away machines and the location of applications, and instead enables users to declare what they want to run. Nomad will then handle where they should run and how to run them.
We are going to update playbooks/hashistack.yml
once again:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
host:~/hashistack ❯ cat playbooks/hashistack.yml
- hosts: consul_instances
become: true
...
- hosts: vault_instances
become: true
...
- hosts: docker_instances
become: true
become_user: root
roles:
- {role: geerlingguy.docker, tags: ['docker']}
- {role: brianshumate.nomad, tags: ['docker', 'nomad']}
vars:
# VARIABLES TO EDIT
nomad_datacenter: "lab"
nomad_domain: "consul"
nomad_iface: eth1
nomad_network_interface: eth1
nomad_servers: ['server01.node.lab.consul:4647']
# CONSTANTS
nomad_consul_address: http://127.0.0.1:8500
nomad_vault_address: http://active.vault.service.lab.consul:8200
nomad_bind_address: "0.0.0.0"
nomad_use_consul: yes
nomad_vault_enabled: yes
nomad_options: { 'driver.raw_exec.enable': '1' }
nomad_network_speed: 10
nomad_bootstrap_expect: 1
nomad_docker_enable: yes
permanent: yes
nomad_ports_http: 4646
nomad_ports_rpc: 4647
nomad_ports_serf: 4648
- Line
14
-15
: we are going to use both the Docker and Nomad role downloaded from Ansible Galaxy. - Line
18
-19
: ensure these variables have the same value as those specified for theconsul_instances
. - Line
20
-21
: specify the interface of your Vagrant boxes bridged to your local network. - Line
22
: a list of at least one Nomad server, used by the clients to join the cluster. Here we are using the hostname ofserver01
in the Consul domain.
This time we can’t run the playbook straight away, buy we will rather have to provide Nomad with a Vault Token.
Nomad Vault Token Configuration
To use the Vault integration, Nomad servers must be provided a Vault token. This token can either be a root token or a periodic token with permissions to create from a token role. The root token is the easiest way to get started, so we are going to employ it here, but we are going to replace it with a role-based token in a following post.
We are therefore going to add the Vault root token by creating an Ansible-Vault file
at playbooks/inventory/group_vars/docker_instances
:
/playbooks $ mkdir -p /playbooks/inventory/group_vars
/playbooks $ ansible-vault create /playbooks/inventory/group_vars/docker_instances
New Vault password:
Confirm New Vault password:
After choosing a password to protect this file (note it down!), ansible-vault
will spawn your preferred editor, where you can paste the following content:
---
nomad_vault_token: <Vault Root Token>
We now just need to tell Ansible how to fetch this password at runtime, so that it can decrypt the content of this file.
Edit the ansible/ansible.cfg
file and uncomment the line containing ask_vault_pass = True
.
Potentially, if you are not in a production scenario, you could also decide to place this password in a text file, and then provide it to Ansible with the vault_password_file
variable.
[defaults]
host_key_checking = False
system_warnings = False
retry_files_enabled = False
remote_user = vagrant
roles_path = /etc/ansible/roles
inventory = /playbooks/inventory
ask_pass = True
ask_vault_pass = True
# vault_password_file = /playbooks/inventory/ansible_vault_pwd.txt
Re-run Ansible against the server: Nomad will be configured with the appropriate token, and it will be able to authenticate against Vault and retrieve secrets:
/playbooks $ ansible-playbook hashistack.yml
SSH password:
Vault password:
PLAY [consul_instances] ****************************************************************
[...]
TASK [brianshumate.nomad : Create directories] *****************************************
changed: [server01.consul] => (item=/var/nomad)
changed: [server02.consul] => (item=/var/nomad)
changed: [server01.consul] => (item=/var/log/nomad)
changed: [server02.consul] => (item=/var/log/nomad)
TASK [brianshumate.nomad : Create config directory] ************************************
changed: [server01.consul]
changed: [server02.consul]
TASK [brianshumate.nomad : Base configuration] *****************************************
changed: [server01.consul]
changed: [server02.consul]
TASK [brianshumate.nomad : Server configuration] ***************************************
skipping: [server02.consul]
changed: [server01.consul]
TASK [brianshumate.nomad : Client configuration] ***************************************
changed: [server01.consul]
changed: [server02.consul]
TASK [brianshumate.nomad : Start Nomad] ************************************************
changed: [server02.consul]
changed: [server01.consul]
PLAY RECAP *****************************************************************************
server01.consul : ok=33 changed=14 unreachable=0 failed=0
server02.consul : ok=32 changed=14 unreachable=0 failed=0
If you didn’t incur in any error, you can head to http://192.168.0.111:4646 (or http://nomad-servers.service.lab.consul:4646) to access Nomad’s Web UI.

Launch Nomad Jobs
Let’s test our installation by running a job on Nomad while also registering it with Consul.
The playbooks/apps
folder is where we are going to store Nomad job specifications, config files, and everything else that our applications might need. We are going to start with a simple Redis container deployed through the Docker driver for Nomad, as shown below. Going into details of Nomad job specifications is out of scope for this blog post, but the HashiCorp website has good tutorials: 1, 2.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
host:~/hashistack ❯ cat playbooks/apps/redis.nomad
job "example" {
datacenters = ["lab"]
type = "service"
group "sampleservice" {
count = 1
restart {
attempts = 2
interval = "30m"
delay = "15s"
mode = "fail"
}
task "sampleservice-db" {
driver = "docker"
config {
image = "redis:3.2"
network_mode = "bridge"
port_map {
sampleservicedbport = 6379
}
labels {
group = "label_db"
}
dns_servers = [
"consul.service.lab.consul"
]
}
resources {
network {
mbits = 1
port "sampleservicedb_port" {}
}
}
service {
name = "sampleservice-db-service"
tags = ["sampleservice", "redis"]
port = "sampleservicedb_port"
check {
name = "alive"
type = "tcp"
port = "sampleservicedb_port"
interval = "10s"
timeout = "2s"
}
}
}
}
}
To deploy it, we can copy this yaml file to one of our hosts, and then issue the nomad job run
command:
[email protected]:/vagrant$ nomad job run /vagrant/playbooks/apps/redis.nomad
==> Monitoring evaluation "f0e4f8b2"
Evaluation triggered by job "example"
Allocation "0d623665" created: node "332508ea", group "sampleservice"
Evaluation status changed: "pending" -> "complete"
==> Evaluation "f0e4f8b2" finished with status "complete"


Core Component 4: Traefik
The final component is Traefik. Traefik is a reverse proxy/load balancer that natively integrates with every major cluster technology, Consul included. The idea is to use it as the public access point to the infrastructure and provide reverse proxying to the underlying hosts, so that we won’t have to keep remembering host/ports combination. In addition, it has full support for Consul as a service catalog, meaning it can detect new services without the need of a restart.
First, we need to create two additional files:
- The Traefik config file (
playbooks/apps/traefik/traefik.toml
) where we enable both the Docker and Consul integration of Traefik. I’m not pasting the full content here, but you can find this file in the GitHub repository of this blogpost. [For a full explanation of the Traefik configuration file, please refer to the official documentation.] - An init script (
playbooks/apps/traefik/traefik_debianinit.j2
) to replace the startup method (systemd
) used in thekibatic.traefik
role. Indeed, while putting this post together I realised that the main version of Ubuntu supported by Vagrant (trusty
) is too old to properly supportsystemd
. I’m not pasting this file as well (given its length), but it will basically allow us to restart the Traefik service even withoutsystemd
.
One final update to the hashistack.yml
playbook:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
host:~/hashistack ❯ cat playbooks/hashistack.yml
- hosts: consul_instances
become: true
...
- hosts: vault_instances
become: true
...
- hosts: docker_instances
become: true
...
- hosts: traefik_instances
become: true
become_user: root
ignore_errors: True
post_tasks:
- name: Copy Traefik Config File
template:
src: /playbooks/apps/traefik/traefik.toml
dest: /etc/traefik.toml
- name: Copy Traefik init script
template:
src: /playbooks/apps/traefik/traefik_debianinit.j2
dest: /etc/init.d/traefik
owner: root
group: root
mode: 0755
roles:
- {role: kibatic.traefik, tags: ['docker', 'nomad', 'traefik']}
vars:
# VARIABLES TO EDIT
traefik_bind_ip: "192.168.0.111"
traefik_consul_master: "consul.service.lab.consul"
- Line
17
: since thekibatic.traefik
role requiressystemd
to restart the Traefik service, this will cause the entire role to fail on our Vagrant boxes running Ubuntu Trusty. So, only as a workaround, I was forced to add thisignore_errors: True
to make Ansible perform also thepost_tasks
. In the next post of this series we are going to move to a more recent version of Ubuntu in order to address this issue. - Line
19
: in a firstpost_task
to be performed after thekibatic.traefik
role (line31
), we are going to copy theplaybooks\apps\traefik\traefik.toml
template onto the host running Traefik. - Line
23
: in the secondpost_task
we are copying the init script on the same host. - Line
34
: this should be set to the public IP of your infrastructure (in this case I’m using the same local IP ofnode-1
).
Run the playbook one more time and then restart all services:
/playbooks $ ansible-playbook hashistack.yml
...
/playbooks $ ansible-playbook restart.yml
...
If you now browse to the Traefik Web UI at http://192.168.0.111:8081/, you will be able to see the Consul catalog in realtime:

Sample Application
With the stack fully deployed, let’s see how to run an application on Nomad, register it to the Consul catalog, and access it via Traefik. For this post we will limit ourselves to use a simple webserver, whereas in a future post we are going to focus on real applications/services that can be used during a penetration test and/or red teaming engagement.
We first need to create a Nomad job description (playbooks/apps/simpleserver.nomad
):
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
host:~/hashistack ❯ cat playbooks/apps/simpleserver.nomad
job "simple-server" {
datacenters = ["lab"]
type = "service"
group "simpleserver" {
count = 1
restart {
attempts = 10
interval = "5m"
delay = "25s"
mode = "delay"
}
ephemeral_disk {
size = 300
sticky = true
migrate = true
}
task "httpserver0" {
driver = "docker"
config {
image = "python:3.6.4-alpine3.7"
command = "python3"
args = [
"-m",
"http.server",
"8000"
]
port_map {
http = 8000
}
dns_servers = [
"consul.service.lab.consul"
]
work_dir = "/var/www/html/"
}
resources {
network {
mbits = 1
port "http" {
static = 8000
}
}
}
service {
name = "httpserver0"
tags = [
"traefik.tags=service",
"traefik.frontend.rule=PathPrefixStrip:/0/",
]
port = "http"
check {
name = "alive"
type = "tcp"
port = "http"
interval = "10s"
timeout = "2s"
}
}
}
}
}
- Lines
20
-28
: we are going to use thedocker
driver, to run apython:3.6.4-alpine3.7
container on which we will run a command to start a webserver listening on port8000
(python3 -m http.server 8000
) - Lines
50
-53
: we are using tags to override the default frontend rule, specifying that this app can be reached at thehttp://<IP>/0/
URL (more on this below).
Now, we could manually copy this file over to a host, and then again manually run Nomad against it, but since I wanted to automate the process as much as possible I’ve created a new playbook to simplify the process of launching new applications on the infrastructure:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
host:~/hashistack ❯ cat playbooks/apps.yml
- hosts: nomad_deployer
become: true
become_user: root
# TASKS
tasks:
- name: Deploy Simple Server
block:
- name: Copy Nomad Job
copy:
src: /playbooks/apps/simpleserver.nomad
dest: /tmp/simpleserver.nomad
- name: Run Nomad Job
command: nomad job run /tmp/simpleserver.nomad
This playbook will first copy the job description on the nomad_deployer
host (defined in playbooks/inventory/hosts
), and then it will schedule it on Nomad:
/playbooks $ ansible-playbook apps.yml
SSH password:
Vault password:
PLAY [nomad_deployer] ******************************************************************
TASK [Gathering Facts] *****************************************************************
ok: [server02.consul]
TASK [Copy Nomad Job] ******************************************************************
changed: [server02.consul]
TASK [Run Nomad Job] *******************************************************************
changed: [server02.consul]
PLAY RECAP *****************************************************************************
server02.consul : ok=3 changed=2 unreachable=0 failed=0

As a quick smoke test, we can use curl
to query some of the frontends:
❯ curl http://192.168.0.111/0/
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<title>Directory listing for /</title>
</head>
<body>
<h1>Directory listing for /</h1>
<hr>
<ul>
</ul>
<hr>
</body>
</html>
❯ curl -H Host:vault.service.consul http://192.168.0.111/ui/
<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta http-equiv="cache-control" content="no-store" />
<meta http-equiv="expires" content="0" />
<meta http-equiv="pragma" content="no-cache" />
<title>Vault</title>
<meta name="viewport" content="width=device-width, initial-scale=1">
</head>
<body>
<script src="/ui/assets/vendor-9e6bb7bde315fb2e3f4c03b643b80ad9.js" integrity="sha256-qY6bwmjLIRtXBYXqbjGa6A04QwXcy4ZqdNJCSXwUwvc= sha512-rC/1hBN/UP8hl3mrFS2NvdwrW0+f1rPtI6F+WbXtfoJnuNscqAzdTs47Ej3Gwn9LUUOhrKcbj+aAr7PEVlbFcQ==" ></script>
<script src="/ui/assets/vault-6f5373908448768c1c4fcffcf835b5fd.js" integrity="sha256-pcQtWwBPivXuTQYaevxDsmXZ236qdCuMNfUHaAt00DM= sha512-4+rJ5+jQQctOCD2dBp2T9/b2RnVq910SnsnceA25INnXUTGx2aSWklpT2xkh0rDsz+qeQXLNDh/rKl6xK4mc2A==" ></script>
<div id="ember-basic-dropdown-wormhole"></div>
</body>
</html>
Conclusion
This post was Part 2 of the “Offensive Infrastructure with Modern Technologies” series, where I hope I gave out enough information to enable you to automatically deploy the full HashiCorp stack with Ansible.
The full source code used in this article, together with a handy cheatsheet, can be found in the related Github repository: https://github.com/marco-lancini/offensive-infrastructure.
In Part 3 we will continue on this path by hardening the setup we deployed in this post.
I hope you found this post useful and interesting, and I’m keen to get feedback on it! Therefore, if you find the information shared in this series is useful, or if something is missing, or if you have ideas on how to improve it, please leave a comment in the area below, or let me know on Twitter.