There are many resources (e.g., [1]) explaining how to use Vault, but none of them goes into the details of setting it up, especially alongside Consul and docker-compose.

I’m not going into the details of Vault and Consul in this blog post, but, for anyone not familiar with the concepts, let’s just say they are open source tools created by Hashicorp for managing secrets, and for simplifying service discovery, respectively.

The complete setup described in this blog post can be found on Github: https://github.com/marco-lancini/docker_vault.

The Use Case

As a security professional, I often find myself performing assessments of different systems, regardless if they are web/mobile applications, or entire infrastructures. Working in a team, one of the issues we often face is how to share credentials securely among the team members. Credentials managers like KeePass are awesome, but they haven’t been designed for collaboration, and those databases are painful to share and keep up-to-date between all the team members.

That’s where Consul comes handy: ideally we would like to quickly spin up a new instance for every assessment, so to handle password management across the team.

The Setup

Here is the idea:

  • we want to spin up a vault server;
  • which in turn uses consul as a backend storage;
  • and, since we are lazy (and we don’t want to keep messing with the command line), we want to interface with the vault server with a handy web interface (vault-ui);
  • all automagically managed by docker-compose.

After a couple of afternoons spent delving into the documentation of the different services, I came up with the following setup:

$ tree docker_compose_vault
.
├── _data
├── _scripts
│   ├── backup.sh
│   ├── clean.sh
│   ├── setup.sh
│   └── unseal.sh
├── backup
│   └── Dockerfile
├── config
│   ├── admin.hcl
│   └── vault.hcl
└── docker-compose.yml

Let’s start by dissecting the docker-compose file:

$ cat docker-compose.yml

version: '2'

services:
    consul:
        container_name: consul
        image: consul:latest
        ports:
            - "8500:8500"
            - "8300:8300"
        volumes:
            - ./config:/config
            - ./_data/consul:/data
        command: agent -server -data-dir=/data -bind 0.0.0.0 -client 0.0.0.0 -bootstrap-expect=1

    vault:
        container_name: vault
        image: vault
        links:
            - consul:consul
        depends_on:
            - consul
        ports:
            - "8200:8200"
        volumes_from:
            - consul
        cap_add:
            - IPC_LOCK
        command: server -config=/config/vault.hcl

    webui:
        container_name: webui
        image: djenriquez/vault-ui
        ports:
            - "8000:8000"
        links:
            - vault:vault
        environment:
            NODE_TLS_REJECT_UNAUTHORIZED: 0
            VAULT_URL_DEFAULT: https://vault:8200

    backup:
        container_name: backup
        build: backup/
        links:
            - consul:consul
        volumes:
            - ./_data/backup:/backup/
        command: consul-backup

  • First of all, we define a consul service using the consul:latest image provided by Docker Hub. We then expose ports 8500 and 8300. We also specify 2 volumes: config for any configuration file we might need, and /data to provide persistent storage that can survive the container (I specified the local folder ./_data/consul, but you can make it point to a folder of your choosing). Finally, we start the agent in -server (not debug!) mode, specifying the container’s /data folder as the directory where to store the data (this mirrors what we defined in the volumes section).

  • Second service is the vault server, based on the vault image provided by Docker Hub. We provide some links to the consul service, from which it is dependant, then we expose port 8200. We then have to instruct to use the volumes defined for the consul service. Finally, we start the server passing the configuration stored in the vault.hcl file.

$ cat config/vault.hcl

backend "consul" {
  	address = "consul:8500"
  	advertise_addr = "http://consul:8300"
  	scheme = "http"
}
listener "tcp" {
    address = "0.0.0.0:8200"
    #tls_cert_file = "/config/server.crt"
    #tls_key_file = "/config/server.key"
    tls_disable = 1
}
disable_mlock = true

  • Third service is the webui, based on the jenriquez/vault-ui image. For this service we just expose port 8000 and provide links to the vault server.

  • Final service is the backup one, based on the Dockerfile defined in the backup/ folder (and shown below). We specify a volume mapped to the local ./_data/backup and provide a link to the consul service.

# cat backup/Dockerfile
FROM golang

# Get Dependencies
RUN go get -v github.com/hashicorp/consul/api
RUN go get -v github.com/docopt/docopt-go

# Build consul-backup
RUN git clone https://github.com/kailunshi/consul-backup.git
RUN cd consul-backup && go build && cp consul-backup /bin/

# Initialize
RUN mkdir -p /backup
WORKDIR /backup

In Action

Now that we have everything ready, let’s start by bootstrapping our setup with docker-compose up:

$ docker-compose up

Creating network "dockercomposevault_default" with the default driver
Creating consul ...
Creating consul ... done
Creating backup ...
Creating vault ...
Creating backup
Creating vault ... done
Creating webui ...
Creating webui ... done
Attaching to consul, vault, webui, backup
consul    | BootstrapExpect is set to 1; this is the same as Bootstrap mode.
consul    | bootstrap = true: do not enable unless necessary
vault     | ==> Vault server configuration:
vault     |
vault     |                      Cgo: disabled
consul    | ==> Starting Consul agent...
vault     |          Cluster Address: https://consul:8301
vault     |               Listener 1: tcp (addr: "0.0.0.0:8200", cluster address: "0.0.0.0:8201", tls: "disabled")
vault     |                Log Level:
vault     |                    Mlock: supported: true, enabled: false
vault     |         Redirect Address: http://consul:8300
vault     |                  Storage: consul (HA available)
vault     |                  Version: Vault v0.9.1
vault     |              Version Sha: 87b6919dea55da61d7cd444b2442cabb8ede8ab1
vault     |
vault     | ==> Vault server started! Log data will stream in below:
vault     |
consul    | ==> Consul agent running!
consul    |            Version: 'v1.0.2'
consul    |            Node ID: 'fef72b0a-2561-2e3c-725c-127373c452b6'
consul    |          Node name: '4d4a6ed4951e'
consul    |         Datacenter: 'dc1' (Segment: '<all>')
consul    |             Server: true (Bootstrap: true)
consul    |        Client Addr: [0.0.0.0] (HTTP: 8500, HTTPS: -1, DNS: 8600)
consul    |       Cluster Addr: 172.19.0.2 (LAN: 8301, WAN: 8302)
consul    |            Encrypt: Gossip: false, TLS-Outgoing: false, TLS-Incoming: false
consul    |
consul    | ==> Log data will now stream in as it occurs:
webui     | yarn run v1.2.1
webui     | $ nodemon ./server.js start_app
webui     | [nodemon] 1.12.1
webui     | [nodemon] to restart at any time, enter `rs`
webui     | [nodemon] watching: *.*
webui     | [nodemon] starting `node ./server.js start_app`
webui     | Vault UI listening on: 8000

The 4 services are up and running, but we still need to initialize and unseal our vault. I scripted this in the setup.sh file, which will:

  1. Initialize the vault, and save the root and unseal keys in the keys.txt file
  2. Unseal the vault with the keys provided
  3. Authenticate to the server using the vault’s root token
  4. Enable username/password authentication, and create a user to be used by the webui (in this case: “webui/webui”)
  5. Create an authentication token to be used by the backup service (backup_token)
  6. List the secret backends and add a new backend for our assessment, with a dummy entry server1_ad
$ cat ./_scripts/setup.sh

## CONFIG LOCAL ENV
echo "[*] Config local environment..."
alias vault='docker-compose exec vault vault "$@"'
export VAULT_ADDR=http://127.0.0.1:8200

## INIT VAULT
echo "[*] Init vault..."
vault init -address=${VAULT_ADDR} > ./_data/keys.txt
export VAULT_TOKEN=$(grep 'Initial Root Token:' ./_data/keys.txt | awk '{print substr($NF, 1, length($NF)-1)}')

## UNSEAL VAULT
echo "[*] Unseal vault..."
vault unseal -address=${VAULT_ADDR} $(grep 'Key 1:' ./_data/keys.txt | awk '{print $NF}')
vault unseal -address=${VAULT_ADDR} $(grep 'Key 2:' ./_data/keys.txt | awk '{print $NF}')
vault unseal -address=${VAULT_ADDR} $(grep 'Key 3:' ./_data/keys.txt | awk '{print $NF}')

## AUTH
echo "[*] Auth..."
vault auth -address=${VAULT_ADDR} ${VAULT_TOKEN}

## CREATE USER
echo "[*] Create user... Remember to change the defaults!!"
vault auth-enable  -address=${VAULT_ADDR} userpass
vault policy-write -address=${VAULT_ADDR} admin ./config/admin.hcl
vault write -address=${VAULT_ADDR} auth/userpass/users/webui password=webui policies=admin

## CREATE BACKUP TOKEN
echo "[*] Create backup token..."
vault token-create -address=${VAULT_ADDR} -display-name="backup_token" | awk '/token/{i++}i==2' | awk '{print "backup_token: " $2}' >> ./_data/keys.txt

## MOUNTS
echo "[*] Creating new mount point..."
vault mounts -address=${VAULT_ADDR}
vault mount  -address=${VAULT_ADDR} -path=assessment -description="Secrets used in the assessment" generic
vault write  -address=${VAULT_ADDR} assessment/server1_ad value1=name value2=pwd

After running this script we should have your vault unsealed, a set of credentials (“webui/webui”) that can be used to login in the webui, and an authentication token to be used by the backup service.

Once done, we can use docker-compose down to stop the services, while all our secrets will be stored in the _data/consul folder:

$ tree docker_compose_vault
.
├── README.md
├── _data
│   ├── backup
│   └── consul
│       ├── checkpoint-signature
│       ├── checks
│       │   ├── cadcd9b286711802922b3d3108ff1ffa
│       │   └── state
│       │       └── cadcd9b286711802922b3d3108ff1ffa
│       ├── node-id
│       ├── raft
│       │   ├── peers.info
│       │   ├── raft.db
│       │   └── snapshots
│       ├── serf
│       │   ├── local.snapshot
│       │   └── remote.snapshot
│       └── services
│           └── bf3c3c78519c4b4f52cace04789f79ab
├── _scripts
│   ├── backup.sh
│   ├── clean.sh
│   ├── setup.sh
│   └── unseal.sh
├── backup
│   └── Dockerfile
├── config
│   ├── admin.hcl
│   └── vault.hcl
└── docker-compose.yml

Next time docker-compose is started, we will only have to unseal the vault, with the unseal.sh script:

$ cat _scripts/unseal.sh

## CONFIG LOCAL ENV
echo "[*] Config local environment..."
alias vault='docker-compose exec vault vault "$@"'
export VAULT_ADDR=http://127.0.0.1:8200

## UNSEAL VAULT
echo "[*] Unseal vault..."
vault unseal -address=${VAULT_ADDR} $(grep 'Key 1:' ./_data/keys.txt | awk '{print $NF}')
vault unseal -address=${VAULT_ADDR} $(grep 'Key 2:' ./_data/keys.txt | awk '{print $NF}')
vault unseal -address=${VAULT_ADDR} $(grep 'Key 3:' ./_data/keys.txt | awk '{print $NF}')

vault-ui

We could stop here and manage our secrets via the command line, or we could streamline the process a little bit more. Just open a browser and point it to http://127.0.0.1:8000. You should be presented with a login page. Insert the credentials and you’ll be able to manage your vault through a convenient web interface.

Backup & Cleanup

At the end of the engagement, we might want to backup our secrets, and remove any leftovers file.

The backup service, based on the consul-backup script, will store the backup on the volume we specified in the docker-compose.yml file (_data/backup in this case).

$ cat _scripts/backup.sh

echo "[*] Executing backup..."
docker-compose run backup consul-backup -i consul:8500 -t $(grep 'backup_token:' ./_data/keys.txt | awk -v RS='\r\n' '{printf $2}') backup_$(date +%Y-%m-%d)
$ ./_scripts/backup.sh
[*] Executing backup...
Starting consul ... done
map[--aclbackupfile:acl.bkp --restore:false <filename>:backup_2017-12-25 --help:false --version:false --address:consul:8500 --token:763743c6-2f8e-a8e1-ee84-da6d903b7c71 --aclbackup:false]
Backup mode:
KV store will be backed up to file:  backup_2017-12-25

$ tree docker_compose_vault
.
├── README.md
├── _data
│   ├── backup
│   │   └── backup_2017-12-25
│   ├── consul
│   │   ├── checkpoint-signature
...

Finally, the clean.sh script can be used to remove any data stored by the scripts or Consul in the _data folder (remember to move any backup file first!)

$ cat _scripts/clean.sh

read -p "[?] Are you sure you want to remove all Vault's data (y/n)? " answer
case ${answer:0:1} in
    y|Y )
        echo "[*] Removing files..."
        echo "[+] Removing: ./_data/consul/"
        rm -rf ./_data/consul/
        echo "[+] Removing: ./_data/backup/"
        rm -rf ./_data/backup/
        echo "[+] Removing: ./_data/keys.txt"
        rm -f ./_data/keys.txt
    ;;
    * )
        echo "[*] Aborting..."
    ;;
esac

Improvements

The setup described in this blog post should be enough to bring anyone up and running with Vault, but it could still be improved.

For example, I have disabled TLS. To re-enable it, just put the server’s certificate in the config folder and uncomment the relevant lines already put in the config\vault.hcl configuration file.

Cheatsheet

What Steps
First Run 1. Start services: docker-compose up
2. Init vault: ./_scripts/setup.sh
3. When done: docker-compose down
Subsequent Runs 1. Start services: docker-compose up
2. Unseal vault: _scripts/unseal.sh
Backup 1. Start services: docker-compose up
2. Run backup: _scripts/backup.sh
Remove all data 1. Stop services: docker-compose down --volumes
2. Clear persisted data: _scripts/clean.sh

The complete setup described in this blog post can be found on Github: https://github.com/marco-lancini/docker_vault.