Sune Keller’s blog

Vault + Swarm Docker secrets plugin (proof of concept)

2019-04-05T00:00:00+02:00

Background

Secrets have been part of Swarm Mode since its inception, making it trivial to provide generic, static secrets to your distributed services. However, not all secrets are equal, and some use cases call for a more dynamic approach. Docker Engine allows installing a plugin and using it as a driver when creating secrets, letting the value of the secret be determined at runtime, thus enabling dynamic use cases. My talk at DockerCon 2019 in San Fransisco will cover how to write a secrets plugin that fetches dynamic secret values from HashiCorp Vault, and how to deploy it as a Swarm service.

Static vs. dynamic secrets

Generic, static secrets will only get you so far. Once you get to a large enough number of secrets, you’ll either need a very good naming convention, or make sure you label secrets very carefully. Even then they might become cumbersome to manage, which risks either creating too broad policies, or drowning yourself in bureaucracy. And there are other secret management solutions out there, and in this post I will discuss a specific use case with HashiCorp Vault.

Basic example of static secrets

Here’s a basic but complete example (adapted from the official documentation) of using the built-in secrets feature:

First, create passwords for a database:

$ cat /dev/urandom | tr -dc '0-9a-zA-Z!@#$%^&*_+-' | head -c 15 | docker secret create db_password -
$ cat /dev/urandom | tr -dc '0-9a-zA-Z!@#$%^&*_+-' | head -c 15 | docker secret create db_root_password -

Then write a Docker Compose file:

version: "3.7"
services:
  db:
    image: mysql:latest
    command: "--default-authentication-plugin=mysql_native_password" # See https://github.com/docker-library/wordpress/issues/313#issuecomment-400836783
    volumes:
      - db_data:/var/lib/mysql
    environment:
      MYSQL_ROOT_PASSWORD_FILE: /run/secrets/db_root_password
      MYSQL_DATABASE: wordpress
      MYSQL_USER: wordpress
      MYSQL_PASSWORD_FILE: /run/secrets/db_password
    secrets:
      - db_root_password
      - db_password
  wordpress:
    depends_on:
      - db
    image: wordpress:latest
    ports:
      - published: 8000
        target: 80
    environment:
      WORDPRESS_DB_HOST: db:3306
      WORDPRESS_DB_USER: wordpress
      WORDPRESS_DB_PASSWORD_FILE: /run/secrets/db_password
    secrets:
      - db_password
secrets:
  db_password:
    external: true
  db_root_password:
    external: true
volumes:
  db_data:

Finally, deploy the stack:

$ docker stack deploy --compose-file docker-compose.yml example1

Wait a while for the database and blog app to start up, and you’ll be able to visit http://localhost:8000 and see the working Wordpress site, all without you ever seeing the password.

As you can see in the Docker Compose file, both MySQL and WordPress are instructed to use files inside the /run/secrets directory for the database passwords. This is the delivery method selected for secrets.

How static secrets work inside Swarm

When you use Swarm secrets without a plugin, secret data and metadata is saved in the Raft store of the Swarm managers. By design, secret data cannot be updated, and the Docker CLI offers no commands to update secret metadata (i.e. labels). You can, however, update secret labels through the Docker Engine API.

Once you create a service with a secret attached, the secret values are placed as files on a private tmpfs (i.e. an in-memory file-system) mounted inside the container, rather than in environment variables, which are too easily divulged to unconcerned parties.

The problem

You can configure Vault with e.g. your database’s sysadmin credentials, and then use a combination of policies and authentication mechanisms to have Vault dynamically create time-limited user accounts with role-based grants. The most basic authentication method Vault offers based on opaque tokens. Tokens can have policies attached to them, indicating what areas in Vault they give access to. But how do you get them to your containers?

You could of course create a static, long-lived token with the right policies attached, and then type that in as a static secret in Swarm. Then you could attach it to the relevant service, and the service could then communicate directly with Vault to read the secret data, e.g. database credentials. But then you get the problem of rotating the token you typed into Swarm, which either becomes a bureaucratic, repetitive task, or else you risk having to put that secret into your CI/CD system, which can lead to secret sprawl. If you don’t rotate the token, you instead run the risk of the token some day being intercepted, and then you have to rotate it - if you notice, that is.

Ideally, you would give a new token to every instance of your service, and use the use-limit feature of Vault to make sure you can detect interception and ensure a stolen token cannot be reused. You can read more about this concept, which Vault calls Response Wrapping. However, with static Swarm secrets, there is no way of making use of response wrapping. If you typed a response wrapped token into Swarm and made use of it in a service, only the first instance would be able to make use of it, which is by design.

A solution

In order to solve this challenge in a satisfying way, you’ll need to use one of the several extension points of Docker Swarm.

Introduction to the pluggable secrets backend

The pluggable secrets backend allows you to specify a “driver” when creating a secret, e.g. docker secret create --driver <driver_name> <name> <file|->. The plugin must advertise that it implements the "secretprovider" interface, and Docker provides a helpful repository for getting started with writing such a plugin in Go.

When a driver is chosen for a secret, the Swarm manager still looks up the metadata in the raft store, but will request the data from the plugin with the given driver name. The corresponding plugin must be installed on the managers in the Swarm.

Liron Levin from TwistLock contributed the pluggable secrets backend back in 2017.

A proof-of-concept plugin

My idea was to write a plugin that when used will call out to Vault to deliver secret values to Swarm service tasks. One of the requirements was that it should support response wrapping. It was not hard to write, given the go-plugin-helpers repo and the excellent official Vault Go client.

The plugin works like this:

Receive request, including the secret name and labels, service name, ID and labels, the task name and ID
Based on the secret labels, the plugin will then
Create a token on behalf of the service task, with a Vault policy on the token with the same name as the service, and then, optionally:
1. Use that token to read a generic key/value secret from a specified path, and optionally:
  1. Return a specific field inside that path
  2. JSON-encode the returned value
2. Optionally use response wrapping to deliver the returned value
The source code for the proof of concept plugin is available on GitLab.

A complication

Now, when I first tried out the plugin, it worked as intended. However, when I scaled up the service to 2 replicas, I noticed two things:

When setting the secret label that indicated that a generic token should be returned, sometimes the two replicas got the same value, and
When setting the label to use response wrapping, sometimes only one of the tasks would succeed, whereas the other would be told by Vault that the response wrapping token had already been used.

When investigating this puzzling finding, I read through some of the code that assigns secrets and other resources to Swarm nodes. The code responsibly caches the values of secrets such that for each node, any given secret is only requested once from either the raft store or the secret plugin. However, that does not help if the plugin returns values that are supposed to be individual for each task, e.g. when using response wrapping.

I set about writing the necessary changes as PRs #2735 and #2735 for docker/swarmkit. They are currently merged/vendored in moby/moby’s master branch, and should release with Docker 19.03.

Limitations

Until Docker 19.03, you cannot use response wrapping with the plugin. It goes without saying that I do not recommend or even suggest using this plugin anywhere near production, or even in daily use. Rather see it as an example of what can be done with Swarm and its extension points.

Also, the plugin currently relies on getting its own access to Vault (a more privileged token that can perform the plugin’s functions) through suboptimal means. Because plugins in Docker, even if run as containers, are currently very different from regular containers, there are several features you cannot make use of. Namely, even though you can have a special type of Swarm service to install the plugin in your Swarm, there is currently no way for you to attach Swarm secrets to such a service, static or otherwise. This leaves you with the problem of safely bootstrapping the plugin itself. The only method I could think of, which really is half-baked, but works in this POC, is to give the plugin access to the Docker socket of the manager node it is installed on, and use a helper service to hold the bootstrapping token. The plugin then uses the Docker API to find the helper service’s container and reads the bootstrapping token from there.

Usage

See the README in the repo for instructions; here is what it says:

Run ./rebuild.sh, you should get the following - note the different tokens in the two task instances of the snitch service:

$ ./rebuild.sh
...
Success! Uploaded policy: snitch
Key              Value
---              -----
created_time     2018-08-30T02:21:27.980476389Z
deletion_time    n/a
destroyed        false
version          1
Key              Value
---              -----
created_time     2018-08-30T02:21:28.088984314Z
deletion_time    n/a
destroyed        false
version          1
fiqw1xaqjqofvinflvmnzo83t
zj2hvlev230x0s1ei9t25ft9m
overall progress: 1 out of 1 tasks
ij2r01ffy6ak: running   [==================================================>]
verify: Service converged
sirlatrom/docker-secretprovider-plugin-vault
5ioiauam5n9nms9neb6szbwxj
n5j855gu0i460bno1aaw3neq9
t99bbs7y5c1y61tyxxhd3msoj
i9sbmcaqc46v5a44u00fkfxfv
snitch.2.y42sqzj8524y@redacted_host    | secret:              this_was_not_wrapped
snitch.2.y42sqzj8524y@redacted_host    | wrapped_secret:      1afd51f9-c1a2-d4ec-8ceb-8e043b77b53a
snitch.1.gpy8rj3oxz0n@redacted_host    | secret:              this_was_not_wrapped
snitch.1.gpy8rj3oxz0n@redacted_host    | wrapped_secret:      6567b96c-338e-cd3b-e9bc-67c65597fd0f
snitch.2.y42sqzj8524y@redacted_host    | unwrapped_secret:    this_was_once_wrapped
snitch.2.y42sqzj8524y@redacted_host    | generic_vault_token: ddef57f5-a235-923c-4e7c-0a519d307f10
snitch.1.gpy8rj3oxz0n@redacted_host    | unwrapped_secret:    this_was_once_wrapped
snitch.1.gpy8rj3oxz0n@redacted_host    | generic_vault_token: b7b27691-1776-ae52-ffc3-b6a59152d12f
snitch.2.lepelzpcjscj@redacted_host    | secret:              this_was_not_wrapped
snitch.2.lepelzpcjscj@redacted_host    | wrapped_secret:      84df01da-11f9-acba-0373-89bd1f161798
snitch.2.lepelzpcjscj@redacted_host    | unwrapped_secret:    this_was_once_wrapped
snitch.2.lepelzpcjscj@redacted_host    | generic_vault_token: 9214b53f-027c-6552-a0a9-1b18783550d1
snitch.1.edwdvkmvpbke@redacted_host    | secret:              this_was_not_wrapped
snitch.1.edwdvkmvpbke@redacted_host    | wrapped_secret:      df1714e1-a1b6-07eb-10c1-7e1ba4e73022
snitch.1.edwdvkmvpbke@redacted_host    | unwrapped_secret:    this_was_once_wrapped
snitch.1.edwdvkmvpbke@redacted_host    | generic_vault_token: bedf29d5-fbdd-6085-7809-f113078c66b1
...

Future work

Configs can be created with the --template-driver option, allowing you to insert placeholders for secrets (as described here) in your config file and have those be resolved each time a task (container) for a service is created. There will be a template_driver equivalent in the Docker Compose file format eventually (here are the pull requests: docker/cli#1746+docker/compose#6530). Once that is in place (tentatively set for Compose file format version 3.8), you’ll be able to combine configs and secrets and secrets plugins to build a powerful and expressive config management solution, while keeping the concerns of the systems involved neatly separated.

My dream is to be able to write a docker-compose file like this (obviously made-up and not realistic):

version: "3.8"
services:
  app:
    image: ...
    configs:
      - source: config.yml
        target: /etc/app/config.yml
    secrets:
      - vault_token
configs:
  config.yml:
    template_driver: golang
    file: config.yml.tmpl
secrets:
  vault_token:
    name: vault_token
    driver: sirlatrom/docker-secretprovider-plugin-vault
    labels:
      dk.almbrand.docker.plugin.secretprovider.vault.type: "vault_token" # Secret will contain a Vault token
      dk.almbrand.docker.plugin.secretprovider.vault.wrap: "true"        # Enable response wrapping

and a config.yml file like this:

vault_addr: https://vault.example.com:8200
vault_token: {{ secret "password" }}

and the config file would contain a response wrapped long-lived token that the app could then use.

Wouldn’t that be great?

Docker stack deploy: update configs and secrets

2019-01-31T00:00:00+01:00

Background

If you’ve ever deployed a Docker stack using docker stack deploy --compose-file docker-compose.yml <stack_name>, and the docker-compose.yml file has configs in it referring to files in your working directory, you may have hit a snag when you wanted to update the contents of those configs.

The problem

In Swarm Mode, configs and secrets are immutable objects with unique names, and there is no way to mutate their contents. You can update a service, though, to make it refer to a different config or secret.

Say you have this docker-compose.yml file:

version: '3.6'
services:
  app:
    image: nginx
    configs:
      - source: nginx.conf
        target: /etc/nginx/nginx.conf
    secrets:
      - cert.pem
configs:
  nginx.conf:
    file: nginx.conf
secrets:
  cert.pem:
    file: cert.pem

If you have the nginx.conf and cert.pem files in your working directory, Docker will read their contents and create the config and secret with their content, and add references to them to the spec of your app service.

However, if you change the contents of either file, and try and run docker stack deploy ... again, you will get an error message saying you cannot create the config or secret, because it already exists.

A solution

If you expand the configs: and secrets: top-level sections by adding a name to each entry, and add an appropriate environment variable as part of that name, you’ll get the desired result.

If you’re working interactively, you can use the ${LINENO} shell variable which will increase with every command you enter.

If on the other hand you’re in a CI/CD type of situation, you can choose one of several methods:

Use a job variable such as ${CI_JOB_ID} in GitLab or ${BUILD_NUMBER} in Jenkins
Calculate a digest of each referenced file and use that to determine whether a config or secret should be updated

The result of method 1 would look like:

version: '3.6'
services:
  app:
    image: nginx
    configs:
      - source: nginx.conf
        target: /etc/nginx/nginx.conf
    secrets:
      - cert.pem
configs:
  nginx.conf:
    name: nginx.conf-${CI_JOB_ID}
    file: nginx.conf
secrets:
  cert.pem:
    name: cert.pem-${CI_JOB_ID}
    file: cert.pem

The downside of using the job number is that you’ll update the configs and secrets and their dependent services every time you run the job, which is not what you’d expect if you only edited one of the files.

As for method 2, to use digests, you could decide on a certain convention for the variable names like this:

version: '3.6'
services:
  app:
    image: nginx
    configs:
      - source: nginx.conf
        target: /etc/nginx/nginx.conf
    secrets:
      - cert.pem
configs:
  nginx.conf:
    name: nginx.conf-${nginx_conf_DIGEST}
    file: nginx.conf
secrets:
  cert.pem:
    name: cert.pem-${cert_pem_DIGEST}
    file: cert.pem

Then you’d have to run a script along the lines of the following to generate the digests and deploy the stack. Notice that configs and secrets cannot have names exceeding 64 characters, which complicates the script a little. You’ll also need the json_xs tool (found in the libjson-xs-perl package in Ubuntu) and the jq tool (in the jq package).

# Get a list of the keys, names and files of configs and secrets
json_xs -f yaml -t json < docker-compose.yml | jq --raw-output '(.configs,.secrets) | to_entries | map(select(.value | has("file")) | .key, .value.name, .value.file)[]' > configs_and_secrets.txt
# Iterate over each three-tuple
while read entry; read name; read file
do
  # Sanitize the variable name for the digest
  sanitized_filename="$(echo $file | sed 's/[./ ]/_/g')"
  echo "$entry.name: $sanitized_filename"
  # Get the part of the name without any variable references
  name_without_references="$(env -i envsubst <<< "${name}")"
  remainder=$(( 64 - ${#name_without_references} ))
  # Export a variable with the digest, truncate to a total of 64 characters
  export ${sanitized_filename}_DIGEST="$(sha512sum $file | awk '{print $1}' | cut -c -${remainder})"
  echo "Use variable ${sanitized_filename}_DIGEST for $entry"
done < configs_and_secrets.txt
# Deploy the stack
docker stack deploy --prune -c docker-compose.yml stack
# Clean up
rm configs_and_secrets.txt

Notice the --prune flag; it should take care of removing the configs and secrets from the previous deployment next time you deploy, since they’ll no longer be referenced by any services.

Deep Dive: Using Packer and Ansible to create a golden VMware image for Docker Enterprise - part 1

2018-12-12T00:00:00+01:00

Background

In part 1 of my series on Declarative Docker Enterprise, I describe how we used Packer to create a golden image for the VMs that will later make up our Docker Enterprise cluster. This post will detail how that is done.

Note: I will use the terms “Golden Image” and “VM template” interchangeably throughout this post.

The problem

In order to spin up VMs for a Docker cluster, you can either:

boot and install each of them individually, with the very likely risk of making manual mistakes along the way, or
create a VM template first, then clone that whenever you need a new VM.

Obviously, if you go and spin up and provision each VM individually, there are several drawbacks:

The risk of package versions being different between the VMs
The risk of manual mistakes along the way, selecting wrong packages or installer options
Wasting precious time on repetitive tasks

If, on the other hand, you create a single VM template that contains all the packages you need, you only need to clone that template whenever you need a new VM.¹

Assuming you regularly create an upgraded golden image, all you need to do to upgrade your cluster is to replace each VM with a new one cloned from an upgraded VM template. As it turns out, this requires some amount of automation work, but yields the benefits of avoiding the drawbacks listed above, and more.

Creating the VM template

Packer’s involvement in our pipeline consists of two phases:

Create a plain Ubuntu 16.04 installation
Build on top of the first phase to add Docker related packages and configuration

The reason for splitting the process of creating the VM template that will be used to create or upgrade the cluster is to shorten iteration times when making changes to phase 2. More than likely, when you first work on defining what packages go into your Docker VM template, you’ll figure out you’ll need a few more, and having to run that Ubuntu installer every time is really not worth it.

Packer comes with a VMware plugin, but using it requires running it on a host with VMware Fusion for OS X, VMware Workstation for Linux and Windows, or VMware Player on Linux. We’d rather use the vSphere API, and it turns out JetBrains have created a Packer plugin for vSphere that supports both creating a new VM from an ISO, and cloning and provisioning a VM and provisioning it using e.g. Ansible.

Creating the base VM template

Using the vsphere-iso value for the type option in the Packer config allows you to point to an ISO file stored in your vCenter and create a VM with that ISO mounted in the virtual optical drive of the VM. Now, you’re likely to want to automate that install, and for that you can use a Kickstart script to answer the Ubuntu installer’s questions. Using a Kickstart script will also allow you to install a few generic packages that are not specific to Docker, but can help in troubleshooting the VM template itself until you get it right.

Providing the ISO

To boot a VM from an ISO file, you must first upload it to a datastore in your vCenter cluster. We’ve opted for hand-crafting the ISO due to rigid networking requirements, and as such point to a static location in our config.

We try and strive for having as few static/hardcoded/one-off components in our pipeline, and as such, another option would be to compose the ISO file from a stock Ubuntu netboot installer image and adding in any relevant customizations as part of our pipeline. In order to get that ISO file uploaded, we could use the tool govc (maintained by VMware in the govmomi repo) and its datastore.upload command once we’ve crafted your ISO file. That would certainly make upgrading major Ubuntu releases easier.

The config for pointing to an ISO file is as follows, assuming you parameterize the location in the ISO_DATASTORE and ISO_PATH environment variables:

{
  "builders": [
    {
      ...
      "iso_paths": [
        "[{{ user `iso_datastore` }}] {{ user `iso_path` }}"
      ]
      ...
    }
  ],
  "variables": {
    ...
    "iso_datastore": "{{ env `ISO_DATASTORE` }}",
    "iso_path": "{{ env `ISO_PATH` }}",
    ...
  }
}

Kickstart

Configuring your install using a Kickstart script

The detailed list of things we configure using in our Kickstart script is as follows:

System language: lang en_US
Language modules to install (in our case: Danish): langsupport da_DK
System keyboard (Danish layout): keyboard dk
Timezone (Europe/Copenhagen): timezone --utc Europe/Copenhagen
Disable password login for root: rootpw --disabled
Initial username and encrypted password (provided as an environment variable by GitLab, substituted in before the Kickstart config file is put on the virtual floppy disk): user automationuser --fullname "Automation user" --iscrypted --password ${AUTOMATION_USER_CRYPTED_PASSWORD}
Install instead of upgrade: install
Use text mode install: text
Reboot after install: reboot
Use web installation: url --url http://dk.archive.ubuntu.com/ubuntu/
Set hardware clock to UTC: preseed clock-setup/utc boolean true
Set preseed time zone: preseed time/zone string Europe/Copenhagen
Set NTP server: preseed clock-setup/ntp-server <redacted>
Use MBR bootloader: bootloader --location=mbr
Zero out the MBR: zerombr yes
Several options for partitioning:

preseed partman-auto/disk string /dev/sda
preseed partman-auto/method string lvm
preseed partman-lvm/device_remove_lvm boolean true
preseed partman-md/device_remove_md boolean true
preseed partman-lvm/confirm boolean true
preseed partman-lvm/confirm_nooverwrite boolean true
preseed partman-auto/choose_recipe select atomic
preseed partman-partitioning/confirm_write_new_label boolean true
preseed partman/choose_partition select finish
preseed partman/confirm boolean true
preseed partman/confirm_nooverwrite boolean true

Enable shadow file and password hashing: auth --useshadow --enablemd5
Disable firewall for Docker compatibility and because we run an external one: firewall --disabled
Skip configuring the X Window System: skipx

Then follows the list of packages we install initially (some are redacted out):

%packages
build-essential
curl
dnsmasq
dnsmasq-base
dnsmasq-utils
dnsutils
htop
man
nfs-common
ntp
open-vm-tools
rng-tools
software-properties-common
ssh
unzip
vim
wget
curl
ca-certificates

The Kickstart post script

The last touch is the post script, which will run after the installation is complete, but noteably before the automation user has been added. That’s the reason why we create the rc.install script, which will run once the installation is done and the VM has rebooted. It copies the SSH public key of the automation user such that tools in the later phases (both Packer and Ansible) can SSH in as the automation user.

%post --nochroot
touch /target/etc/installdate

umask 077
mkdir -p /media
mount /dev/fd0 /media
cp /media/files/ntp.conf /target/etc/ntp.conf
mkdir -p /target/usr/local/share/ca-certificates/
cp /media/files/almbrand-corporate-pki-root-ca.crt /target/usr/local/share/ca-certificates/almbrand-corporate-pki-root-ca.crt

#### Configure sudoers
cat > /target/etc/sudoers.d/defaults << EOF
Defaults  secure_path="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
Defaults        env_keep="http_proxy https_proxy"
EOF
cat << EOF > /target/etc/sudoers.d/automationuser_nopasswd
automationuser ALL=(ALL) NOPASSWD:ALL
EOF

#### Keep a safe copy /etc/rc.local for later
mv /target/etc/rc.local /target/etc/rc.local.dist

cat > /target/etc/rc.local << EOF
#!/bin/sh -e
#
# rc.local
#
# This script is executed at the end of each multiuser runlevel.
# Make sure that the script will "exit 0" on success or any other
# value on error.
#
# In order to enable or disable this script just change the execution
# bits.
#
# By default this script does nothing.

if [ -x /etc/rc.install ]
then
    /etc/rc.install && mv /etc/rc.install /etc/rc.install.1
else
    echo /etc/rc.install does not exist >> /root/rc.install.log
fi

exit 0
EOF

chmod +x /target/etc/rc.local

#### Create /etc/rc.install ####
cat > /target/etc/rc.install << EOF
#!/bin/bash

# rc.install
#

#### automationuser SSH public key
mkdir -p /home/automationuser/.ssh
chmod 0700 /home/automationuser/.ssh
chmod 0600 /home/automationuser/.ssh/authorized_keys
mount /dev/fd0 /media
cp /media/files/authorized_keys /home/automationuser/.ssh/authorized_keys
umount /media
chown -R automationuser. /home/automationuser/.ssh

#### Corporate PKI Root CA certificate
(
  update-ca-certificates
) 2>&1 | tee /tmp/certinst.log
EOF

chmod +x /target/etc/rc.install

Making a Kickstart script available to the installer

You have two options:

Bake the kickstart script into the ISO file, or
Use a neat trick

Here’s the neat trick: JetBrains’ Packer plugin for vSphere allows you to create and provision a floppy disk with select files and attach it to your VM such that the installer can reach the files on it. This way, we’re able to keep our Kickstart script in the same repo as our Packer config and avoid having to run a file server to host any files we need during provisioning, thus reducing external dependencies for a successful VM template build.

Specifically, you add these keys in your Packer template file, assuming your Kickstart script file is called ks.cfg and your auxillary files are in a files/ subdirectory relative to your Packer template file:

{
  "builders": [
    {
      ...
      "floppy_dirs": [
        "{{ template_dir }}/files"
      ],
      "floppy_files": [
        "{{ template_dir }}/ks.cfg"
      ],
      ...
    }
  ]
}

Speeding up iterations

Set the create_snapshot option to true to have a snapshot created after the VM has been successfully installed. This will allow you to use the linked_clone option in the next phase, which leads to much faster cloning of the VM, increasing your iteration speed.

Another aspect of keeping iterations fast is to avoid installing too many packages in this first phase, since the cost in terms of waiting time is larger than in the next phase.

SSH access to the VM template during provisioning

This step is not really needed as long as you’re only using Kickstart to run your provisioning, but you may easily end up in a situation where more advanced provisioning will require a tool such as Ansible, or to execute commands on the created VM template before it is considered done. For such purposes (and, indeed, for the configuration to be accepted by the vSphere builder plugin), you have to tell what SSH user and private key file will be used to access the VM once provisioned. In our case, we make sure to put that user’s public key file on the VM as described in The Kickstart post script section above, specifically these lines:

mount /dev/fd0 /media
cp /media/files/authorized_keys /home/automationuser/.ssh/authorized_keys
umount /media

You’ll want to set the following options and make sure your SSH private key is accessible from .ssh/id_rsa inside your pipeline’s working directory (or, if you’re running this from a laptop, whichever directory you’re running it from then). You can modify the "ssh_private_key_file" and "ssh_username" values to suit your needs.

json
{
  "builders": [
    {
      ...
      "ssh_private_key_file": "{{ template_dir }}/.ssh/id_rsa",
      "ssh_username": "automationuser",
      ...
    }
  ]
}

The final config

As you can probably see from the config, we parameterize almost all the values. This is because we have multiple vCenters, and we need to build the templates in all of them. In our repo, this file is saved as from-iso.json.

{
  "builders": [
    {
      "CPU_limit": -1,
      "CPU_reservation": 0,
      "CPUs": "2",
      "RAM": "4096",
      "boot_wait": "2s",
      "cluster": "{{ user `cluster` }}",
      "convert_to_template": true,
      "create_snapshot": true,
      "datacenter": "{{ user `datacenter` }}",
      "datastore": "{{ user `datastore` }}",
      "disk_size": 51200,
      "disk_thin_provisioned": true,
      "floppy_dirs": [
        "{{ template_dir}}/files"
      ],
      "floppy_files": [
        "{{ template_dir }}/ks.cfg"
      ],
      "folder": "{{ user `folder` }}",
      "guest_os_type": "ubuntu64Guest",
      "iso_paths": [
        "[{{ user `iso_datastore` }}] {{ user `iso_path` }}"
      ],
      "network": "/{{ user `datacenter` }}/network/{{ user `network` }}",
      "network_card": "vmxnet3",
      "password": "{{ user `vsphere_password` }}",
      "resource_pool": "{{ user `resource_pool` }}",
      "ssh_private_key_file": "{{ template_dir }}/.ssh/id_rsa",
      "ssh_username": "automationuser",
      "type": "vsphere-iso",
      "username": "{{ user `vsphere_user` }}",
      "vcenter_server": "{{ user `vsphere_server` }}",
      "vm_name": "{{ user `vm_name` }}"
    }
  ],
  "post-processors": [],
  "variables": {
    "cluster": "{{ env `CLUSTER` }}",
    "datacenter": "{{ env `DATACENTER` }}",
    "datastore": "{{ env `DATASTORE` }}",
    "folder": "{{ env `FOLDER` }}",
    "iso_datastore": "{{ env `ISO_DATASTORE` }}",
    "iso_path": "{{ env `ISO_PATH` }}",
    "network": "{{ env `NETWORK` }}",
    "resource_pool": "{{ env `RESOURCE_POOL` }}",
    "vm_name": "{{ env `BASE_VM_TEMPLATE_NAME` }}",
    "vsphere_password": "{{ env `VSPHERE_PASSWORD` }}",
    "vsphere_server": "{{ env `VSPHERE_SERVER` }}",
    "vsphere_user": "{{ env `VSPHERE_USER` }}"
  }
}

Creating the Docker VM template

We perform the VM template creation from within our GitLab CI pipeline, but there isn’t much to it other than setting the relevant environment variables (all listed in the "variables" key in the config above). In order to avoid having to install Packer on our GitLab runner, we run Packer from a Docker image thusly:

$ docker run \
  --rm \
  --volume ${PWD}:/data --workdir /data \
  --tmpfs /tmp \
  --env BASE_VM_TEMPLATE_NAME \
  --env CLUSTER \
  --env DATACENTER \
  --env DATASTORE \
  --env FOLDER \
  --env ISO_DATASTORE \
  --env ISO_PATH \
  --env NETWORK \
  --env VSPHERE_SERVER \
  --env VSPHERE_USER \
  --env VSPHERE_PASSWORD \
  $packer_image \
  build from-iso.json

As you can see, the variables are all defined outside of the command, meaning Docker will take the values of the variables in the environment and pass them on to the container.

Additionally, we make sure to make the working directory available to Packer by mounting the current working directory on /data inside the container and set it as the working directory of the container with --workdir /data.

Another important detail is that $packer_image bit - we build our own Packer image for Docker in order to add a few more tools in there, namely govc and jq, as well as the two Packer vSphere builders released by JetBrains. In fact, since the next phase will use Ansible to provision Docker Enterprise on the next template, we base our final image off of William Yeh’s Ansible image on Docker Hub. The Dockerfile for that image looks like this:

ARG         packer_version=1.3.3@sha256:e65fb210abc027b7d66187d34eb095fffa2fd8401e7032196f760d7866c6484c
FROM        hashicorp/packer:${packer_version} AS packer

FROM        williamyeh/ansible:alpine3@sha256:8072eb5536523728d4e4adc5e75af314c5dc3989e3160ec4f347fc0155175ddf

# Copy in corporate certificates
COPY        *.crt /usr/local/share/ca-certificates/
RUN         update-ca-certificates

# Add utilities used inside later Packer builds
RUN         apk add --update bash jq wget curl
COPY --from=packer /bin/packer /bin/packer

# Download the Packer vSphere builders from the JetBrains infra repo
ARG         vsphere_packer_builders_version=2.1
ADD         https://github.com/jetbrains-infra/packer-builder-vsphere/releases/download/${vsphere_packer_builders_version}/packer-builder-vsphere-clone.linux /bin/packer-builder-vsphere-clone.linux
ADD         https://github.com/jetbrains-infra/packer-builder-vsphere/releases/download/${vsphere_packer_builders_version}/packer-builder-vsphere-iso.linux /bin/packer-builder-vsphere-iso.linux

# Install GOVC
ARG         govmomi_version=v0.19.0
ADD         https://github.com/vmware/govmomi/releases/download/${govmomi_version}/govc_linux_386.gz /bin/govc.gz
RUN         cd /bin && gunzip govc.gz && chmod +x /bin/packer-builder-vsphere-iso.linux /bin/packer-builder-vsphere-clone.linux govc

# Add a default ansible.cfg
COPY        ansible.cfg /etc/ansible/ansible.cfg

# Add an automation user
RUN         adduser -u 1000 -D automationuser
USER        automationuser

# Set the entrypoint to run Packer by default
ENTRYPOINT  [ "/bin/packer" ]

You then build and push that image to your registry (Docker Hub, DTR or whichever you prefer) and use that to run Packer.

Conclusion

Building a golden image VM template with Packer and vSphere is perfectly doable, the tools are all there, readily available and well documented.

The next part of this deep dive will deal with actually installing Docker Enterprise in a subsequent VM template.

Obviously you’ll still need to specify the unique config (e.g. hostname, VM name, possibly IP address) of each VM, but that’s what Terraform is for. Read about that in an upcoming blog post. ↩

Declarative Docker Enterprise with Packer, Terraform, Ansible and GitLab - part 2

2018-08-21T00:00:00+02:00

This is the second part in a series about building and upgrading Docker EE clusters while striving for a declarative approach. See part 1 for more background.

Post updated Aug 22 2018: Added conclusion section.

Creation

The first time a cluster is to be created, things are a little different. There are no existing VMs, and thus no services running on them. This makes things simpler in terms of how we apply the planned changes using Terraform.

Terraform config

In broad terms, the nodes that will make up the cluster are divided into three groups, which is reflected in our Terraform config:

UCP Controllers (named managers)
UCP Workers (named workers)
DTR replicas (named dtrs)

Here’s a list of variables that we define for every VM (using a map from VM name to value):

MAC address
Deployment stage
vSphere Resource Pool
vSphere host ¹
vCenter folder path
Number of vCPUs
Memory size in MB
Disk size ²
Datacenter name (e.g. dc1 or dc2)
Primary network name ³
Storage network name ³ ⁴
Storage IP address ⁴
Storage policy name ⁴

That’s a lot! Additionally, we have to ask the network team to create DHCP reservations and VIP addresses, ask our storage vendor to add the storage IP addresses to the whitelist for the given storage policy, and request TLS certificates for UCP and DTR from our internal PKI. Those are all tasks ripe for automation, but we haven’t got there yet.

But once all that is defined, we’re good to go and can run our pipeline in GitLab.⁵

GitLab pipeline

The GitLab pipeline is configured as a number of .gitlab-ci.yml files. Since we’re using GitLab Enterprise, we can include other YAML files from our main .gitlab-ci.yml file, leading to a slightly neater decomposition. However, we specify the same CI jobs for every cluster:

Pre*
Plan
Apply
Upgrade
Re-run Ansible

* The “Pre” job is a single job that is always run before the plan phase.

As can be seen in the above screenshot, we divide our pipeline into three phases (the “Upstream” phase comes from the job being triggered by the previous repo’s CI pipeline), namely:

Pre
Plan
Apply

Preparing essential Docker images

The “Pre” phase has one proper job: Pull any essential Docker images from a registry that already exists (we still need to solve this catch-22), save them in a .tar.gz file for later use.

Planning

The “Plan” phase has a job for each cluster. The job gives the Terraform variables file pertaining to the cluster as the argument to the -var-file option of the terraform plan command and saves the output in a file that is archived in GitLab. The job output log will show what actions Terraform will perform if the plan is applied. For a new cluster, this will include a number of VMs in each of the managers, workers and dtrs resource groups, as well as a number of so-called null resources which are used, among other things, to insert custom scripts at different points in the lifecycle of applying the plan. Namely, this is how we run Ansible after Terraform has created the VMs. Terraform keeps track of the null resources, but they don’t represent an object in any other system.

Applying

Hang on, this is where it gets complicated!

The “Apply” phase’s main job is the “Apply” job. It fetches the plan from the “Plan” job and applies it by running terraform apply <plan.out>. For a new cluster, the following things will happen in sequence:

Terraform creates the UCP Controller VMs (managers) by cloning the Docker + UCP VM template from part 1
Terraform creates an Ansible inventory file only including the manager nodes in a group called ucp (sorry about the naming inconsistencies)
Terraform runs the script in the ansible_managers null resource
1. Ansible runs a playbook that consists of multiple nested playbooks that contain the tasks for setting up the Docker Swarm cluster and installing UCP
  1. The current Swarm cluster status of each manager node is collected
  2. If there are no managers in any existing cluster (always the case first time around), the first manager will initialize a new Swarm cluster
  3. The UCP configuration file from our repo is copied into the VM and created as a Swarm config using the docker config create command
  4. The TLS certificates that we inject into the pipeline using a GitLab secret variable are copied into the VM and copied to a newly created ucp-controller-server-certs Docker volume (instructions here)
  5. The configured version of UCP is installed using docker run ... docker/ucp:<version> install (see the install docs)
  6. We wait for the ucp-reconcile container on the first manager node to exit successfully (see the UCP architecture page), indicating that UCP is up and running
  7. We get the join token for Swarm managers from the first manager node by running docker swarm join-token manager
  8. We collect a list of remaining manager nodes to join to the cluster
  9. For each of the remaining manager nodes, one at a time, we do the following:
    1. Run docker swarm join --token <token> <first-node-addr>:2377 to join the Swarm cluster
    2. Wait for the ucp-reconcile container on the first manager node to exit successfully (see the UCP architecture page), indicating that UCP is up and running

Once the manager nodes have been joined into a working Swarm/UCP cluster, we need to perform additional configuration of UCP for it to suit our needs. This is also done as part of the pipeline. Our Terraform config has a couple of additional variables:

Teams (a mapping from team name to LDAP filter)
Grants (a mapping from collection to a tuple of a team name and a role)

Using the UCP HTTP API, we create the teams, checking for each one that it does not already exist, and configure its LDAP sync settings. We then create all the collections from the “Grants” variable. Some collections run several levels deep, and for those, we add a dummy entry to create their parent collections first. Once the collections have been created, we create grants for the teams with the specified roles. All this required some extra Ansible foo - the import_tasks action comes in handy, as it allows running a sequence of tasks for each element in a given map/dict or list variable.

Terraform then creates the Worker and DTR nodes
Terraform creates an Ansible inventory file including the manager nodes in a group called ucp (sorry about the naming inconsistencies), the worker nodes in a group called worker and the DTR nodes in a group called dtr
Terraform then runs the script in the ansible_all null resource
1. Ansible runs a playbook that consists of multiple nested playbooks that contain the tasks for joining the worker nodes, install DTR and join DTR replicas
  1. We get the join token for Swarm workers from the first manager node by running docker swarm join-token worker
  2. The current Swarm cluster status of each non-manager node is collected
  3. If the node is not already in the cluster, we join it by running docker swarm join --token <token> <first-node-addr>:2377
  4. We add a Swarm node label to every node based on its value from the Terraform config’s deployment stage variable (this is later used for constraining Swarm services to specific stages when the user has access to multiple stages), and one that signifies what datacenter the node is in
  5. We also add a Swarm node label to assign each node to a UCP collection by setting the com.docker.ucp.access.label label, e.g. to prod, which will restrict which UCP users will be able to see and interact with/schedule services on the node
  6. We wait for the ucp-reconcile container on the first manager node to exit successfully (see the UCP architecture page), indicating that the UCP worker components are up and running
  7. When all worker and DTR nodes are correctly configured as plain workers, we need to install DTR:
    1. If there are no existing DTR containers on any DTR node (we check for the presence of a dtr-api-* container), we install DTR using the docker run ... docker/dtr install command, referring to the load-balanced VIP address of our UCP cluster (using the --ucp-url option), and provide the NFS address of our externalized storage (using the --nfs-storage-url option) as well as the TLS certificates (using the --dtr-{ca,cert,key} options)
    2. For each of the remaining DTR nodes, we run the docker run ... docker/dtr join command, which replicates the DTR database to the joining replica nodes and makes them part of the DTR cluster

Phew, that was a lot, and that was even skipping over a few details!

Upgrading

The upgrade starts out with a list of what template was used for each VM currently in the cluster, then swaps out one at a time with the most recent template.

E.g., if the template list looked like this:

templates = {
    "ucp1"    = "Ubuntu1604DockerTemplate-20180604-154739-master-50798-push"
    "ucp2"    = "Ubuntu1604DockerTemplate-20180604-154739-master-50798-push"
    "ucp3"    = "Ubuntu1604DockerTemplate-20180604-154739-master-50798-push"
    "worker1" = "Ubuntu1604DockerTemplate-20180604-154739-master-50798-push"
    "worker2" = "Ubuntu1604DockerTemplate-20180604-154739-master-50798-push"
    "worker3" = "Ubuntu1604DockerTemplate-20180604-154739-master-50798-push"
}

… we first look up the deployment stage for each node. We process them in this order: UCP, DTR, Workers (in increasing deployment stage order, i.e. dev –> prod). Then we proceed by swapping in the new templat name for the ucp1 node, and the template list will look like this instead:

templates = {
    "ucp1"    = "Ubuntu1604DockerTemplate-20180820-022846-master-63776-pipeline"
    "ucp2"    = "Ubuntu1604DockerTemplate-20180604-154739-master-50798-push"
    "ucp3"    = "Ubuntu1604DockerTemplate-20180604-154739-master-50798-push"
    "worker1" = "Ubuntu1604DockerTemplate-20180604-154739-master-50798-push"
    "worker2" = "Ubuntu1604DockerTemplate-20180604-154739-master-50798-push"
    "worker3" = "Ubuntu1604DockerTemplate-20180604-154739-master-50798-push"
}

You’ll notice the template for ucp1 having changed from Ubuntu1604DockerTemplate-20180604-154739-master-50798-push to Ubuntu1604DockerTemplate-20180820-022846-master-63776-pipeline. A useful tool called json2hcl (Docker image) helps us convert between Hashicorp’s HCL configuration format and JSON, allowing us to manipulate the config with jq.

We then run terraform plan followed by terraform apply, and Terraform should then show the ucp1 VM as tainted since its template VM name has changed, forcing the VM to be destroyed and a new one created in its place. Thankfully, Terraform allows us to hook into the destroy process, running scripts before the VM is destroyed. Furthermore, we can abort the destruction of the VM if our script errors out, allowing us to stop the process and carry out manual intervention when facing specific failure conditions. Terraform remembers its progress and will re-attempt to replace the VM next time we run the pipeline.

Here’s the general process for upgrades:

Retrieve list of templates for current nodes from Terraform
For each stage (usually in this order: UCP, DTR, other workers):
1. For each VM in that stage:
  1. Update the template name to the most recent from the VMware vCenter catalog (the selection mechanism is likely to be improved to be more granular)
  2. Run terraform plan followed by terraform apply, resulting in:
    1. Terraform running a pre-destroy script that runs an Ansible playbook:
      1. Draining the node from running tasks⁶
      2. Performing additional tasks for UCP* or DTR** nodes
      3. Having the node leave the swarm and remove it from the node list
    2. Terraform destroying the existing VM in vSphere
    3. Terraform creating a new VM in vSphere cloned from the new template
    4. Terraform running the scripts of the ansible_managers and ansible_all null resources as described in the Apply section:
      1. Joining the new VM to the Swarm cluster, wait for it to be marked as healthy
      2. Putting the VM in the right UCP collection (using node labels) so tasks can be scheduled on it
      3. Performing additional tasks for UCP* or DTR** nodes
      4. When the last VM has been successfully upgraded to the same Docker Engine version, UCP and DTR are upgraded if the desired version is newer than what’s currently installed:
        
        We take a backup of UCP using the docker run ... docker/ucp backup command, and copy the resulting archive to the GitLab runner for archival.
        
        UCP is upgraded by running docker run ... docker/ucp upgrade command, and the command runs synchronously until the upgrade completes.
        
        We take a backup of DTR’s metadata (not its image data, which resides on an NFS share), which takes a considerable time, and copy the resulting archive to the GitLab runner for archival. Note that we have to run the output of the backup command through gzip for the resulting file size to be small enough that Ansible’s expensive fetch action doesn’t consume all available memory on our GitLab runner VM.
        
        The process is similar for DTR, docker run ... docker/dtr upgrade, which upgrades each component one at a time on each DTR replica.

* If the node being upgraded is one of the UCP controllers, when it is to be destroyed, it is demoted such that the other controllers will be made to know about the (temporary) new number of controllers.

** If the node being upgraded is one of the DTR replicas, when it is to be destroyed, it is first removed from the DTR cluster such that the other replicas will know about the (temporary) new number of replicas. When its replacement has been joined to the cluster as a worker node, it is additionally joined to the DTR cluster to become one of the available replicas.

Conclusion

Using this method, we can use the same approach whether we upgrade Docker Engine, add new Ubuntu packages or apply a security patch, by simply doing that once in our VM template and rolling the upgrade out across our cluster. There’s no ambiguity about whether a node is updated or not; it is either based on the latest VM template, or it isn’t. If it is, it’s up to date.

Once we’ve gone through a few more supervised rolling upgrades, we want it to be a regular scheduled job (GitLab can help with visibility there compared with an oldfashioned cron job), e.g. running weekly.

In an upcoming post, I will provide a deep dive into the configuration of the various tools involved, namely Packer, Terraform, Ansible and GitLab, and any quirks that may be useful to other organizations.

The direct host assignment will be replaced with DRS group membership, the management of which was introduced in the Terraform vSphere Provider v1.5.0. ↩
Since the disk size must be identical to that of the VM template when using linked clones, this is practically always the same for all VMs. ↩
These are made as indirect lookups to prevent data duplication. ↩ ↩²
These related to our use of the Netapp Docker Volume Plugin. ↩ ↩² ↩³
Naturally, we have to define numerous variables in GitLab itself, too, in order to be able to externalize some configuration and to provide secret variables. ↩
When a Swarm node is put in the drain availability status, it currently doesn’t wait for tasks to exit gracefully – rather, they are stopped rather shortly after, thus potentially causing downtime if the service does not have any replicas on other nodes in the stage. ↩

Declarative Docker Enterprise with Packer, Terraform, Ansible and GitLab - part 1

2018-08-17T00:00:00+02:00

Background

At Alm. Brand, we’ve been running Docker in production since the first beta of Docker Universal Control Plane (UCP). This is the story about how we moved on to a more automated and declarative approach.

We started with greenfield services, a simple service discovery, config management and dynamic load balancing setup and accelerated quickly. After having proven the new Docker based platform, interest grew in Dockerizing and migrating the legacy apps running on our organically grown application server infrastructure, which had become painful to manage, and required daily firefighting.

In our DockerCon Europe 2017 talk, we describe our process and the gains from our migration journey, so I will not dive further into that in this post. Rather, I will describe how we became victims of our own success, and what we did to better the situation.

The problem

In the summer of 2017, it became clear that co-locating our license constrained legacy apps and our open-source based greenfield services would prevent us from scaling our infrastructure to match the growing number of new services and migrated apps. As we’re an Enterprise™, it would become the end of 2017 before we were able to start working on a solution, since, finally, the resources in the cluster were becoming exhausted, causing outages and painful firefighting in both dev/test and production.

The most straightforward solution would be to build a new Docker cluster and migrate the greenfield services to it, thus freeing up resources on the first cluster. To pull that off as fast as possible, that would of course require several key employees to be available within a short amount of time in order to modify external load balancers, create DHCP reservations, open firewall ports, create and provision VMs, etc.

However, we were determined to make better use of our learnings from building and running the first cluster. We temporarily scaled up the first cluster, the bulk cost of which was additional licenses for the proprietary legacy software, and thus bought ourselves time for creating a better solution.

Our solution

Some of our learnings from the first two years of running a Docker cluster were:

Keeping OS packages on VMs up to date, even with config management tools, does not completely prevent configuration drift
Manual work leads to human error, some of which will only be discovered further down the line
If only a few people work daily with a system, it will become less approachable to newcomers, unless an effort is made to codify and automate it
As much as possible of the data and process involved in creating or modifying a cluster should be versioned and executable as a pipeline
When the processes are different for changing different components, it requires more knowledge and context switching. Simplifying processes can save time and reduce errors.

Creating a golden image

Since we run our own VMware based infrastructure, we have to create VMs from scratch. Fortunately, HashiCorp Packer is a very useful tool in that regard. JetBrains have made a VMware vSphere plugin for Packer that helps create a new VM or VM template from a given ISO file. The plugin also supports creating a virtual floppy drive to be made available to the installer, which we make use of to supply a Kickstart script. The Kickstart script configures the system account that we use in the following stages, and installs basic packages and configuration such as NTP, internal CA certificates, corporate proxy environment variables, timezone, etc.

This process runs in a GitLab pipeline for a repo aptly named golden-image-base. In order to keep iteration times short, we save the resulting VM as a template at this stage and hand off to another repo’s pipeline, named golde-image-docker-ucp.

In the Docker + UCP repo’s pipeline, we clone the base VM template and use Packer’s Ansible provisioner to provision it further. Among other things, we use govc to add a separate disk for the /var/lib/docker partition as recommended in the CIS CE Benchmark v1.1.0 section 1.1. We also installed a configurable version of Docker Engine and pre-pull the UCP and DTR images from Docker Hub. To get the list of images to pull for UCP, you can run docker run --rm docker/ucp images --list. Beware that the output contains carriage returns (\r), which can interfere with automation, so pipe it through tr -d '\r'. For DTR, the command is docker run --rm docker/dtr images, and it also yields carriage returns in the output.

The final VM template is now ready to be used by the pipeline of the next repo in the process, ucp-provisioner. That will be described in the next part of the series.

Using the new config and secret templating in Docker CE 18.03

2018-04-01T00:00:00+02:00

Background

A really helpful new feature just landed in Docker CE 18.03. Since Swarm Secrets were introduced in Docker 1.13 (January 2017) and Swarm Configs came around in 17.06, getting sensitive data and configuration files securely distributed to your Swarm service containers is now more or less a solved problem. Using Swarm configs, you no longer have to bake configuration files into a custom image, or use external means to distribute the configuration files to all nodes in your Swarm cluster.

The problem

However, if dense config files also contain secret data, what are your options? Putting the configuration file in a Swarm secret will make it cumbersome for operators to inspect the non-sensitive parts of the configuration, and using a Swarm config will make it too easy to reveal any secrets as part of daily operations.

Some of the official images on Docker Store have built-in entrypoint scripts to help specifying the file containing e.g. the MariaDB root password by pointing an environment variable to a file, which could very well be a Swarm secret located in the /run/secrets/ directory. However, most images have not been adapted to support Swarm secrets natively.

A simple example is the official Redis image. Given that you wanted to set other configuration options than the password, you would have to do one of the following things:

Expose the password to operators: Create a redis.conf Swarm config including the password in plaintext, visible to everyone who can inspect the config through the Docker API or who looks over the shoulders of such a person.
Hide all config from operators: Put the entire redis.conf file inside a Swarm secret, making it hard for operators to see the non-sensitive contents of the Redis configuration. If they copy out the contents from the container, they will still have unwillingly exfiltrated the secret password and risked its exposure.
Hack together some custom templating:
1. Create a redis.conf Swarm config with a placeholder for the password,
2. Create a Swarm secret with the password,
3. Add a custom entrypoint script to replace it with the contents of the correct secret file inside /run/secrets/,
4. Write a Dockerfile which includes the entrypoint script, distribute the resulting image to your Swarm through a public or private registry and keep track of official image updates forever, repeating this step for every important feature or security update,
5. Become miserable because of point 4.

As you can see, the choices are less than ideal.

The improvement

Since Docker CE 18.03, however, you can have the best of both worlds. Using the new --template-driver option to docker config create and docker secret create, you can now use the Golang templating language to insert secret references as well as other templating placeholders directly in your Swarm configs and have them rendered only when each service task is created.

The possibilities listed in the relevant pull request on the Moby project are:

{{ env "VAR" }}
{{ .Task.ID }}
{{ secret "sometarget" }}¹

I’ve discovered that you can also use {{ config "sometarget" }}, though I’d be careful with introducing obscure dependencies between different Swarm configs.

Adding documentation is tracked in this GitHub issue.

Example

Here’s a step-by-step example to illustrate the feature:

Create a Swarm secret with the intended password for Redis:

$ openssl rand -hex 32 | tr -d '\n' | docker secret create redis_pw -
s3agzeg68rvf2nhimk20mjm05

Create a redis.conf file e.g. using the default Redis config,
Change the TCP port in redis.conf to show we’re changing the defaults, something we’d like to be able to see as an operator:
```
port 2100
```
Set the requirepass option in redis.conf as a templated secret reference, the value of which we’d like to avoid be able to read during daily operations:
```
requirepass {{ secret "redis_pw" }}
```

Create a Swarm config for redis.conf using the new --template-driver option, and inspect its contents to show that the redis_pw secret is not revealed:

$ docker config create --template-driver golang redis.conf ./redis.conf
$ docker config inspect --pretty redis.conf | grep '^requirepass'
requirepass {{ secret "redis_pw" }}

Create an attachable overlay network and a Swarm service using the official Redis image:

$ docker network create --driver overlay --attachable redis_network
$ docker service create \
  --name redis \
  --secret redis_pw \
  --network redis_network \
  --config source=redis.conf,target=/usr/local/etc/redis/redis.conf \
  redis:alpine \
  redis-server /usr/local/etc/redis/redis.conf

Try running a Redis CLI against the Redis service, and you should be rejected:

$ docker run --rm --network redis_network redis:alpine redis-cli -h redis -p 2100 set x "I'm in"
NOAUTH Authentication required.

Try running a CLI in a service with access to the secret:

$ docker service create \
  --detach \
  --name redis-cli-test \
  --secret redis_pw \
  --network redis_network \
  --restart-condition on-failure \
  redis:alpine \
  sh -c 'redis-cli -h redis -p 2100 -a "$(cat /run/secrets/redis_pw)" set x "I'"'"'m in"; redis-cli -h redis -p 2100 -a "$(cat /run/secrets/redis_pw)" get x'

Wait a couple of seconds - currently we have to specify --detach with --restart-condition on-failure since the synchronous docker service create command (without --detach) interprets a task in state Completed to be a failure, and will not return you to the command line.²
Inspect the logs to see it working, and you should get the following result:
```
$ docker service logs --raw redis-cli-test
OK
I'm in
```

The example says "some_target" because it is possible to specify a source and a target for a secret when adding it to a Swarm service like so: --secret source=prod_redis_pw,target=redis_pw, and you have to use the target name rather than the source name in the templated reference in your Swarm config. ↩
Note to self: I should create an issue in docker/cli for this. ↩