Orchestrating Secrets

By the end of this exercise, you should be able to:

  • Declare secrets in Swarm and Kubernetes
  • Provision secrets to a swarm service or Kubernetes deployment
  • Configure environment variables and application logic to consume secrets in either orchestrator

Prerequisites

  • A Swarm with at least one node (docker swarm init on any node with Docker installed will do if you don't already have a swarm running).
  • A Kubernetes cluster with at least one master and one worker (see the Kubernetes Basics demo in this book for setup instructions).

Creating Secrets

  1. Create a new secret named my-secret with the value abc1234 by using the following command to pipe STDIN to the secret value:

    PS: node-0 Administrator> echo 'abc1234' | docker secret create my-secret -

    Note this won't work on a node that isn't a swarm manager, since secrets get registered in swarm's state database.

  2. Alternatively, secret values can be read from a file. In the current directory create a file called password.txt and add the value my-pass to it. Create a secret with this value:

    PS: node-0 Administrator> docker secret create password ./password.txt

Managing Secrets

The Docker CLI provides API objects for managing secrets similar to all other Docker assets:

  1. List your current secrets:

    PS: node-0 Administrator> docker secret ls

  2. Print secret metadata:

    PS: node-0 Administrator> docker secret inspect password

  3. Delete a secret:

    PS: node-0 Administrator> docker secret rm my-secret

Using Secrets

Secrets are assigned to Swarm services upon creation of the service, and provisioned to containers for that service as they spin up.

  1. Create a service authorized to use the password secret:

    PS: node-0 Administrator> docker service create `
    --name demo `
    --secret password `
    microsoft/nanoserver ping 8.8.8.8 -t

  2. Use docker service ps demo to determine what node your service container is running on; connect to that node, and connect to the container (remember to use docker container ls to find the container ID):

    PS: node-x Administrator> docker container exec -it <container ID> powershell

  3. Inspect the secrets in this container where they are mounted by default, at /run/secrets:

    PS C:\> cd C:\ProgramData\Docker\secrets
PS C:\ProgramData\Docker\secrets> ls
PS C:\ProgramData\Docker\secrets> cat password
PS C:\ProgramData\Docker\secrets> exit

    This is the only place secret values sit unencrypted in memory.

Preparing an image for use of secrets

Containers need to consume secrets from their mountpoint, either C:\ProgramData\Docker\secrets by default, or a custom mount point if defined. In many cases, existing application logic expects secret values to appear behind environment variables; in the following, we set up such a situation as an example.

  1. Create a new directory image-secrets on node-0 and navigate to this folder. In this folder create a file named app.py and add the following content; this is a Python script that consumes a password from a file with a path specified by the environment variable PASSWORD_FILE:

    import os
print('***** Docker Secrets ******')
print('USERNAME: {0}'.format(os.environ['USERNAME']))

fname = os.environ['PASSWORD_FILE']
with open(fname) as f:
    content = f.readlines()

print('PASSWORD_FILE: {0}'.format(fname))
print('PASSWORD: {0}'.format(content[0]))

  2. Create a file called Dockerfile with the following content:

    FROM training/python-windows:3.6.1-nano
RUN mkdir -p /app
WORKDIR /app
COPY . /app
CMD python ./app.py; sleep 100000

  3. Build the image and push it to a registry so it's available to all nodes in your swarm:

    PS: node-0 image-secrets> docker image build -t <Docker ID>/secrets-demo:1.0 .
PS: node-0 image-secrets> docker image push <Docker ID>/secrets-demo:1.0

  4. Create and run a service using this image, and use the -e flag to create environment variables that point to your secrets:

    PS: node-0 image-secrets> docker service create `
    --name secrets-demo `
    --replicas=1 `
    --secret source=password,target=custom\path\password `
    -e USERNAME="jdoe" `
    -e PASSWORD_FILE="C:\ProgramData\Docker\secrets\custom\path\password" `
    <Docker ID>/secrets-demo:1.0

    The --secret flag parses a few tokens:

    • source indicates the name of the secret (defined when you did docker secret create...)
    • target indicates the path that the secret file will be mounted at relative to C:\ProgramData\Docker\secrets; in this case the secret will be available in a file C:\ProgramData\Docker\secretscustom\path\password.

  5. Figure out which node your container is running on, head over there, connect to the container, and run python app.py; the -e flag in service create has set environment variables to point at your secrets, allowing your app to find them where it expects.

Kubernetes Secrets

Secrets in Kubernetes are manipulated very similarly to Swarm; one interesting difference is the ability to package multiple values or files into the same secret. Below we reproduce the final example from the Swarm section above, but we'll pass in both the username and password in separate files contained in a single Kubernetes secret, rather than passing the username in directly as an environment variable.

  1. First we'll need a linux-appropriate version of your secrets-demo image you made for Windows above. In a directory kube-secret on kube-0, make a Dockerfile with the following content:

    FROM python:2.7
RUN mkdir -p /app
WORKDIR /app
COPY . /app
CMD python ./app.py && sleep 1000

    Also, make a file app.py in the same directory containing:

    import os
print '***** Docker Secrets ******'
print 'USERNAME: {0}'.format(os.environ['USERNAME'])

fname = os.environ['PASSWORD_FILE']
f = open(fname)
try:
    content = f.readlines()
finally:
    f.close()

print 'PASSWORD_FILE: {0}'.format(fname)
print 'PASSWORD: {0}'.format(content[0])

    Build and push this image:

    [centos@kube-0 kube-secret]$ docker image build \
    -t <Docker ID>/secrets-demo:linux-1.0 .
[centos@kube-0 kube-secret]$ docker image push <Docker ID>/secrets-demo:linux-1.0

  2. Place a username and password in files username and password:

    [centos@kube-0 kube-secret]$ echo "jdoe" > username
[centos@kube-0 kube-secret]$ echo "my-pass" > password

  3. Create a secret that captures both these files:

    [centos@kube-0 kube-secret]$ kubectl create secret generic user-pass \
    --from-file=./username --from-file=./password

    The generic keyword here indicates we're going to create the secret from a local file; user-pass will be the name of the secret we can refer to later; and the --from-file keys list the files we want to include in this secret.

  4. Create a pod definition in a file secretpod.yaml that uses this secret to map the username file contents directly onto an environment variable, and mount the password file in the container with a second environment variable pointing at its path:

    apiVersion: v1
kind: Pod
metadata:
  name: secretpod
spec:
  containers:
  - name: democon
    image: <Docker ID>/secrets-demo:linux-1.0
    env:
      - name: USERNAME
        valueFrom:
          secretKeyRef:
            name: user-pass
            key: username
      - name: PASSWORD_FILE
        value: "/custom/path/pass"
    volumeMounts:
    - name: passvol
      mountPath: "/custom/path"
      readOnly: true
  volumes:
  - name: passvol
    secret:
      secretName: user-pass
      items:
      - key: password
        path: pass
  restartPolicy: Never

  5. Spin the pod up, connect a bash shell to it, check that the environment variables are populated as you'd expect and that the python script works correctly:

    [centos@kube-0 kube-secret]$ kubectl create -f secretpod.yaml
[centos@kube-0 kube-secret]$ kubectl exec -it secretpod bash
root@secretpod:/app# echo $USERNAME
root@secretpod:/app# echo $PASSWORD_FILE
root@secretpod:/app# python app.py

  6. Look in /custom/path inside your running container, and notice that only password is present, mounted as pass. username wasn't mentioned in the items list for for the user-pass secret, and so wasn't mounted in the corresponding volume. In this way you can pick and choose which files from a single secret are mounted into a running container.

  7. Compare the config in secretpod.yaml to the last docker service create... command in the last section. Identify the corresponding pieces of syntax between the Swarm and Kubernetes declarations. Where are environment variables declared? How are secrets associated with the service or pod? How do you point an environment variable at a secret mounted in a file?

Conclusion

In this exercise we have learned how to create, inspect and use secrets in both Swarm and Kubernetes. As is often the case, Swarm's syntax is a bit shorter and simpler, but Kubernetes offers more flexibility and expressiveness, allowing the user to package multiple tokens in a single secret object, and selectively mount them in only the containers that need them.