Updating a Service

By the end of this exercise, you should be able to:

  • Update a swarm service's underlying image, controlling update parallelism, speed, and rollback contingencies

Creating Rolling Updates

  1. First, let's change one of our services a bit: open orchestration-workshop/dockercoins/worker/worker.py in your favorite text editor, and find the following section:

    def work_once():
        log.debug("Doing one unit of work")
        time.sleep(0.1)
    

    Change the 0.1 to a 0.01. Save the file, exit the text editor.

  2. Rebuild the worker image with a tag of <Docker ID>/dockercoins_worker:1.1, and push it to Docker Hub.

  3. Start the update, and wait for it to converge:

    [centos@node-0 ~]$ docker service update dc_worker \
        --image <Docker ID>/dockercoins_worker:1.1
    

    Tasks are updated to our new 1.1 image one at a time.

Parallelizing Updates

  1. We can also set our updates to run in batches by configuring some options associated with each service. Change the update parallelism to 2 and the delay to 5 seconds on the worker service by editing its definition in the docker-compose.yml:

    worker:
      image: <Docker ID>/dockercoins_worker:1.0
      networks:
      - dockercoins
      deploy:
        replicas: 10
        update_config:
          parallelism: 2
          delay: 5s
    
  2. Roll back the worker service to 1.0:

    [centos@node-0 ~]$ docker stack deploy -c=docker-compose.yml dc
    
  3. On node-1, watch your updates:

    [centos@node-1 ~]$ watch -n1 "docker service ps dc_worker \
        | grep -v Shutdown.*Shutdown"
    

    You should see two tasks get shutdown and restarted with the 1.0 image every five seconds.

Auto-Rollback Failed Updates

In the event of an application or container failure on deployment, we'd like to automatically roll the update back to the previous version.

  1. Update the worker service with some parameters to define rollback:

    [centos@node-0 ~]$ docker service update \
        --update-failure-action=rollback \
        --update-max-failure-ratio=0.2 \
        --update-monitor=20s \
        dc_worker
    

    These parameters will trigger a rollback if more than 20% of services tasks fail in the first 20 seconds after an update.

  2. Make a broken version of the worker service to trigger a rollback with; try removing all the import commands at the top of worker.py, for example. Then rebuild the worker image with a tag <Docker ID>/dockercoins_worker:bugged, push it to Docker Hub, and attempt to update your service:

    [centos@node-0 ~]$ docker image build -t <Docker ID>/dockercoins_worker:bugged .
    [centos@node-0 ~]$ docker image push <Docker ID>/dockercoins_worker:bugged
    [centos@node-0 ~]$ docker service update \
        dc_worker --image <Docker ID>/dockercoins_worker:bugged
    
  3. The connection to node-1 running watch should show the :bugged tag getting deployed, failing, and rolling back to :1.0 automatically over the next minute or two.

Shutting Down a Stack

  1. To shut down a running stack:

    [centos@node-0 ~]$ docker stack rm <stack name>
    

    Where the stack name can be found in the output of docker stack ls.

Conclusion

In this exercise, we explored deploying and redeploying an application as stacks and services. Note that relaunching a running stack updates all the objects it manages in the most non-disruptive way possible; there is usually no need to remove a stack before updating it. In production, rollback contingencies should always be used to cautiously upgrade images, cutting off potential damage before an entire service is taken down.