📦 Docker container autonomy

I have a home server that I run off of a Synology 920+ (a great choice for a home NAS and docker container host). On it, I run a number of docker containers that host many services I use on my home network, things like Flame as a home dashboard, PiHole, Snippet Box, Jupyer Notebooks, the *arr stack, along with a number of other services.

Whilst I have a lot of services, keeping them up to date can be a bit of a nuisance, with manually checking, manually updating, manually restarting. The key here is things are manual. Let me show how I try to do all of this automatically instead.

My main 3 requirements are:

Reduce container downtime - I want to reduce downtime so I have all my services available on demand, and when a container isn’t working right, fix it automatically.
Keep containers up to date - I want to be running the latest stable container all the time, without me having to manually complete all the necessary steps to do so.
Notify me when things change - Keep me updated when containers are updated, so that I know when they happened, but importantly can monitor for any issues, and rollback a container if necessary

The automation magic

I use 3 main containers to help me keep all of my other containers up to date. I have listed them below, and their uses.

Autoheal - Monitor and restart unhealthy docker containers.
Watchtower - A process for automating Docker container base image updates.
Diun - Notifying me via Pushover that my containers have been updated.

Autoheal

I have set Autoheal to watch all containers using an environment variable. If the container has a healthcheck on it, Autoheal will restart the container if it starts to become unhealthy. A container restart will solve ~99% of issues.

Note: Autoheal will not heal any containers that are linked to it, so its best to apply individual healthchecks to all containers.

To set a health status on the container you can add the healthcheck key to a docker compose service. The one in the example below will check if the service has internet connectivity, and if it does not, will show as unhealthy.

  service_name:
    image: "containous/whoami"
    container_name: whoami
    healthcheck:
      test: "curl -f api.ipify.org || exit 1"
      interval: 60s
      timeout: 30s
      retries: 3
      start_period: 30s

Watchtower

I have Watchtower set to update any container which has a specific label on it. This stops Watchtower updating any random containers I might have running. Watchtower will pull the latest image, as well as restarting the affected container with the new image. Finally if there are any containers depending on the updated container, it will restart those too.

To set a label on the container you can add the label key to a docker compose service, as per the example below.

  service_name:
    image: "containous/whoami"
    container_name: whoami
    labels:
      - "com.centurylinklabs.watchtower.enable=true"

Diun

I have set Diun to monitor all my containers every 6 hours and send me a notification via Pushover, when a new one has been created or updated. Diun can be configured in many ways, so its best to read the documentation to get the desired functionality.

Compose File

The following the docker compose file that can be started using docker-compose up -d. Before starting you will need to update some of the PUSHOVER variables, and the /data volume mappings in the duin service, to match your system/needs. As you can see all these services share the hosts docker socket file with the container, so the service can act on the hosts docker container status updates.

version: '3'

services:
  autoheal:
    container_name: autoheal
    image: willfarrell/autoheal:latest
    restart: always
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
    environment:
      - AUTOHEAL_CONTAINER_LABEL=all    # Heal all containers that look unhealthy

  diun:
    container_name: diun
    image: crazymax/diun:latest
    restart: always
    command: serve
    volumes:
      - ./data:/data
      - /var/run/docker.sock:/var/run/docker.sock:ro
    environment:
      # Standard configuration from https://crazymax.dev/diun/usage/basic-example/
      - TZ=Europe/London
      - LOG_LEVEL=info
      - LOG_JSON=false
      - DIUN_WATCH_WORKERS=20
      - DIUN_WATCH_SCHEDULE=0 */6 * * *
      - DIUN_PROVIDERS_DOCKER=true
      - DIUN_PROVIDERS_DOCKER_WATCHBYDEFAULT=true
      # Pushover specific configuration from https://crazymax.dev/diun/notif/pushover/
      - DIUN_NOTIF_PUSHOVER_TOKEN=<<APP API TOKEN>>
      - DIUN_NOTIF_PUSHOVER_RECIPIENT=<<USER API TOKEN>>
      - DIUN_NOTIF_PUSHOVER_TEMPLATETITLE=DOCKER > {{ .Entry.Image.Path }} was {{ if (eq .Entry.Status "new") }}created{{ else }}updated{{ end }}
      - DIUN_NOTIF_PUSHOVER_TEMPLATEBODY=Image **{{ .Entry.Image.Path }}** {{ if (eq .Entry.Status "new") }}was **created**{{ else }}was **updated**{{ end }} at {{ .Entry.Manifest.Created.Format "02 Jan 2006 15:04:05" }}

  watchtower:
    container_name: watchtower
    image: containrrr/watchtower:latest
    restart: always
    # Command line arguments https://containrrr.dev/watchtower/arguments/
    # The following will update only containers with the label on them at 4am every day and remove the old images.
    command: --cleanup --schedule "0 0 4 * * *" --label-enable
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock

Conclusion

At this moment the solution I have documented works for me and my use cases. I am aware there are probably other (probably better) solutions out there, but some of these utilise more advanced services such as Kubernetes, which I don’t understand at the moment. If you do have any other other ideas about how I can improve this further, feel free to email me.