Setting up a rootless Docker server (with Ansible) and deploying an application.

As a web software developer, you will often face problems like setting up various servers, be it web servers like Apache, nginx or Litespeed, or the working environment for a scripting language like PHP and Node.js. This setup brings a lot of compatibility and security issues with it. In the recent years, Docker has established itself as the “to-go” solution, when setting up a local or remote working environment. While on the local developer’s machine, it’s as easy as doing brew install docker or apt install docker, when it comes to creating a real live production server, it’s not that simple. Usually production servers should not be running any software as a root (administrator) user, to avoid security issues, and this is how Docker is started by default - with full system privileges. As you can imagine, having an insecure application in an enterprise environment is not desirable. Now there is a setup for Docker called “rootless mode”, that allows running the software with another user, that has restricted privileges. I will be showing how to install Docker + Docker Compose using terminal commands and automate the process, using Ansible - a server configuration tool.

About operating systems

I will be installing the rootless Docker on Ubuntu 20.04 “Focal”, but you can adapt this to pretty much any OS with minimal changes. However, if you’re thinking of doing an installation on CentOS 7.x, please don’t do this to yourself - I’ve tried to do it and spent countless hours fixing the outdated and broken OS and its package repositories and dependencies. At the end you will not be able to fully use the latest Docker, because CentOS 7 uses Linux core 3 and Docker has some features that are only supported in Linux core 5. You might have better luck with CentOS 8.x or 9.x, but I’ve not tested these versions.

Rootless docker installation

I will be writing the terminal commands and showing the Ansible configuration in parallel, so you, as readers, can choose what ot use.

Ansible setup

If you don’t want to deal with Ansible and just want to get the Docker installation steps, you can skip this section.

1
2
3
4
5
6
7
8
9
10
# Install the Anbisle tool and sshpass (required for remote SSH connections by Ansible)
sudo apt install ansible sshpass python3-pip
# Since Ansible is a Python application, we will also need some python libraries
sudo pip3 install docker docker-compose jsondiff
# (Optional) - if you install docker, do not install docker-py or you will get an error, as this may cause package corruption and is not recommended. Install docker-py only if you need Python 2.6 support.
#sudo pip3 install docker-py

# Install a package for working with the Docker API
ansible-galaxy collection install community.docker
ansible-galaxy collection install ansible.posix

If you’re using an old OS, Like Ubuntu 20.04 Focal, the Ansible version that will be installed (2.9.6) is old and buggy. You will have to take a few extra actions to get up-to-date or your run books might fail with unexplicable errors (like I did).

(Optional) Getting the latest version of Ansible

  1. Remove the old version of Ansible
1
2
sudo apt remove ansible
sudo apt --purge autoremove
  1. Prepare the package repository for the new Ansible
1
2
3
sudo apt -y install software-properties-common
sudo apt-add-repository ppa:ansible/ansible
sudo apt update
  1. Install Ansible
1
sudo apt install ansible

Set up behind proxy

Both Docker and Ansible support the default Linux proxy settings, using http_proxy and https_proxy environmental variables, so if you find youself behind a corporate proxy, just set it like this:

1
2
export http_proxy="http://<my_proxy_url>:9000/"
export https_proxy="http://<my_https_proxy_url>:9000/"

Setting up the environment

Install Docker dependencies

1
sudo apt install dbus-user-session uidmap slirp4netns python3-pip

Add a docker user group (needed later to assign to the user)

1
2
# https://linux.die.net/man/8/groupadd
sudo groupadd -g 3000 docker

Add a docker user

1
2
# https://linux.die.net/man/8/useradd
sudo useradd -m -g -p "<your_encoded_linux_user_password_here>" -u 3000 docker docker

(Optional) Enable login session lingering, to avoid Docker being stopped on user logout

1
sudo touch /var/lib/systemd/linger/docker

Full Ansible file

docker-environment.yml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
# Prepare the environment for rootless Docker: https://docs.docker.com/engine/security/rootless/
- name: Pre-requisites for installing and running Docker on Ubuntu
hosts: all
become: true

tasks:
- name: Install requirement for docker on ubuntu - dbus-user-session
apt:
name: dbus-user-session
state: present
update_cache: yes

- name: Install required newuidmap and newgidmap (uidmap package)
apt:
name: uidmap
state: present
update_cache: yes

- name: Ensure group ""{{ docker_usergroup }}"" exists with correct gid
group:
name: docker
state: present
gid: 3000

# https://docs.ansible.com/ansible/latest/collections/ansible/builtin/user_module.html
- name: Add docker user - "{{ docker_username }}"
user:
name: "{{ docker_username }}"
# Set specific ID for the user
uid: 3000
shell: /bin/bash
group: "{{ docker_usergroup }}"
# no - Only add the user to the group Docker and remove them from any other group
# yes - Add the user to the specified groups without removing them from other groups
append: no
# https://docs.ansible.com/ansible/latest/reference_appendices/faq.html#how-do-i-generate-encrypted-passwords-for-the-user-module
password: "<your_encoded_linux_user_password_here>"

# Systemd is killing all processes after SSH is disconnected, so we have to enable session lingering:
# This one: https://serverfault.com/questions/463366/does-getting-disconnected-from-an-ssh-session-kill-your-programs
# This one: https://github.com/systemd/systemd/issues/8486
- name: "Check if {{ docker_username }} lingers"
stat: "path=/var/lib/systemd/linger/{{ docker_username }}"
register: linger

# https://docs.ansible.com/ansible/latest/collections/ansible/builtin/command_module.html
- name: "Enable linger for {{ docker_username }}"
command: "loginctl enable-linger {{ docker_username }}"
when: not linger.stat.exists

# https://packages.ubuntu.com/search?keywords=prometheus-node-exporter
# - name: Install node exporter, for monitoring
# apt:
# name: prometheus-node-exporter
# state: present
# update_cache: yes

- name: Install pip, needed for working with Docker
apt:
name: python3-pip
state: present
update_cache: yes

- name: Install docker python package
pip:
name: docker

# Without this, commands like wget throw an error (wget error: bad address)
- name: Needed for docker networking
apt:
name: slirp4netns
state: present
update_cache: yes
Run Ansible with this configuration file
1
ansible-playbook docker-environment.yml -i hosts.ini -l <server_group> --ask-pass --ask-become-pass

Downloading rootless Docker and docker-compose

Manual installation

For this step, it’s better if you first login with the “docker” user that we just created and go on from there.

1
2
3
4
5
6
7
8
9
10
# Move to the home directory
cd ~/

# Download and install Docker
curl -fsSL https://get.docker.com/rootless -o dockerRootless.sh
sh ./dockerRootless.sh

# Download and install docker-compose
mkdir -o ~/.docker/cli-plugins
curl -fsSL https://github.com/docker/compose/releases/download/v2.0.0/docker-compose-linux-amd64 -o ~/.docker/cli-plugins

Full Ansible file

docker-install.yml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
# Install and run rootless Docker: https://docs.docker.com/engine/security/rootless/
# https://docs.ansible.com/ansible/latest/user_guide/become.html
- name: Install Docker and Docker Compose on CentOS
hosts: all
remote_user: docker
#become: yes
#become_user: docker

tasks:
# Running rootless Docker - https://docs.docker.com/engine/security/rootless/
- name: Download the Rootless Docker installer
get_url:
url: https://get.docker.com/rootless
dest: ~/dockerRootless.sh
mode: 0700
# This is totally optional
# environment:
# http_proxy: "http://<my_proxy_url>:9000/"
# https_proxy: "http://<my_proxy_url>:9000/"

- name: Install Rootless Docker
shell: ~/dockerRootless.sh

# https://docs.ansible.com/ansible/latest/collections/ansible/builtin/file_module.html
- name: Create a directory if it does not exist
file:
path: ~/.docker/cli-plugins/
state: directory
mode: "0700"

- name: Download Docker compose
get_url:
url: https://github.com/docker/compose/releases/download/v2.0.0/docker-compose-linux-amd64
dest: ~/.docker/cli-plugins/docker-compose
mode: 0700
# This is totally optional
# environment:
# http_proxy: "http://<my_proxy_url>:9000/"
# https_proxy: "http://<my_proxy_url>:9000/"
Run Ansible with this configuration file
1
ansible-playbook docker-install.yml -i hosts.ini -l <server_group> --ask-pass

And of course, here is how the hosts.ini file looks like, for those of you who want to use Ansible

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
; Use the name in the brackets, when referencing the server_group in the ansible-playbook command
[beta_nodes]
bme_beta ansible_connection=ssh ansible_ssh_extra_args='-o StrictHostKeyChecking=no' ansible_host=my.beta.host.domain.name

[beta_nodes:vars]
ansible_python_interpreter=/usr/bin/python3
docker_username=docker
docker_usergroup=docker
environment=beta

[stg_nodes]
bme_stg ansible_connection=ssh ansible_ssh_extra_args='-o StrictHostKeyChecking=no' ansible_host=my.staging.host.domain.name

[stg_nodes:vars]
ansible_python_interpreter=/usr/bin/python3
docker_username=docker
docker_usergroup=docker
environment=stg

Conclussion

That’s it - if you just wanted to see how to install rootless Docker, you’re good to go. If you want to see how to start an application in Docker, stay and read the next section.

Deploying the application in Docker

First, create your Dockerfile and docker-compose.yml files

1
2
3
4
5
FROM registry-jpe2.r-local.net/dockerhub/library/nginx:latest

COPY ./emptyfile.conf /etc/nginx/conf.d/default.conf
COPY ./emptyfile.conf /etc/nginx/snippets/ssl-list.conf
COPY ./nginx.conf /etc/nginx/nginx.conf
docker-compose.yml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
version: "3.7"

services:
nginx:
build:
context: ./apps/nginx/
ports:
- 8080:8080
restart: always
networks:
- web
volumes:
- ./apps/nginx/.pipeline/base/nginx/docker.conf:/var/nginxconf/upstream.conf:ro

simple-web:
image: yeasy/simple-web:latest
scale: 4
networks:
- web

You may have noticed that we have two applications (services) in our docker-compose file. I will show why later.

Create the reverse proxy configuration for Nginx

Nginx reverse proxy configuration
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
worker_processes    auto;
pid /var/cache/nginx/nginx.pid;

events { }

http {
include /var/nginxconf/upstream.conf;

log_format my_format '$remote_addr - $remote_user [$time_local] '
'"$request" $status $body_bytes_sent '
'"$http_referer" "$http_user_agent" '
'$proxy_host $upstream_addr';

server {
listen 8080;

access_log /var/log/nginx/access.log my_format;

location / {
proxy_pass $scheme://bmaas;
}
}
}
Nginx load balancer configuration
1
2
3
4
upstream bmaas {
# Here we are listing the host names of the applications we want to load balance.
server simple-web:80 max_fails=3 fail_timeout=60s;
}

Starting up our application

1
docker compose up -d

Zero-downtime updates

Now it’s time to see why did we need Nginx alongside our application. While we may make this work with only the application build, most of the time you will want to be able to update the application and deploy changes without interrupting your users. To do this, we need a way to simultaneously run the new version and the old version of the application, and stop the old version only after no users use it anymore. Here is one way to do this.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
reload_nginx() {
docker exec nginx-proxy /usr/sbin/nginx -s reload
}

zero_downtime_deploy() {
# The name of the application we will be updating
service_name=simple-web
old_container_id=$(docker ps -f name=$service_name -q | tail -n1)

# bring a new container online, running new code
# (nginx continues routing to the old container only)
docker-compose up -d --no-deps --scale $service_name=2 --no-recreate $service_name

# wait for new container to be available
new_container_id=$(docker ps -f name=$service_name -q | head -n1)
new_container_ip=$(docker inspect -f '{{range.NetworkSettings.Networks}}{{.IPAddress}}{{end}}' $new_container_id)
docker exec nginx-proxy curl --silent --include --retry-connrefused --retry 30 --retry-delay 1 --fail http://$new_container_ip:8082/ || exit 1

# start routing requests to the new container (as well as the old)
reload_nginx

# take the old container offline
docker stop $old_container_id
docker rm $old_container_id

docker-compose up -d --no-deps --scale $service_name=1 --no-recreate $service_name

# stop routing requests to the old container
reload_nginx
}

zero_downtime_deploy

The above shell script takes care of creating a new instance of our application, checking if it’s running correctly and updating Nginx, so it can start sending traffic to the correct application container.

Inspiration

https://docs.docker.com/engine/install/ubuntu/
https://docs.docker.com/engine/security/rootless/
https://github.com/docker/compose
https://docs.docker.com/compose/compose-file/compose-file-v3/
https://docs.docker.com/engine/install/linux-postinstall/
https://serverfault.com/questions/736452/ansible-how-to-run-one-task-host-by-host
https://serverfault.com/questions/750856/how-to-run-multiple-playbooks-in-order-with-ansible
https://linuxize.com/post/how-to-create-groups-in-linux/
https://www.cyberciti.biz/faq/ubuntu-add-user-to-group/
https://www.cyberciti.biz/faq/create-a-user-account-on-ubuntu-linux/
https://www.pluralsight.com/blog/tutorials/linux-add-user-command
https://docs.ansible.com/ansible/2.3/intro_inventory.html
https://www.linkedin.com/pulse/how-launch-manage-docker-through-ansible-playbook-abhishek-biswas/

Creating a Single Sign-On application in Okta

Comments