Setting up a rootless Docker server (with Ansible) and deploying an application.

uncategorized 14 minutes read (About 2103 words)

As a web software developer, you will often face problems like setting up various servers, be it web servers like Apache, nginx or Litespeed, or the working environment for a scripting language like PHP and Node.js. This setup brings a lot of compatibility and security issues with it. In the recent years, Docker has established itself as the “to-go” solution, when setting up a local or remote working environment. While on the local developer’s machine, it’s as easy as doing brew install docker or apt install docker, when it comes to creating a real live production server, it’s not that simple. Usually production servers should not be running any software as a root (administrator) user, to avoid security issues, and this is how Docker is started by default - with full system privileges. As you can imagine, having an insecure application in an enterprise environment is not desirable. Now there is a setup for Docker called “rootless mode”, that allows running the software with another user, that has restricted privileges. I will be showing how to install Docker + Docker Compose using terminal commands and automate the process, using Ansible - a server configuration tool.

About operating systems

I will be installing the rootless Docker on Ubuntu 20.04 “Focal”, but you can adapt this to pretty much any OS with minimal changes. However, if you’re thinking of doing an installation on CentOS 7.x, please don’t do this to yourself - I’ve tried to do it and spent countless hours fixing the outdated and broken OS and its package repositories and dependencies. At the end you will not be able to fully use the latest Docker, because CentOS 7 uses Linux core 3 and Docker has some features that are only supported in Linux core 5. You might have better luck with CentOS 8.x or 9.x, but I’ve not tested these versions.

Rootless docker installation

I will be writing the terminal commands and showing the Ansible configuration in parallel, so you, as readers, can choose what ot use.

Ansible setup

If you don’t want to deal with Ansible and just want to get the Docker installation steps, you can skip this section.

1
2
3
4
5
6
7
8
9
10
# Install the Anbisle tool and sshpass (required for remote SSH connections by Ansible)
sudo apt install ansible sshpass python3-pip
# Since Ansible is a Python application, we will also need some python libraries
sudo pip3 install docker docker-compose jsondiff
# (Optional) - if you install docker, do not install docker-py or you will get an error, as this may cause package corruption and is not recommended. Install docker-py only if you need Python 2.6 support.
#sudo pip3 install docker-py

# Install a package for working with the Docker API
ansible-galaxy collection install community.docker
ansible-galaxy collection install ansible.posix

If you’re using an old OS, Like Ubuntu 20.04 Focal, the Ansible version that will be installed (2.9.6) is old and buggy. You will have to take a few extra actions to get up-to-date or your run books might fail with unexplicable errors (like I did).

(Optional) Getting the latest version of Ansible

  1. Remove the old version of Ansible
1
2
sudo apt remove ansible
sudo apt --purge autoremove
  1. Prepare the package repository for the new Ansible
1
2
3
sudo apt -y install software-properties-common
sudo apt-add-repository ppa:ansible/ansible
sudo apt update
  1. Install Ansible
1
sudo apt install ansible

Set up behind proxy

Both Docker and Ansible support the default Linux proxy settings, using http_proxy and https_proxy environmental variables, so if you find youself behind a corporate proxy, just set it like this:

1
2
export http_proxy="http://<my_proxy_url>:9000/"
export https_proxy="http://<my_https_proxy_url>:9000/"

Setting up the environment

Install Docker dependencies

1
sudo apt install dbus-user-session uidmap slirp4netns python3-pip

Add a docker user group (needed later to assign to the user)

1
2
# https://linux.die.net/man/8/groupadd
sudo groupadd -g 3000 docker

Add a docker user

1
2
# https://linux.die.net/man/8/useradd
sudo useradd -m -g -p "<your_encoded_linux_user_password_here>" -u 3000 docker docker

(Optional) Enable login session lingering, to avoid Docker being stopped on user logout

1
sudo touch /var/lib/systemd/linger/docker

Full Ansible file

docker-environment.yml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
# Prepare the environment for rootless Docker: https://docs.docker.com/engine/security/rootless/
- name: Pre-requisites for installing and running Docker on Ubuntu
  hosts: all
  become: true

  tasks:
    - name: Install requirement for docker on ubuntu - dbus-user-session
      apt:
        name: dbus-user-session
        state: present
        update_cache: yes

        - name: Install required newuidmap and newgidmap (uidmap package)
      apt:
        name: uidmap
        state: present
        update_cache: yes

    - name: Ensure group ""{{ docker_usergroup }}"" exists with correct gid
      group:
        name: docker
        state: present
        gid: 3000

    # https://docs.ansible.com/ansible/latest/collections/ansible/builtin/user_module.html
    - name: Add docker user - "{{ docker_username }}"
      user:
        name: "{{ docker_username }}"
        # Set specific ID for the user
        uid: 3000
        shell: /bin/bash
        group: "{{ docker_usergroup }}"
        # no - Only add the user to the group Docker and remove them from any other group
        # yes - Add the user to the specified groups without removing them from other groups
        append: no
        # https://docs.ansible.com/ansible/latest/reference_appendices/faq.html#how-do-i-generate-encrypted-passwords-for-the-user-module
        password: "<your_encoded_linux_user_password_here>"

    # Systemd is killing all processes after SSH is disconnected, so we have to enable session lingering:
    # This one: https://serverfault.com/questions/463366/does-getting-disconnected-from-an-ssh-session-kill-your-programs
    # This one: https://github.com/systemd/systemd/issues/8486
    - name: "Check if {{ docker_username }} lingers"
      stat: "path=/var/lib/systemd/linger/{{ docker_username }}"
      register: linger

    # https://docs.ansible.com/ansible/latest/collections/ansible/builtin/command_module.html
    - name: "Enable linger for {{ docker_username }}"
      command: "loginctl enable-linger {{ docker_username }}"
      when: not linger.stat.exists

    # https://packages.ubuntu.com/search?keywords=prometheus-node-exporter
    # - name: Install node exporter, for monitoring
    #   apt:
    #     name: prometheus-node-exporter
    #     state: present
    #     update_cache: yes

    - name: Install pip, needed for working with Docker
      apt:
        name: python3-pip
        state: present
        update_cache: yes

    - name: Install docker python package
      pip:
        name: docker

    # Without this, commands like wget throw an error (wget error: bad address)
    - name: Needed for docker networking
      apt:
        name: slirp4netns
        state: present
        update_cache: yes
Run Ansible with this configuration file
1
ansible-playbook docker-environment.yml -i hosts.ini -l <server_group> --ask-pass --ask-become-pass

Downloading rootless Docker and docker-compose

Manual installation

For this step, it’s better if you first login with the “docker” user that we just created and go on from there.

1
2
3
4
5
6
7
8
9
10
# Move to the home directory
cd ~/

# Download and install Docker
curl -fsSL https://get.docker.com/rootless -o dockerRootless.sh
sh ./dockerRootless.sh

# Download and install docker-compose
mkdir -o ~/.docker/cli-plugins
curl -fsSL https://github.com/docker/compose/releases/download/v2.0.0/docker-compose-linux-amd64 -o ~/.docker/cli-plugins

Full Ansible file

docker-install.yml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
# Install and run rootless Docker: https://docs.docker.com/engine/security/rootless/
# https://docs.ansible.com/ansible/latest/user_guide/become.html
- name: Install Docker and Docker Compose on CentOS
  hosts: all
  remote_user: docker
  #become: yes
  #become_user: docker

  tasks:
    # Running rootless Docker - https://docs.docker.com/engine/security/rootless/
    - name: Download the Rootless Docker installer
      get_url:
        url: https://get.docker.com/rootless
        dest: ~/dockerRootless.sh
        mode: 0700
      # This is totally optional
      # environment:
      #  http_proxy: "http://<my_proxy_url>:9000/"
      #  https_proxy: "http://<my_proxy_url>:9000/"

    - name: Install Rootless Docker
      shell: ~/dockerRootless.sh

    # https://docs.ansible.com/ansible/latest/collections/ansible/builtin/file_module.html
    - name: Create a directory if it does not exist
      file:
        path: ~/.docker/cli-plugins/
        state: directory
        mode: "0700"

    - name: Download Docker compose
      get_url:
        url: https://github.com/docker/compose/releases/download/v2.0.0/docker-compose-linux-amd64
        dest: ~/.docker/cli-plugins/docker-compose
        mode: 0700
      # This is totally optional
      # environment:
      #  http_proxy: "http://<my_proxy_url>:9000/"
      #  https_proxy: "http://<my_proxy_url>:9000/"
Run Ansible with this configuration file
1
ansible-playbook docker-install.yml -i hosts.ini -l <server_group> --ask-pass

And of course, here is how the hosts.ini file looks like, for those of you who want to use Ansible

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
; Use the name in the brackets, when referencing the server_group in the ansible-playbook command
[beta_nodes]
bme_beta ansible_connection=ssh ansible_ssh_extra_args='-o StrictHostKeyChecking=no' ansible_host=my.beta.host.domain.name

[beta_nodes:vars]
ansible_python_interpreter=/usr/bin/python3
docker_username=docker
docker_usergroup=docker
environment=beta

[stg_nodes]
bme_stg ansible_connection=ssh ansible_ssh_extra_args='-o StrictHostKeyChecking=no' ansible_host=my.staging.host.domain.name

[stg_nodes:vars]
ansible_python_interpreter=/usr/bin/python3
docker_username=docker
docker_usergroup=docker
environment=stg

Conclussion

That’s it - if you just wanted to see how to install rootless Docker, you’re good to go. If you want to see how to start an application in Docker, stay and read the next section.

Deploying the application in Docker

First, create your Dockerfile and docker-compose.yml files

1
2
3
4
5
FROM registry-jpe2.r-local.net/dockerhub/library/nginx:latest

COPY ./emptyfile.conf /etc/nginx/conf.d/default.conf
COPY ./emptyfile.conf /etc/nginx/snippets/ssl-list.conf
COPY ./nginx.conf /etc/nginx/nginx.conf
docker-compose.yml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
version: "3.7"

services:
  nginx:
    build:
      context: ./apps/nginx/
    ports:
      - 8080:8080
    restart: always
    networks:
      - web
    volumes:
      - ./apps/nginx/.pipeline/base/nginx/docker.conf:/var/nginxconf/upstream.conf:ro

  simple-web:
    image: yeasy/simple-web:latest
    scale: 4
    networks:
      - web

You may have noticed that we have two applications (services) in our docker-compose file. I will show why later.

Create the reverse proxy configuration for Nginx

Nginx reverse proxy configuration
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
worker_processes    auto;
pid                 /var/cache/nginx/nginx.pid;

events { }

http {
    include /var/nginxconf/upstream.conf;

    log_format my_format '$remote_addr - $remote_user [$time_local] '
                      '"$request" $status $body_bytes_sent '
                      '"$http_referer" "$http_user_agent" '
                      '$proxy_host $upstream_addr';

    server {
        listen 8080;

        access_log /var/log/nginx/access.log my_format;

        location / {
            proxy_pass $scheme://bmaas;
        }
    }
}
Nginx load balancer configuration
1
2
3
4
upstream bmaas {
    # Here we are listing the host names of the applications we want to load balance.
    server simple-web:80 max_fails=3 fail_timeout=60s;
}

Starting up our application

1
docker compose up -d

Zero-downtime updates

Now it’s time to see why did we need Nginx alongside our application. While we may make this work with only the application build, most of the time you will want to be able to update the application and deploy changes without interrupting your users. To do this, we need a way to simultaneously run the new version and the old version of the application, and stop the old version only after no users use it anymore. Here is one way to do this.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
reload_nginx() {
  docker exec nginx-proxy /usr/sbin/nginx -s reload
}

zero_downtime_deploy() {
  # The name of the application we will be updating
  service_name=simple-web
  old_container_id=$(docker ps -f name=$service_name -q | tail -n1)

  # bring a new container online, running new code
  # (nginx continues routing to the old container only)
  docker-compose up -d --no-deps --scale $service_name=2 --no-recreate $service_name

  # wait for new container to be available
  new_container_id=$(docker ps -f name=$service_name -q | head -n1)
  new_container_ip=$(docker inspect -f '{{range.NetworkSettings.Networks}}{{.IPAddress}}{{end}}' $new_container_id)
  docker exec nginx-proxy curl --silent --include --retry-connrefused --retry 30 --retry-delay 1 --fail http://$new_container_ip:8082/ || exit 1

  # start routing requests to the new container (as well as the old)
  reload_nginx

  # take the old container offline
  docker stop $old_container_id
  docker rm $old_container_id

  docker-compose up -d --no-deps --scale $service_name=1 --no-recreate $service_name

  # stop routing requests to the old container
  reload_nginx
}

zero_downtime_deploy

The above shell script takes care of creating a new instance of our application, checking if it’s running correctly and updating Nginx, so it can start sending traffic to the correct application container.

Inspiration

https://docs.docker.com/engine/install/ubuntu/
https://docs.docker.com/engine/security/rootless/
https://github.com/docker/compose
https://docs.docker.com/compose/compose-file/compose-file-v3/
https://docs.docker.com/engine/install/linux-postinstall/
https://serverfault.com/questions/736452/ansible-how-to-run-one-task-host-by-host
https://serverfault.com/questions/750856/how-to-run-multiple-playbooks-in-order-with-ansible
https://linuxize.com/post/how-to-create-groups-in-linux/
https://www.cyberciti.biz/faq/ubuntu-add-user-to-group/
https://www.cyberciti.biz/faq/create-a-user-account-on-ubuntu-linux/
https://www.pluralsight.com/blog/tutorials/linux-add-user-command
https://docs.ansible.com/ansible/2.3/intro_inventory.html
https://www.linkedin.com/pulse/how-launch-manage-docker-through-ansible-playbook-abhishek-biswas/

#configuration #docker #docker-compose #ubuntu #rootless docker #production server #server setup #setting docker for production #ansible #docker configuration with ansible
Android: Could not find method implementation() for arguments Starting up with ElasticSearch on Linux or MacOS

Comments