The DevOps Series Using Docker with Ansible

This article is the eighth in the DevOps series. This month, we shall learn to set up Docker in the host system and use it with Ansible.

Docker provides operating system level virtualisation in the form of containers. These containers allow you to run standalone applications in an isolated environment. The three important features of Docker containers are isolation, portability and repeatability. All along we have used Parabola GNU/Linux-libre as the host system, and executed Ansible scripts on target virtual machines (VM) such as CentOS and Ubuntu.

Docker containers are extremely lightweight and fast to launch. You can also specify the amount of resources that you need such as the CPU, memory and network. The Docker technology was launched in 2013, and released under the Apache 2.0 licence. It is implemented using the Go programming language. A number of frameworks have been built on top of Docker for managing these clusters of servers. The Apache Mesos project, Google’s Kubernetes, and the Docker Swarm project are popular examples. These are ideal for running stateless applications and help you to easily scale horizontally.

Setting it up

The Ansible version used on the host system (Parabola GNU/Linux-libre x86_64) is 2.3.0.0. Internet access should be available on the host system. The ansible/ folder contains the following file:

ansible/playbooks/configuration/docker.yml

Installation

The following playbook is used to install Docker on the host system:

---

- name: Setup Docker

hosts: localhost

gather_facts: true

become: true

tags: [setup]

tasks:

- name: Update the software package repository

pacman:

update_cache: yes

- name: Install dependencies

package:

name: “{{ item }}”

state: latest

with_items:

- python2-docker

- docker

- service:

name: docker

state: started

- name: Run the hello-world container

docker_container:

name: hello-world

image: library/hello-world

The Parabola package repository is updated before proceeding to install the dependencies. The python2-docker package is required for use with Ansible. Hence, it is installed along with the docker package. The Docker daemon service is then started and the library/hello-world container is fetched and executed. A sample invocation and execution of the above playbook is shown below:

$ ansible-playbook playbooks/configuration/docker.yml -K --tags=setup

SUDO password:

PLAY [Setup Docker] *****************************************

TASK [Gathering Facts] **************************************

ok: [localhost]

TASK [Update the software package repository] ***************

changed: [localhost]

TASK [Install dependencies] *********************************

ok: [localhost] => (item=python2-docker)

ok: [localhost] => (item=docker)

TASK [service] **********************************************

ok: [localhost]

TASK [Run the hello-world container] ************************

changed: [localhost]

PLAY RECAP **************************************************

localhost : ok=5 changed=2 unreachable=0 failed=0

With the verbose ‘-v’ option to ansible-playbook, you will see an entry for LogPath, such as /var/lib/docker/containers/<container-id>/<container-id>-json.log. In this log file, you will see the output of the execution of the hello-world container. This output is the same when you run the container manually as shown below:

$ sudo docker run hello-world

Hello from Docker!

This message shows that your installation appears to be working correctly.

To generate this message, Docker took the following steps:

1. The Docker client contacted the Docker daemon.

2. The Docker daemon pulled the hello-world image from the Docker Hub.

3. The Docker daemon created a new container from that image, which runs the executable that produces the output you are currently reading.

4. The Docker daemon streamed that output to the Docker client, which sent it to your terminal.

To try something more ambitious, you can run an Ubuntu container with:

$ docker run -it ubuntu bash

You can share images, automate workflows, and more with a free Docker ID at https://cloud.docker.com/.

For more examples and ideas, do visit https://docs.docker.com/engine/userguide/.

An example

A deep learning (DL) Docker project is available (https://github.com/floydhub/dl-docker) with support for frameworks, libraries and software tools. We can use Ansible to build the entire DL container from the source code of the tools. The base OS of the container is Ubuntu 14.04, and will include the following software packages:

• TensorFlow
• Caffe
• Theano
• Keras
• Lasagne
• Torch
• iPython/Jupyter Notebook
• Numpy
• SciPy
• Pandas
• Scikit Learn
• Matplotlib
• OpenCV

The playbook to build the DL Docker image is given below:

- name: Build the dl-docker image

hosts: localhost

gather_facts: true

become: true

tags: [deep-learning]

vars:

DL_BUILD_DIR: “/tmp/dl-docker”

DL_DOCKER_NAME: “floydhub/dl-docker”

tasks:

- name: Download dl-docker

git:

repo: https://github.com/saiprashanths/dl-docker.git

dest: “{{ DL_BUILD_DIR }}”

- name: Build image and with buildargs

docker_image:

path: “{{ DL_BUILD_DIR }}”

name: “{{ DL_DOCKER_NAME }}”

dockerfile: Dockerfile.cpu

buildargs:

tag: “{{ DL_DOCKER_NAME }}:cpu”

We first clone the deep learning Docker project sources. The docker_image module in Ansible helps us to build, load and pull images. We then use the Dockerfile.cpu file to build a Docker image targeting the CPU. If you have a GPU in your system, you can use the Dockerfile.gpu file. The above playbook can be invoked using the following command:

$ ansible-playbook playbooks/configuration/docker.yml -K --tags=deep-learning

Depending on the CPU and RAM you have, it will take a considerable amount of time to build the image with all the software. So be patient!

Jupyter Notebook

The built dl-docker image contains Jupyter Notebook, which can be launched when you start the container. An Ansible playbook for the same is provided below:

- name: Start Jupyter notebook

hosts: localhost

gather_facts: true

become: true

tags: [notebook]

vars:

DL_DOCKER_NAME: “floydhub/dl-docker”

tasks:

- name: Run container for Jupyter notebook

docker_container:

name: “dl-docker-notebook”

image: “{{ DL_DOCKER_NAME }}:cpu”

state: started

command: sh run_jupyter.sh

You can invoke the playbook using the following command:

$ ansible-playbook playbooks/configuration/docker.yml -K --tags=notebook

The Dockerfile already exposes the port 8888, and hence you do not need to specify the same in the above docker_container configuration. After you run the playbook, using the ‘docker ps’ command on the host system, you can obtain the container ID as indicated below:

$ sudo docker ps

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES

a876ad5af751 floydhub/dl-docker:cpu “sh run_jupyter.sh” 11 minutes ago Up 4 minutes 6006/tcp, 8888/tcp dl-docker-notebook

You can now log in to the running container using the following command:

$ sudo docker exec -it a876 /bin/bash

You can then run an ‘ifconfig’ command to find the local IP address (‘172.17.0.2’ in this case), and then open http://172.17.0.2:8888 in a browser on your host system to see the Jupyter Notebook. A screenshot is shown in Figure 1.

TensorBoard

TensorBoard consists of a suite of visualisation tools to understand the TensorFlow programs. It is installed and available inside the Docker container. After you log in to the Docker container, at the root prompt, you can start TensorBoard by passing it a log directory as shown below:

# tensorboard --logdir=./log

You can then open http://172.17.0.2:6006/ in a browser on your host system to see the TensorBoard dashboard as shown in Figure 2.

Docker image facts

The docker_image_facts Ansible module provides useful information about a Docker image. We can use it to obtain the image facts for our dl-docker container as shown below:

- name: Get Docker image facts

hosts: localhost

gather_facts: true

become: true

tags: [facts]

vars:

DL_DOCKER_NAME: “floydhub/dl-docker”

tasks:

- name: Get image facts

docker_image_facts:

name: “{{ DL_DOCKER_NAME }}:cpu”

The above playbook can be invoked as follows:

$ ANSIBLE_STDOUT_CALLBACK=json ansible-playbook playbooks/configuration/docker.yml -K --tags=facts

The ANSIBLE_STDOUT_CALLBACK environment variable is set to ‘json’ to produce a JSON output for readability. Some important image facts from the invocation of the above playbook are shown below:

“Architecture”: “amd64”,

“Author”: “Sai Soundararaj <saip@outlook.com>”,

“Config”: {

“Cmd”: [

“/bin/bash”

],

“Env”: [

“PATH=/root/torch/install/bin:/root/caffe/build/tools:/root/caffe/python:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin”,

“CAFFE_ROOT=/root/caffe”,

“PYCAFFE_ROOT=/root/caffe/python”,

“PYTHONPATH=/root/caffe/python:”,

“LUA_PATH=/root/.luarocks/share/lua/5.1/?.lua;/root/.luarocks/share/lua/5.1/?/init.lua;/root/torch/install/share/lua/5.1/?.lua;/root/torch/install/share/lua/5.1/?/init.lua;./?.lua;/root/torch/install/share/luajit-2.1.0-beta1/?.lua;/usr/local/share/lua/5.1/?.lua;/usr/local/share/lua/5.1/?/init.lua”,

“LUA_CPATH=/root/torch/install/lib/?.so;/root/.luarocks/lib/lua/5.1/?.so;/root/torch/install/lib/lua/5.1/?.so;./?.so;/usr/local/lib/lua/5.1/?.so;/usr/local/lib/lua/5.1/loadall.so”,

“LD_LIBRARY_PATH=/root/torch/install/lib:”,

“DYLD_LIBRARY_PATH=/root/torch/install/lib:”

],

“ExposedPorts”: {

“6006/tcp”: {},

“8888/tcp”: {}

},

“Created”: “2016-06-13T18:13:17.247218209Z”,

“DockerVersion”: “1.11.1”,

“Os”: “linux”,

“task”: { “name”: “Get image facts” }

You are encouraged to read the ‘Getting Started with Docker’ user guide available at http://docs.ansible.com/ansible/latest/guide_docker.html to know more about using Docker with Ansible.