DevOps Series Ansible Deployment of Nginx to Serve Static Files and to Use Goaccess for Log Analysis

1
6909

This is the 15th article in the DevOps series, in which the author demonstrates how an Nginx server can be set up to serve static file content (HTML, CSS, JavaScript), and to use Goaccess for log analysis.

Nginx is a free and open source Web server written in C by Igor Sysoev. It is designed to handle thousands of client connections. It is also popularly used as a load balancer. It does not require much memory and can also be used as an HTTP cache server or a mail proxy server. It was released in 2004 and uses a BSD-like licence.

Goaccess is also free and open source software. It is a real-time Web log analyser written in C by Gerardo Orellana. You can run it remotely on a *nix terminal or access it through a browser. It requires only ncurses as a dependency when used from a terminal. It supports a number of Web log formats such as Apache, Nginx, Elastic load balancing, etc. You can also use a terminal dashboard with it and export reports to HTML. It was first released in 2010 and uses the MIT licence.

GNU/Linux

A Debian 9 (x86_64) guest virtual machine (VM) instance using KVM/QEMU is chosen to install Nginx.

The host system is a Parabola GNU/ Linux-libre x86_64 system and Ansible is installed on it using the distribution package manager. The version of Ansible used is 2.5.0 as indicated below:

$ ansible --version

ansible 2.5.0

config file = /etc/ansible/ansible.cfg

configured module search path = [u’/home/guest/.ansible/plugins/modules’, u’/usr/share/ansible/plugins/modules’]

ansible python module location = /usr/lib/python2.7/site-packages/ansible

executable location = /usr/bin/ansible

python version = 2.7.14 (default, Jan 5 2018, 10:41:29) [GCC 7.2.1 20171224]

You should add an entry to the /etc/hosts file for the guest ‘debian’ VM as follows:

192.168.122.140 debian

On the host system, we will create a project directory structure to store the Ansible playbooks, the inventory and static Web files:

ansible/inventory/kvm/

/playbooks/configuration/

/files/

The inventory/kvm/inventory file contains the following:

debian ansible_host=192.168.122.140 ansible_connection=ssh ansible_user=debian

The ansible/files/ directory has the following:

ansible/files/_site

/example.domain

/goaccessrc

The default Debian 9 installation does not install the sudo package. Start the new Debian VM, log in as the root user, and install the sudo package. You should also provide sudo access to the user account, which is ‘debian’ (in this example).

root@debian:~# apt-get install sudo

root@debian:~# adduser debian sudo

Adding user `debian' to group `sudo'...

Adding user debian to group sudo

Done.

You should now be able to issue commands from the host system to the guest OS, using Ansible. For example:

$ ansible -i inventory/kvm/inventory debian -m ping

debian | SUCCESS => {

"changed": false,

"ping": "pong"

}
Figure 1: Nginx default home page

Nginx

The first step is to install the Nginx Web server. The software package repository is updated, and then the Nginx package is installed. The Nginx Web server is started and we wait for the server to listen on port 80. The playbook to install Nginx is as follows:

---

- name: Install nginx

hosts: debian

become: yes

become_method: sudo

gather_facts: true

tags: [nginx]

tasks:

- name: Update the software package repository

apt:

update_cache: yes

- name: Install nginx

package:

name: “{{ item }}”

state: latest

with_items:

- nginx

- name: Start nginx

service:

name: nginx

state: started

- wait_for:

port: 80

The above playbook can be invoked using the command shown below:

$ ansible-playbook -i inventory/kvm/inventory playbooks/configuration/nginx.yml --tags nginx -vv -K

The -vv represents the verbosity in the Ansible output. You can use up to four v’s for a more detailed output. The -K option prompts for the sudo password for the debian user account. You can now open a Web browser on the local host with the URL http://192.168.122.140 and you should see the default Nginx home page as shown in Figure 1.

Serving static files

You can use any static site generator to create your HTML, CSS and JavaScript files. You will need to copy the generated files to the files/_site directory. The Ansible playbook to copy the static files to the Nginx Web server location is given below:

- name: Copy static files

hosts: debian

become: yes

become_method: sudo

gather_facts: true

tags: [static]

tasks:

- name: Stop nginx server

service:

name: nginx

state: stopped

- name: Remove existing directory

shell: /bin/rm -rf /var/www/html/home/*

- name: Home directory should exist

file:

path: /var/www/html/home

state: directory

mode: 0755

- name: Copy latest files

copy:

src: ../../files/_site/

dest: /var/www/html/home/

directory_mode: yes

- name: Copy example.domain nginx file

copy:

src: ../../files/example.domain

dest: /etc/nginx/sites-available/

- name: Create link file

file:

src: /etc/nginx/sites-available/example.domain

dest: /etc/nginx/sites-enabled/example.domain

owner: root

group: root

force: yes

state: link

- name: Start nginx

service:

name: nginx

state: started

- wait_for:

port: 80
Figure 2: Goaccess dashboard

The Nginx Web server is first stopped, as we are changing the Web content. If you have multiple instances of the Web server running, you can run the playbook in a rolling upgrade fashion. In this way, if client requests occur during the upgrade, you will not have a downtime. In this playbook, we first remove the contents of the old directory, and then copy the new contents. Depending on the volume of content, this may or may not suit your requirements. You can also use the synchronise module in Ansible (http://docs.ansible.com/ansible/latest/modules/synchronize_module.html). This uses rsync to synchronise files between your local host and the remote server.The example.domain is the Nginx configuration file for the website. As per the Nginx configuration, it is copied to the /etc/nginx/sites-available directory, and a symbolic link file to it is created in the /etc/nginx/sites-enabled directory. The content of the example.domain is as follows:

server {

listen 80;

listen [::]:80;

server_name example.domain www.example.domain;

root /var/www/html;

# Add index.php to the list if you are using PHP

index index.html index.htm index.nginx-debian.html;

location / {

return 301 $scheme://www.example.domain/home$request_uri;

}

location /home {

try_files $uri $uri/ =404;

}

}

The Nginx Web server is then started to serve the newly copied content. The Ansible playbook for copying and serving the static files can be invoked as follows:

$ ansible-playbook -i inventory/kvm/inventory playbooks/configuration/nginx.yml --tags static -vv -K

You can now refresh the http://192.168.122.140 page on your local host to see your static Web content.

Goaccess

The Goaccess interactive, real-time Web log analyser is available in the Debian package repository. After updating the software APT repository, Goaccess is installed. The configuration format to parse the Nginx logs is made available in the goaccessrc file. Its contents are as follows:

log-format %h %^[%d:%t %^] “%r” %s %b “%R” “%u” %T %^

time-format %H:%M:%S

date-format %d/%b/%Y

The above code is copied to ~/.goaccessrc. The Ansible playbook to install Goaccess is given below:

- name: Install goaccess

hosts: debian

become: yes

become_method: sudo

gather_facts: true

tags: [goaccess]

tasks:

- name: Update the software package repository

apt:

update_cache: yes

- name: Install goaccess

package:

name: “{{ item }}”

state: latest

with_items:

- goaccess

- name: Copy goaccessrc

copy:

src: ../../files/goaccessrc

dest: “/home/{{ ansible_ssh_user }}/.goaccessrc”

The above playbook’s invocation and sample output is shown below for reference:

$ ansible-playbook -i inventory/kvm/inventory playbooks/configuration/nginx.yml --tags goaccess -vv -K

ansible-playbook 2.5.0

config file = /etc/ansible/ansible.cfg

configured module search path = [u’/home/guest/.ansible/plugins/modules’, u’/usr/share/ansible/plugins/modules’]

ansible python module location = /usr/lib/python2.7/site-packages/ansible

executable location = /usr/bin/ansible-playbook

python version = 2.7.14 (default, Jan 5 2018, 10:41:29) [GCC 7.2.1 20171224]

Using /etc/ansible/ansible.cfg as config file

SUDO password:

PLAYBOOK: nginx.yml ******************************************************************************************************

3 plays in playbooks/configuration/nginx.yml

PLAY [Install nginx] *****************************************************************************************************

TASK [Gathering Facts] *************************************************************************************************

task path: /home/guest/ansible/playbooks/configuration/nginx.yml:2

ok: [debian]

META: ran handlers

META: ran handlers

META: ran handlers

PLAY [Copy static files] **************************************************************************************************

TASK [Gathering Facts] ***************************************************************************************************task path: /home/guest/ansible/playbooks/configuration/nginx.yml:29

ok: [debian]

META: ran handlers

META: ran handlers

META: ran handlers

PLAY [Install goaccess] **************************************************************************************************

TASK [Gathering Facts] ***************************************************************************************************

task path: /home/guest/ansible/playbooks/configuration/nginx.yml:79

ok: [debian]

META: ran handlers

TASK [Update the software package repository] ****************************************************************************

task path: /home/guest/ansible/playbooks/configuration/nginx.yml:87

changed: [debian] => {“cache_update_time”: 1523274608, “cache_updated”: true, “changed”: true}

TASK [Install goaccess] **************************************************************************************************

task path: /home/guest/ansible/playbooks/configuration/nginx.yml:91

ok: [debian] => (item=goaccess) => {“cache_update_time”: 1523274608, “cache_updated”: false, “changed”: false, “item”: “goaccess”}

TASK [Copy goaccessrc] ***************************************************************************************************

task path: /home/guest/ansible/playbooks/configuration/nginx.yml:98

changed: [debian] => {“changed”: true, “checksum”: “b8981bb97a7727d8fbd7d92cd1730b9cac19a2b0”, “dest”: “/home/debian/.goaccessrc”, “gid”: 0, “group”: “root”, “md5sum”: “f85cd220742cc6735e74327388744f3d”, “mode”: “0644”, “owner”: “root”, “size”: 98, “src”: “/home/debian/.ansible/tmp/ansible-tmp-1523274615.53-165679868036633/source”, “state”: “file”, “uid”: 0}

META: ran handlers

META: ran handlers

PLAY RECAP ***************************************************************************************************************

debian : ok=6 changed=2 unreachable=0 failed=0

You can now log in to the guest VM and run Goaccess from the terminal for the Nginx access.log file as follows:

$ sudo goaccess -f /var/log/nginx/access.log -p ~/.goaccessrc -a

You will see the Goaccess dashboard as illustrated in  Figure 2.

You can also export the dashboard to an HTML report

that you can view in a browser:

$ sudo goaccess -f /var/log/nginx/access.log -p ~/.goaccessrc -a > report.html
Figure 3: Goaccess HTML report

A sample screenshot of the HTML report is shown in Figure 3.

You are encouraged to read the Goaccess manual page at https://goaccess.io/man to learn more about its usage.

1 COMMENT

LEAVE A REPLY

Please enter your comment!
Please enter your name here