DevOps Series Deploying Graylog Using Ansible

By
Shakthi Kannan
-
0
9371

This 11th article in the DevOps series is a tutorial on installing Graylog software using Ansible.

Graylog is a free and open source log management software that allows you to store and analyse all your logs from a central location. It requires MongoDB (a document-oriented, NoSQL database) to store meta information and configuration information. The actual log messages are stored in Elasticsearch. It is written using the Java programming language and released under the GNU General Public License (GPL) v3.0.

Access control management is built into the software, and you can create roles and user accounts with different permissions. If you already have an LDAP server, its user accounts can be used with the Graylog software. It also provides a REST API, which allows you to fetch data to build your own dashboards. You can create alerts to take actions based on the log messages, and also forward the log data to other output streams. In this article, we will install the Graylog software and its dependencies using Ansible.

GNU/Linux

An Ubuntu 16.04.3 LTS guest virtual machine (VM) instance will be used to set up Graylog using KVM/QEMU. The host system is a Parabola GNU/Linux-libre x86_64 system. Ansible is installed on the host system using the distribution package manager. The version of Ansible used is:

$ ansible --version
 
ansible 2.4.1.0
 
config file = /etc/ansible/ansible.cfg
 
configured module search path = [u’/home/shakthi/.ansible/plugins/modules’, u’/usr/share/ansible/plugins/modules’]
 
ansible python module location = /usr/lib/python2.7/site-packages/ansible
 
executable location = /usr/bin/ansible
 
python version = 2.7.14 (default, Sep 20 2017, 01:25:59) [GCC 7.2.0]

Add an entry to the /etc/hosts file for the guest ‘ubuntu’ VM as indicated below:

192.168.122.25 ubuntu

On the host system, let’s create a project directory structure to store the Ansible playbooks:

ansible/inventory/kvm/
 
/playbooks/configuration/
 
/playbooks/admin/

An ‘inventory’ file is created inside the inventory/kvm folder that contains the following code:

ubuntu ansible_host=192.168.122.25 ansible_connection=ssh ansible_user=ubuntu ansible_password=password
Figure 1: Graylog login page
Figure 2: Graylog home page

You should be able to issue commands using Ansible to the guest OS. For example:

$ ansible -i inventory/kvm/inventory ubuntu -m ping
 
ubuntu | SUCCESS => {
 
“changed”: false,
 
“failed”: false,
 
“ping”: “pong”
 
}

Pre-requisites

The Graylog software has a few dependency packages that need to be installed as pre-requisites. The APT package repository is updated and upgraded before installing the pre-requisite software packages.

---
 
- name: Pre-requisites
 
hosts: ubuntu
 
become: yes
 
become_method: sudo
 
gather_facts: true
 
tags: [prerequisite]
 
tasks:
 
- name: Update the software package repository
 
apt:
 
update_cache: yes
 
- name: Update all the packages
 
apt:
 
upgrade: dist
 
- name: Install pre-requisite packages
 
package:
 
name: “{{ item }}”
 
state: latest
 
with_items:
 
- apt-transport-https
 
- openjdk-8-jre-headless
 
- uuid-runtime
 
- pwgen

The above playbook can be invoked as follows:

$ ansible-playbook -i inventory/kvm/inventory playbooks/configuration/graylog.yml --tags prerequisite -K

The ‘-K’ option prompts for the sudo password for the ‘ubuntu’ user. You can append multiple ‘-v’ to the end of the playbook invocation to get a more verbose output.

Figure 3: Graylog node activated

MongoDB

Graylog uses MongoDB to store meta information and configuration changes. The MongoDB software package that ships with Ubuntu 16.04 is supported by the latest Graylog software. The Ansible playbook to install the same is as follows:

The Ubuntu software package for MongoDB is called the ‘mongodb-server’. It is installed, and the database server is started. The Ansible playbook waits for the MongoDB server to start and listen on the default port 27017. The above playbook can be invoked using the following command:

$ ansible-playbook -i inventory/kvm/inventory playbooks/configuration/graylog.yml --tags mongodb -K
Figure 4: Random HTTP message generator

Elasticsearch

Elasticsearch is a search engine that is written in Java and released under the Apache licence. It is based on Lucene (an information retrieval software library) and provides a full-text search feature. The elastic.co website provides .deb packages that can be used to install the same on Ubuntu. The Ansible playbook for this is provided below:

- name: Install Elasticsearch
 
hosts: ubuntu
 
become: yes
 
become_method: sudo
 
gather_facts: true
 
tags: [elastic]
 
tasks:
 
- name: Add key
 
apt_key:
 
url: https://artifacts.elastic.co/GPG-KEY-elasticsearch
 
state: present
 
- name: Add elastic deb sources
 
lineinfile:
 
path: /etc/apt/sources.list.d/elastic-5.x.list
 
create: yes
 
line: ‘deb https://artifacts.elastic.co/packages/5.x/apt stable main’
 
- name: Update the software package repository
 
apt:
 
update_cache: yes
 
- name: Install Elasticsearch
 
package:
 
name: elasticsearch
 
state: latest
 
- name: Update cluster name
 
lineinfile:
 
path: /etc/elasticsearch/elastisearch.yml
 
create: yes
 
regexp: ‘^#cluster.name: my-application’
 
line: ‘cluster.name: graylog’
 
- name: Daemon reload
 
systemd: daemon_reload=yes
 
- name: Start elasticsearch service
 
service:
 
name: elasticsearch.service
 
state: started
 
- wait_for:
 
port: 9200
 
- name: Test Curl query
 
shell: curl -XGET ‘localhost:9200/?pretty’

The stable elastic.co repository package is installed before installing Elasticsearch. The cluster name is then updated in the /etc/elasticsearch/elasticsearch.yml configuration file. The system daemon services are reloaded, and the Elasticsearch service is started. The Ansible playbook waits for the service to run and listen on port 9200.

The above playbook can be invoked as follows:

$ ansible-playbook -i inventory/kvm/inventory playbooks/configuration/graylog.yml --tags elastic -K

You can perform a manual query to verify that Elasticsearch is running using the following Curl command:

$ curl -XGET ‘localhost:9200/?pretty’
 
{
 
“name” : “cFn-3YD”,
 
“cluster_name” : “elasticsearch”,
 
“cluster_uuid” : “nuBTSlFBTk6PDGyrfDCr3A”,
 
“version” : {
 
“number” : “5.6.5”,
 
“build_hash” : “6a37571”,
 
“build_date” : “2017-12-04T07:50:10.466Z”,
 
“build_snapshot” : false,
 
“lucene_version” : “6.6.1”
 
},
 
“tagline” : “You Know, for Search”
 
}
Figure 5: Graylog input random HTTP message generator

Graylog

The final step is to install Graylog itself. The .deb package available from the graylog2.org website is installed and then the actual ‘graylog-server’ package is installed. The configuration file is updated with credentials for the ‘admin’ user with a hashed string for the password ‘osfy’. The Web interface is also enabled with the default IP address of the guest VM. The Graylog service is finally started. The Ansible playbook to install Graylog is as follows:

- name: Install Graylog
 
hosts: ubuntu
 
become: yes
 
become_method: sudo
 
gather_facts: true
 
tags: [graylog]
 
tasks:
 
- name: Install Graylog repo deb
 
apt:
 
deb: https://packages.graylog2.org/repo/packages/graylog-2.3-repository_latest.deb
 
- name: Update the software package repository
 
apt:
 
update_cache: yes
 
- name: Install Graylog
 
package:
 
name: graylog-server
 
state: latest
 
- name: Update database credentials in the file
 
replace:
 
dest: “/etc/graylog/server/server.conf”
 
regexp: “{{ item.regexp }}”
 
replace: “{{ item.replace }}”
 
with_items:
 
- { regexp: ‘password_secret =’, replace: ‘password_secret = QXHg3Eqvsu PmFxUY2aKlgimUF05plMPXQ Hy1stUiQ1uaxgIG27 K3t2MviRiFLNot09U1ako T30njK3G69KIzqIoYqdY3oLUP’ }
 
- { regexp: ‘#root_username = admin’, replace: ‘root_username = admin’ }
 
- { regexp: ‘root_password_sha2 =’, replace: ‘root_password_sha2 = eabb9bb2efa089223 d4f54d55bf2333ebf04a29094bff00753536d7488629399’}
 
- { regexp: ‘#web_enable = false’, replace: ‘web_enable = true’ }
 
- { regexp: ‘#web_listen_uri = http://127.0.0.1:9000/’, replace: “web_listen_uri = http://{{ ansible_default_ipv4.address }}:9000/” }
 
- { regexp: ‘rest_listen_uri = http://127.0.0.1:9000/api/’, replace: “rest_listen_uri = http://{{ ansible_default_ipv4.address }}:9000/api/” }
 
- name: Start graylog service
 
service:
 
name: graylog-server.service
 
state: started

The above playbook can be run using the following command:

$ ansible-playbook -i inventory/kvm/inventory playbooks/configuration/graylog.yml --tags graylog -K

Web interface

You can now open the URL http://192.168.122.25:9000 in a browser on the host system to see the default Graylog login page as shown in Figure 1.

The user name is ‘admin’ and the password is ‘osfy’. You will then be taken to the Graylog home page as shown in Figure 2.

The guest VM is a single node, and hence if you traverse to System -> Nodes, you will see this node information as illustrated in Figure 3.

You can now test the Graylog installation by adding a data source as input by traversing System -> Input in the Web interface. The ‘random HTTP message generator’ is used as a local input, as shown in Figure 4.

The newly created input source is now running and visible as a local input in the Web page as shown in Figure 5.

After a few minutes, you can observe the created messages in the Search link as shown in Figure 6.

Figure 6: Graylog random HTTP messages

Uninstalling Graylog

An Ansible playbook to stop the different services, and to uninstall Graylog and its dependency software packages, is given below for reference:

---
 
- name: Uninstall Graylog
 
hosts: ubuntu
 
become: yes
 
become_method: sudo
 
gather_facts: true
 
tags: [uninstall]
 
tasks:
 
- name: Stop the graylog service
 
service:
 
name: graylog-server.service
 
state: stopped
 
- name: Uninstall graylog server
 
package:
 
name: graylog-server
 
state: absent
 
- name: Stop the Elasticsearch server
 
service:
 
name: elasticsearch.service
 
state: stopped
 
- name: Uninstall Elasticsearch
 
package:
 
name: elasticsearch
 
state: absent
 
- name: Stop the MongoDB server
 
service:
 
name: mongodb
 
state: stopped
 
- name: Uninstall MongoDB
 
package:
 
name: mongodb-server
 
state: absent
 
- name: Uninstall pre-requisites
 
package:
 
name: “{{ item }}”
 
state: absent
 
with_items:
 
- pwgen
 
- uuid-runtime
 
- openjdk-8-jre-headless
 
- apt-transport-https

The above playbook can be invoked using:

$ ansible-playbook -i inventory/kvm/inventory playbooks/admin/uninstall-graylog.yml -K

RELATED ARTICLESMORE FROM AUTHOR