top of page

Automate Prometheus Deployment With Ansible

Prometheus

Prometheus, at its core, is a very simple and powerful tool to monitor and track your servers. Let’s take a use case. A company deploys and services 100s of ec2 instances on Amazon. How will that company be able to keep track of all the events and services that are running on those instances? What if one of those instances seem fine and is up and running, however, the core service it is meant for has crashed. The only way to check this is through the log files, but who will be responsible for checking through the log files of hundreds of instances?

This is where a service like Prometheus is so useful. Prometheus can monitor hundreds of instances and display metrics and statistics from all of them to you in one simple dashboard. You can even get more in-depth dashboards by connecting Grafana to your Prometheus service.

Grafana is a web application that gives you visualization tools. In a nutshell, you can create and use dashboards from Grafana to visualize all the statistics and metrics presented by Prometheus.

Project Scope

Today we will be creating an Ansible playbook to automate the creation of an EC2 instance as well as the download of grafana, node_exporter, and Prometheus into that instance. We will tackle this project in small steps.

Step 1

Manually create an ec2 box and download Prometheus, node_exporter, and grafana into and ensure everything is running.

Step 2

Create an Ansible playbook that will go through that same process in step 1.

Step 3

Write into the playbook the ability to create an ec2 instance.

End

With the run of a single playbook, an ec2 instance should be deployed with Prometheus, node_exporter, and grafana installed into it.

Step 1: Prometheus and Grafana Installation

Pre-requisites:

  • Download AWS CLI and run aws configure

  • Create an ec2 instance with Ubuntu 16.04

  • Make sure ports 80, 22, 9090, 9100, and 3000 are open on your ec2 instance

  • Ssh into your ec2 instance

Download nginx:

We will be using Nginx to host our web service. Follow these steps to download and install Nginx onto your Ubuntu box.

$ sudo apt-get update
$ sudo apt-get install nginx

After installing check your web server.

$ systemctl status nginx

The server should be active and running. Confirm this by going to your ec2 instance’s public ip. You should see a ‘Welcome to nginx!’ message.

Download prometheus:

Create two new users to isolate the ownership of Prometheus’ core files and directories.

$ sudo useradd --no-create-home --shell /bin/false prometheus
$ sudo useradd --no-create-home --shell /bin/false node_exporter

Create the directories to store Prometheus’ files and data.

$ sudo mkdir /etc/prometheus
$ sudo mkdir /var/lib/prometheus

Set the user and group ownership on the new directories to the prometheus user.

$ sudo chown prometheus:prometheus /etc/prometheus
$ sudo chown prometheus:prometheus /var/lib/prometheus

Download the latest version of prometheus: https://prometheus.io/download/

$ cd ~
$ curl -LO https://github.com/prometheus/prometheus/releases/download/v2.21.0/prometheus-2.21.0.linux-amd64.tar.gz 

Use the sha256sum command to generate a checksum and compare it to the checksum on the prometheus website to ensure correct download.

$ sha256sum prometheus-2.21.0.linux-amd64.tar.gz

Tar the download.

$ tar xvf prometheus-2.21.0.linux-amd64.tar.gz

Copy the binaries to the proper directories.

$ sudo cp prometheus-2.21.0.linux-amd64/prometheus /usr/local/bin/
$ sudo cp prometheus-2.21.0.linux-amd64/promtool /usr/local/bin/

Set the user and group ownership on the binaries to the prometheus user created before.

$ sudo chown prometheus:prometheus /usr/local/bin/prometheus
$ sudo chown prometheus:prometheus /usr/local/bin/promtool

Copy the consoles and console_libraries directories to /etc/prometheus.

$ sudo cp -r prometheus-2.21.0.linux-amd64/consoles /etc/prometheus
$ sudo cp -r prometheus-2.21.0.linux-amd64/console_libraries /etc/prometheus

Set the user and group ownership on the directories to the prometheus user.

$ sudo chown -R prometheus:prometheus /etc/prometheus/consoles
$ sudo chown -R prometheus:prometheus /etc/prometheus/console_libraries

Remove files that are no longer needed

$ rm -rf prometheus-2.21.0.linux-amd64.tar.gz prometheus-2.21.0.linux-amd64

Create a config file to run prometheus.

$ sudo nano /etc/prometheus/prometheus.yml

Copy the contents to the config file.

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'prometheus'
    scrape_interval: 5s
    static_configs:
      - targets: ['localhost:9090']

Set the user and group ownership on the configuration file to the prometheus user.

$ sudo chown prometheus:prometheus /etc/prometheus/prometheus.yml

Start up Prometheus as the prometheus user, providing the path to both the configuration file and the data directory.

$ sudo -u prometheus /usr/local/bin/prometheus \
$   --config.file /etc/prometheus/prometheus.yml \
$   --storage.tsdb.path /var/lib/prometheus/ \
$   --web.console.templates=/etc/prometheus/consoles \
$   --web.console.libraries=/etc/prometheus/console_libraries

Create a new service file.

$ sudo nano /etc/systemd/system/prometheus.service

Copy contents to service file.

[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target

[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus \
    --config.file /etc/prometheus/prometheus.yml \
    --storage.tsdb.path /var/lib/prometheus/ \
    --web.console.templates=/etc/prometheus/consoles \
    --web.console.libraries=/etc/prometheus/console_libraries

[Install]
WantedBy=multi-user.target

Reload systemd.

$ sudo systemctl daemon-reload

Start Prometheus.

$ sudo systemctl start prometheus

Check the service status.

$ sudo systemctl status prometheus

Enable service to start on boot.

$ sudo systemctl enable prometheus

Download the latest version of node_exporter: https://prometheus.io/download/

$ cd ~
$ curl -LO https://github.com/prometheus/node_exporter/releases/download/v1.0.1/node_exporter-1.0.1.linux-amd64.tar.gz

Use the sha256sum command to generate a checksum and compare it to the checksum on the prometheus website to ensure correct download.

$ sha256sum node_exporter-1.0.1.linux-amd64.tar.gz

tar the download.

$ tar xvf node_exporter-1.0.1.linux-amd64.tar.gz

Copy files to correct directory and change ownership to node_exporter user created earlier.

$ sudo cp node_exporter-1.0.1.linux-amd64/node_exporter /usr/local/bin
$ sudo chown node_exporter:node_exporter /usr/local/bin/node_exporter

Remove files which are no longer needed.

$ rm -rf node_exporter-1.0.1.linux-amd64.tar.gz node_exporter-1.0.1.linux-amd64

Create a service file for node_exporter.

$ sudo nano /etc/systemd/system/node_exporter.service 

Copy the content into the service file.

[Unit]
Description=Node Exporter
Wants=network-online.target
After=network-online.target

[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter

[Install]
WantedBy=multi-user.target

Reload systemd.

$ sudo systemctl daemon-reload

Start node_exporter.

$ sudo systemctl start node_exporter

Check the status.

$ sudo systemctl status node_exporter

Enable node_exporter to start on boot.

$ sudo systemctl enable node_exporter

Open the config file once more and add to it.

$ sudo nano /etc/prometheus/prometheus.yml

Your config file should look like this.

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'prometheus'
    scrape_interval: 5s
    static_configs:
      - targets: ['localhost:9090']
  - job_name: 'node_exporter'
    scrape_interval: 5s
    static_configs:
      - targets: ['localhost:9100']

Restart prometheus.

$ sudo systemctl restart prometheus

Check the status.

$ sudo systemctl status prometheus

You should be able to enter the public IP now and you should see the prometheus dashboard.

Download the latest version of Grafana: https://grafana.com/grafana/download

$ sudo apt-get install -y adduser libfontconfig
$ wget https://dl.grafana.com/oss/release/grafana_7.2.0_amd64.deb
$ sudo dpkg -i grafana_7.2.0_amd64.deb

Set up and start the Grafana server.

$ sudo systemctl daemon-reload && sudo systemctl enable grafana-server && sudo systemctl start grafana-server

Enter the public IP with port 3000 online to check the grafana dashboard. Add a new data source and select prometheus as the data source. Set the Prometheus server URL as the public IP URL with port 9090. Click add to test the connection and save the new data source.

You have successfully set up Prometheus and grafana on your ec2 instance manually. Our next step is to turn these command prompts into an Ansible playbook.


Part 2: Ansible

Ansible is a simple automation engine that automates cloud provisioning, configuration management, application deployment, intra-service orchestration, and many others. It uses the YAML format to develop code for it. We will be creating many Ansible playbooks and roles to automate the installation process in part 1.

Directory Structure

roles
  /grafana
    /tasks
      main.yml
  /nginx
    /tasks
      main.yml
  /node_exporter
    /files
      node_exporter.service
      prometheus2.yml
    /tasks
      main.yml
  /prometheus
    /files
      prometheus.service
      prometheus.yml
    /tasks
      main.yml
  /ubuntu_update
    /tasks
      main.yml
main.yml

Lets compare the command line prompts to the Ansible yaml files.

roles/ubuntu_update/tasks/main.yml

Command line:

$ sudo apt-get update

main.yml:

---
  - name: Run the equivalent of "apt-get update" as a separate step
    apt:
      update_cache: yes

roles/nginx/tasks/main.yml

Command line:

$ sudo apt-get update

main.yml:

---
  - name: Download nginx
    apt:  
	  pkg:
      -  nginx

roles/prometheus/tasks/main.yml

Command line:

$ sudo useradd --no-create-home --shell /bin/false prometheus
$ sudo useradd --no-create-home --shell /bin/false node_exporter
$ sudo mkdir /etc/prometheus
$ sudo mkdir /var/lib/prometheus
$ sudo chown prometheus:prometheus /etc/prometheus
$ sudo chown prometheus:prometheus /var/lib/prometheus
$ cd ~
$ curl -LO https://github.com/prometheus/prometheus/releases/download/v2.21.0/prometheus-2.21.0.linux-amd64.tar.gz 
$ sha256sum prometheus-2.21.0.linux-amd64.tar.gz
$ tar xvf prometheus-2.21.0.linux-amd64.tar.gz
$ sudo cp prometheus-2.21.0.linux-amd64/prometheus /usr/local/bin/
$ sudo cp prometheus-2.21.0.linux-amd64/promtool /usr/local/bin/
$ sudo chown prometheus:prometheus /usr/local/bin/prometheus
$ sudo chown prometheus:prometheus /usr/local/bin/promtool
$ sudo cp -r prometheus-2.21.0.linux-amd64/consoles /etc/prometheus
$ sudo cp -r prometheus-2.21.0.linux-amd64/console_libraries /etc/prometheus
$ sudo chown -R prometheus:prometheus /etc/prometheus/consoles
$ sudo chown -R prometheus:prometheus /etc/prometheus/console_libraries
$ rm -rf prometheus-2.21.0.linux-amd64.tar.gz prometheus-2.21.0.linux-amd64
$ sudo nano /etc/prometheus/prometheus.yml
global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'prometheus'
    scrape_interval: 5s
    static_configs:
      - targets: ['localhost:9090']
$ sudo chown prometheus:prometheus /etc/prometheus/prometheus.yml
$ sudo -u prometheus /usr/local/bin/prometheus \
$   --config.file /etc/prometheus/prometheus.yml \
$   --storage.tsdb.path /var/lib/prometheus/ \
$   --web.console.templates=/etc/prometheus/consoles \
$   --web.console.libraries=/etc/prometheus/console_libraries
$ sudo nano /etc/systemd/system/prometheus.service
[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target

[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus \
    --config.file /etc/prometheus/prometheus.yml \
    --storage.tsdb.path /var/lib/prometheus/ \
    --web.console.templates=/etc/prometheus/consoles \
    --web.console.libraries=/etc/prometheus/console_libraries

[Install]
WantedBy=multi-user.target
$ sudo systemctl daemon-reload
$ sudo systemctl start prometheus
$ sudo systemctl status prometheus
$ sudo systemctl enable prometheus

main.yml:

---
  - name: Create prometheus user
    user:
	  name: prometheus
	  state: present
	  shell: /bin/false
	  system: no 
	  createhome: no
	become_user: root
  - name: Create directory
	file:
	  path: "{{ item }}"
	  state: directory
	  owner: prometheus
	  group: prometheus
	with_items: 
	  - /etc/prometheus
	  - /var/lib/prometheus
  - name: Unarchive a file that needs to be downloaded
	unarchive:
	  src: https://github.com/prometheus/prometheus/releases/download/v2.21.0/prometheus-2.21.0.linux-amd64.tar.gz
	  dest: /home/ubuntu
	  remote_src: yes
  - name: Copy file
	copy:
	  src: /home/ubuntu/prometheus-2.21.0.linux-amd64/prometheus
	  dest: /usr/local/bin/
  - name: Copy file
	copy:
	  src: /home/ubuntu/prometheus-2.21.0.linux-amd64/promtool
	  dest: /usr/local/bin/
  - name: Change file permission
	file:
	  path: /usr/local/bin/prometheus
	  owner: prometheus
	  group: prometheus
	  mode: '755'
  - name: Change file permission
	file:
	  path: /usr/local/bin/promtool
	  owner: prometheus
	  group: prometheus
  - name: Copy file
	copy:
	  src: /home/ubuntu/prometheus-2.21.0.linux-amd64/consoles
	  dest: /etc/prometheus
  - name: Copy file
	copy:
	  src: /home/ubuntu/prometheus-2.21.0.linux-amd64/console_libraries
	  dest: /etc/prometheus
  - name: Change file permission
	file:
	  path: /etc/prometheus/consoles
	  owner: prometheus
	  group: prometheus
  - name: Change file permission
	file:
	  path: /etc/prometheus/console_libraries
	  owner: prometheus
	  group: prometheus
  - name: Delete unwanted files
	file:
	  state: absent
	  path: /home/ubuntu/prometheus-2.21.0.linux-amd64
  - name: Delete unwanted files
	file:
	  state: absent
	  path: /home/ubuntu/prometheus-2.21.0.linux-amd64.tar.gz
  - name: Copy file
	copy:
	  src: /home/ubuntu/ansible-grafana/roles/prometheus/files/prometheus.yml
	  dest: /etc/prometheus/prometheus.yml
  - name: Change file permission
	file:
	  path: /etc/prometheus/prometheus.yml
	  owner: prometheus
	  group: prometheus
  - name: Copy file
	copy:
	  src: /home/ubuntu/ansible-grafana/roles/prometheus/files/prometheus.service
	  dest: /etc/systemd/system/prometheus.service
  - name: Start prometheus
	systemd:
	  state: started
	  name: prometheus
	  daemon_reload: yes
      enabled: yes

We also have a couple of service and setup files in the prometheus directory.

roles/prometheus/files/prometheus.service

[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target

[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus \
    --config.file /etc/prometheus/prometheus.yml \
    --storage.tsdb.path /var/lib/prometheus/ \
    --web.console.templates=/etc/prometheus/consoles \
    --web.console.libraries=/etc/prometheus/console_libraries

[Install]
WantedBy=multi-user.target

roles/prometheus/files/prometheus.yml

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'prometheus'
    scrape_interval: 5s
    static_configs:
      - targets: ['localhost:9090']

roles/node_eporter/tasks/main.yml

Command line:

$ cd ~
$ curl -LO https://github.com/prometheus/node_exporter/releases/download/v1.0.1/node_exporter-1.0.1.linux-amd64.tar.gz
$ sha256sum node_exporter-1.0.1.linux-amd64.tar.gz
$ tar xvf node_exporter-1.0.1.linux-amd64.tar.gz
$ sudo cp node_exporter-1.0.1.linux-amd64/node_exporter /usr/local/bin
$ sudo chown node_exporter:node_exporter /usr/local/bin/node_exporter
$ rm -rf node_exporter-1.0.1.linux-amd64.tar.gz node_exporter-1.0.1.linux-amd64
$ sudo nano /etc/systemd/system/node_exporter.service 
[Unit]
Description=Node Exporter
Wants=network-online.target
After=network-online.target

[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter

[Install]
WantedBy=multi-user.target
$ sudo systemctl daemon-reload
$ sudo systemctl start node_exporter
$ sudo systemctl status node_exporter
$ sudo systemctl enable node_exporter
$ sudo nano /etc/prometheus/prometheus.yml
global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'prometheus'
    scrape_interval: 5s
    static_configs:
      - targets: ['localhost:9090']
  - job_name: 'node_exporter'
    scrape_interval: 5s
    static_configs:
      - targets: ['localhost:9100']
$ sudo systemctl restart prometheus
$ sudo systemctl status prometheus

main.yml:

---
  - name: Create node_exporter user
	user:
	  name: node_exporter
	  state: present
	  shell: /bin/false
	  system: no 
	  createhome: no
	become_user: root
  - name: Unarchive a file that needs to be downloaded
	unarchive:
	  src: https://github.com/prometheus/node_exporter/releases/download/v1.0.1/node_exporter-1.0.1.linux-amd64.tar.gz
	  dest: /home/ubuntu
	  remote_src: yes
  - name: Copy file
	copy:
	  src: /home/ubuntu/node_exporter-1.0.1.linux-amd64/node_exporter
	  dest: /usr/local/bin/
  - name: Change file permission
	file:
	  path: /usr/local/bin/node_exporter
	  owner: node_exporter
	  group: node_exporter
	  mode: '755'
  - name: Delete unwanted files
	file:
	  state: absent
	  path: /home/ubuntu/node_exporter-1.0.1.linux-amd64.tar.gz
  - name: Delete unwanted files
	file:
	  state: absent
	  path: /home/ubuntu/node_exporter-1.0.1.linux-amd64
  - name: Copy file
	copy:
	  src: /home/ubuntu/ansible-grafana/roles/node_exporter/files/node_exporter.service
	  dest: /etc/systemd/system/node_exporter.service
  - name: Start node_exporter
	systemd:
	  state: started
	  name: node_exporter
	  daemon_reload: yes
	  enabled: yes
  - name: Copy file
	copy:
	  src: /home/ubuntu/ansible-grafana/roles/node_exporter/files/prometheus2.yml
	  dest: /etc/prometheus/prometheus.yml
  - name: Restart prometheus
	systemd:
	  name: prometheus
      state: restarted

We also have a couple of service and setup files in the prometheus directory.

roles/node_exporter/files/node_exporter.service

[Unit]
Description=Node Exporter
Wants=network-online.target
After=network-online.target

[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter

[Install]
WantedBy=multi-user.target

roles/node_exporter/files/prometheus2.yml

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'prometheus'
    scrape_interval: 5s
    static_configs:
      - targets: ['localhost:9090']
  - job_name: 'node_exporter'
    scrape_interval: 5s
    static_configs:
      - targets: ['localhost:9100']

roles/grafana/tasks/main.yml

Command line:

$ sudo apt-get install -y adduser libfontconfig
$ wget https://dl.grafana.com/oss/release/grafana_7.2.0_amd64.deb
$ sudo dpkg -i grafana_7.2.0_amd64.deb
$ sudo systemctl daemon-reload && sudo systemctl enable grafana-server && sudo systemctl start grafana-server

main.yml:

---
  - name: Download adduser libfontconfig
	apt: 
	  pkg:
	  - adduser
	  - libfontconfig
  - name: Download grafana 
	apt:
	  deb: https://dl.grafana.com/oss/release/grafana_7.2.0_amd64.deb   
  - name: Start grafana
	systemd:
	  state: started
	  name: grafana-server
	  daemon_reload: yes
	  enabled: yes

./main.yml

- name: run roles on remote box
  hosts: localhost
  become: true
  roles:
  - ubuntu_update
  - github
  - nginx
  - prometheus
  - node_exporter
  - grafana

Part 3: AWS automation

The next step is to automate the creation of the aws ec2 instance as well as push the code to github and pull it into the remote ec2 box. Therefore the directory structure is updated as well as the ./main.yml file.

roles
  /github
    /tasks
      main.yml
  /grafana
    /tasks
      main.yml
  /nginx
    /tasks
      main.yml
  /node_exporter
    /files
      node_exporter.service
      prometheus2.yml
    /tasks
      main.yml
  /prometheus
    /files
      prometheus.service
      prometheus.yml
    /tasks
      main.yml
  /ubuntu_update
    /tasks
      main.yml
main.yml

./main.yml

---
  - name: Create an EC2 instance
	hosts: localhost
	gather_facts: False
	tasks:
      - name: Launch instance
	    ec2:
	      key_name: #enter your key name
	      group: #enter your security group
	      instance_type: #enter instance type
	      image: #enter image id
	      wait: true
	      region: #enter region
	    register: ec2
	  - name: Print all ec2 variables
	    debug: var=ec2
	  - name: Get theh Ip address
	    debug: var=ec2.instances[0].public_dns_name
	

	  - name: add host to group 'just_created' with variable foo=42
	    add_host:
	      name: "{{ ec2.instances[0].public_dns_name }}"
	      groups: ec2_hosts
	      ansible_host: "{{ ec2.instances[0].public_dns_name }}"
	      ansible_ssh_user: ubuntu
	      ansible_ssh_private_key_file: ~/#enter key location
	  
	  - name: Wait for a while
	    pause: seconds=60
	

- name: run roles on remote box
  hosts: ec2_hosts
  become: true
  roles:
    - ubuntu_update
    - github
    - nginx
    - prometheus
    - node_exporter
    - grafana

roles/github/tasks/main.yml

---
  - name: Clone a repo 
	git:
	  repo: https://github.com/ansible/ansible-examples.git
      dest: /home/ubuntu

Run the command:

sudo ansible-playbook main.yml

This should go ahead and create an ec2 box as well as download Prometheus and Grafana into that box.

Congratulations on automating the Prometheus deployment with Ansible!

15 views0 comments

Comments


bottom of page