Monitoring with Prometheus – Example with Ansible and Vagrant

What is Prometheus

Prometheus is an open-source systems monitoring and alerting toolkit originally built at SoundCloud. For starters you can consider Prometheus is a port scanning utility which can scan given number of application port to collect and store various time series based monitoring/telemetry matrices which can be pulled or exposed as HTTP service. Which can further be used to create additional dashboard for better visualization.

It was originally designed to capture HTTP traffic, however, since its inception it has evolved to be overall monitoring platform to virtually monitor anything from Infrastructure, Application, containers, IoT sensors etc. All you need to do is expose your monitoring matrices in HTTP using set of client APIs, virtually for all popular languages, and you are all set.

Out of the box, Prometheus comes with server which does the port scanning to collect and store all matrices and an alerting server which can be used to send notifications based on various rules. Additionally there are numerous plugs, node-exporters, push gateways which allow different applications to be able to export their monitoring data into Prometheus server.

Architecture Overview

Even though this blog post is meant to be a crash course on getting Prometheus up and running, I would still highlight some key components, which are essential to have a successful monitoring platform to work. Take a quick look into the architecture diagram[1].

If the diagram is too confusing, then for sake of this article only 3 components are important, we can keep the node_exporter and pushgateway for later as they are kind of value added features not the core.

  • Prometheus Server — Which will send alert based on the rules defined as part of configuration

  • Alerting Server [optional if you care about alerting ] — Which will notify the predefined alerts/template based messages to various channels including email, slack, hipchat, pagerduty etc.

  • Grafana Server [optional if you care about Dashboard ]– Create dashboard. Grafana can also generate alert based similar to Prometheus.

Installation and Setup — Prometheus Server

Prometheus is written in Golang. Either you get the source code from github, in which case you have to compile it, or directly get the latest binary.

Open browser at http://127.0.0.1:9090

You should see something like this.

Now you can scroll through metrics and you would see some out of the box metric mostly for the local machine.

Exporting your custom metric to Prometheus

In case the node_exporter is not enough for your monitoring needs, then you might consider writing your own script to export the required matrices to Prometheus server.

You start you can get the sample code which demonstrate the gauge metric and display it on web server. Follow the instructions on the github page.

If you open browser at http://localhost:8000 you should see something like this

If you can see the “bcr_gauge_example” is the metrics that you are exporting to Prometheus.

Configure Prometheus to Scan the example exporter

Copy paste this example prometheus.yml into your local installation

 

This tells Prometheus to can the example_export app. Now restart the Prometheus server, and access the Prometheus GUI you should be able to see our metrics. Type “bcr_gauge_example{instance=”localhost:8000″,job=”example_export”}” into the console bar of the GUI, it will filter out our data and display it.

Make sure to chose 10s interval [ highlighted in the image below]

By this time you have Installed, Configured and exported your custom configurations to Prometheus server on Linux box. Please leave comment below if you find anything missing.