Sharing is caring!

Overview

Basically, all active systems needs to be monitored. Monitoring is a pivotal component because we want to make sure that everything works perfectly. There is no exception for micro-services architecture as well. We have to monitor our API in real time and historical in order to know whether it is running well over the time. Another reason is to spot any bottleneck in specific API calls. So we can have a proper reaction if we find any problem in our API.

Prometheus

Prometheus is gaining its popularity as a monitoring tools for storing time series data. It provides various metric types, such as: Counter, Gauge, Histogram and Summary. Also, Prometheus gives functional expression language to select and aggregate time series data. The result can be viewed as a graph or tabular data.

However, sometimes we need better visualization. For this case, we could combine Prometheus with Grafana in order to have an amazing tools to analyze and visualize the data. Prometheus provides the data, meanwhile Grafana provides dashboards as an interface to analyze and visualize the data.

Prometheus vs ELK

In my previous article, I talked about logging monitoring with ELK stack. Meanwhile, Prometheus is another monitoring tools as well. What is the different then? Well, based on their FAQ page, Prometheus is a system to collect and process metrics, not an event logging system. So it is clear both tools have different functionality. One is for collecting event logging (ELK) and the other one is to collect and process metrics (Prometheus).

How Does It Work

Simply, we configure our monitored targets in the configuration file. Then Prometheus starts to collects metrics from targets by scraping an HTTP endpoint on these targets.

More about Prometheus and Grafana, visit their website on www.prometheus.io and www.grafana.com.

Use Case

In this article, I will monitor request count and the request latency. Then I will expose into metrics endpoint. Prometheus will scrap data from metrics endpoint and visualize them using Grafana.

For this purpose, I copied lorem-rate-limit folder into new folder and give it a name lorem-metrics.

Note: Prometheus and Grafana will be run on Docker.

Step by Step

Before we begin, we need to download required libraries through go get command.

Step 1: Update instrument.go

First, create new struct, name it metricsMiddleware. This struct extends Service function with two additional parameters: requestCounter and requestLatency.

Next, we create a new function and give it a name Metrics. The function will accept two input parameters, which are requestCounter and requestLatency. And it will return ServiceMiddleware wrapper, which basically is another function that return Service.

And we implement Service function for our new metricsMiddleware struct.

Step 2: transport.go

We need another router for handling metrics path. Inside MakeHttpHandler function, add:

Step 3: main.go

In this file, first we need to declare variable for requestCounter. Since this is a counter, we will use Prometheus Counter metric type (See sub-heading Prometheus). And another variable for requestLatency with Summary metric type.

And chaining it into existing service:

Note: see highlighted line for including metrics into existing service.

Step 4: Prometheus and Grafana

As I mentioned above, we will going to setup Prometheus and Grafana using docker compose file. I will use official docker images, which are prom/prometheus and grafana/grafana.

Prometheus

For running Prometheus inside docker, at minimum we need to create prometheus.yml file and mount the configuration file into prometheus image.

Note: I am using IP Address (highlighted) instead of localhost for lorem service because this configuration will be executed inside container.

Grafana

For Grafana, we need create an empty folder to be mounted inside container then named it grafana_db (you are free to select another name). The purpose for this folder is as a storage location for SQLite data. As you may know, Grafana is using SQLite for its data store. Next step is to specify administrator password by overriding GF_SECURITY_ADMIN_PASSWORD environment variables.

docker-compose-prometheus-grafana.yml

Step 5: Run

Now it is time to run the demo. First we will start our lorem-service and check whether path /metrics exists. We will invoke several request into the service in order to retrieve metrics data. Next start our container to bring up Prometheus and Grafana.

lorem-service

Prometheus – Grafana

If everything runs well, you can check Prometheus at localhost:9090 and Grafana at localhost:3000. Grafana will ask you username and password. You can type admin as user name and your password based on your GF_SECURITY_ADMIN_PASSWORD in the compose file.

We need to configure Grafana for the first time. First we need to set up the data source. Create new data source, give it a Name, choose Type: Prometheus and set URL into localhost:9090. Next we need to setup a dashboard. Create new dashboard (Menu -> Dashboard -> New) and click Graph icon. Then click Panel Title and click Edit. Then you will see Metrics tab under the panel. You can use query expression to gather the metrics from lorem-service. For this example, I will gather total request count per method (word, sentence or paragraph).

Note: you can try with another query, such as for request latency.

Conclusion

Whenever we expose our API publicly, it is the right moment to think about its visibility. We need to know whether everything works properly or not. Therefore we need monitoring tools.

Go-kit has shown one of its lovely feature to support this functionality. We do not need to refactor our existing code. Only adding one metrics functionality, wrap it into service and expose metrics URL. That’s it.

However, even we have a very sophisticated and advance monitoring system, without our reaction, everything is useless. Because monitoring needs our reaction. Properly.

All right, that’s all for today. You can still checkout my code on my github.