Micro-services Using go-kit: Hystrix Circuit Breaker


The idea to apply circuit breaker pattern is to give protection and control over latency and failure, in order to stop cascading failure in a distributed environment. In addition, it provides a fail fast and rapid recovery for our services.

Sharing is caring!

Overview

circuit breakerI wrote a lot about service monitoring in my previous article. The purpose of monitoring is, whenever we spot problems, we could follow up them in a fast and proper way. However, sometimes it is not enough. Now let assume if a service is failed and our monitoring tools already spot it by sending alerts. Meanwhile we investigate or fix the issues, there will be number of failure requests which potentially will cascade the error across multiple systems. Of course, with assumption our services are so popular then it reaches 300 requests per second.

Circuit Breaker Pattern

To overcome this issue, we can use the circuit breaker pattern. The idea of applying circuit breaker pattern is to gracefully degrade the service functionality whenever the request fail. Circuit breaker pattern will allow the micro-service to continue operating when related service fails. At the same time, it will give time to recover for the failing service.

Go-kit itself, per this article is written, provides three circuit breaker libraries. There are gobreaker, handybreaker and hystrix. For the purpose of this article, I will choose Hystrix as an implementation for this pattern.

Quote from their Hystrix page:

Hystrix is a library that helps you control the interactions between these distributed services by adding latency tolerance and fault tolerance logic. Hystrix does this by isolating points of access between the services, stopping cascading failures across them, and providing fallback options, all of which improve your system’s overall resiliency¹.

Use Case

In this article, I will implement circuit breaker pattern using Hystrix library into Lorem Service. I will display a Service currently unavailable message to acknowledge whenever circuit in the open state. Of course, the fallback message can be anything based on your requirement. Also, the  library will try to reconnect to the service in a specific interval to check its availability. If the request is successfully made, the circuit then enters into close state again.

For this purpose, I copied lorem-consul folder, into a new lorem-hystrix folder.

Step by Step

Before we begin, we need to download required libraries through go get command.

Step 1: circuitbreaker.go

First step is to create a new file with name circuitbreaker.go. Inside this file, I will create a function with name Hystrix, with input parameters command name, fallback message and logger. This function return an endpoint.Middleware. This is needed because I want to call an endpoint in a chainable fashion.

Basically, I only leverage the Hystrix function provided by go-kit, in order to provide fallback message and logging function.

Step 2: discover/main.go

In this step, I will configure Hystrix command for applying settings for a circuit. Then passing the existing Lorem endpoint to the Hystrix function.

Next, I will configure Hystrix stream handler and make it run on port 9000. This Hystrix stream handler is needed if we want to display the metrics in realtime. The display itself shows in the Hystrix Dashboard.

Step 3: Run It

Before running the examples, I need to launch consul agent and the other one is Hystrix dashboard.

For Hystrix Dashboard, I used docker image from mlabouardy/hystrix-dashboard. The dashboard will run on http://localhost:8181/hystrix.

Next, start three Lorem Service instances.

And the client as well.

In order to show the circuit breaker works, as well as the fallback message, I will execute the request in a forever loop manner.

Now you can try the circuit breaker by shutting down all three instances for example. After few seconds, start all the instances again. Similarly, you will see output as below:

Step 4: Configure Hystrix Dashboard

hystrix dashboard

Onto Hystrix dashboard, the only configuration is to enter the Hystrix Stream Handler URL (mentioned in Step 2). In this case, I enter http://192.168.1.103:9000/ for the URL configuration. Then click Monitor Stream to start real time monitoring.

These are several screenshots from the dashboard after I played around by shutting down and/or starting up Lorem Sevice instances:

circuit open
Circuit Open
Circuit Close
Circuit Close
Grafana + Hystrix
Grafana + Hystrix Dashboard

Summary

Circuit breaker is a design pattern to make a service resilient and self-healing. This approach will prevent cascading failures in a distributed environments because of its fault tolerance mechanism.

Go-kit itself already provide three libraries to support this pattern. Arguably, my decision to use Hystrix compare to other is because Hystrix has already shipped with its dashboard. Well, maybe I am wrong about it. Maybe the gobreaker or handybreaker have their dashboards too. I open to any discussion.

Alright, that’s all for now. You can check the sample on my github, under lorem-hystrix folder.

PS: whenever you want to know more about circuit breaker pattern, you can visit a very nice article from Martin Fowler.

References

Author: ru rocker

I am a professional software developer with more than 10 years experiences. I am a certified Java Developer (SCJP and SCWCD). However, In the recent months, I have more interest in DevOps and start to become a polyglot developer. Python and Go-lang become my favorite programming languages besides Java.

2 thoughts on “Micro-services Using go-kit: Hystrix Circuit Breaker”

  1. Hi,I tried to run the above experiment on mu machine,but when I was trying to kick-start the client with this command –
    o run lorem-hystrix/discover.d/main.go \
    > -consul.addr localhost -consul.port 8500
    I received this error.I checked the folder github.com/go-kit/kit/sd/consul but could not find a subscriber file with the below function.
    What should I do??
    # command-line-arguments
    lorem-hystrix/discover.d/main.go:74: undefined: consul.NewSubscriber

    1. Hi

      It seems the go-kit package has been updated and removed the NewSubscriber function. It is replaced by Instancer. I already push the latest source code into my repository.

Leave a Reply

Your email address will not be published. Required fields are marked *