Micro-services Using go-kit: Hystrix Circuit Breaker
The idea to apply circuit breaker pattern is to give protection and control over latency and failure, in order to stop cascading failure in a distributed environment. In addition, it provides a fail fast and rapid recovery for our services.
Overview
I wrote a lot about service monitoring in my previous article. The purpose of monitoring is, whenever we spot problems, we could follow up them in a fast and proper way. However, sometimes it is not enough. Now let assume if a service is failed and our monitoring tools already spot it by sending alerts. Meanwhile we investigate or fix the issues, there will be number of failure requests which potentially will cascade the error across multiple systems. Of course, with assumption our services are so popular then it reaches 300 requests per second.
Circuit Breaker Pattern
To overcome this issue, we can use the circuit breaker pattern. The idea of applying circuit breaker pattern is to gracefully degrade the service functionality whenever the request fail. Circuit breaker pattern will allow the micro-service to continue operating when related service fails. At the same time, it will give time to recover for the failing service.
Go-kit itself, per this article is written, provides three circuit breaker libraries. There are gobreaker, handybreaker and hystrix. For the purpose of this article, I will choose Hystrix as an implementation for this pattern.
Quote from their Hystrix page:
Hystrix is a library that helps you control the interactions between these distributed services by adding latency tolerance and fault tolerance logic. Hystrix does this by isolating points of access between the services, stopping cascading failures across them, and providing fallback options, all of which improve your system’s overall resiliency¹.
Use Case
In this article, I will implement circuit breaker pattern using Hystrix library into Lorem Service. I will display a Service currently unavailable message to acknowledge whenever circuit in the open state. Of course, the fallback message can be anything based on your requirement. Also, the library will try to reconnect to the service in a specific interval to check its availability. If the request is successfully made, the circuit then enters into close state again.
For this purpose, I copied lorem-consul folder, into a new lorem-hystrix folder.
Step by Step
Before we begin, we need to download required libraries through go get command.
|
1 2 |
# Download required libraries go get github.com/afex/hystrix-go/hystrix |
Step 1: circuitbreaker.go
First step is to create a new file with name circuitbreaker.go. Inside this file, I will create a function with name Hystrix, with input parameters command name, fallback message and logger. This function return an endpoint.Middleware. This is needed because I want to call an endpoint in a chainable fashion.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 |
func Hystrix(commandName string, fallbackMesg string, logger log.Logger) endpoint.Middleware { return func(next endpoint.Endpoint) endpoint.Endpoint { return func(ctx context.Context, request interface{}) (response interface{}, err error) { var resp interface{} if err := hystrix.Do(commandName, func() (err error) { resp, err = next(ctx, request) return err }, func(err error) error { logger.Log("fallbackErrorDesc", err.Error()) resp = struct { Fallback string `json:"fallback"` }{ fallbackMesg, } return nil }); err != nil { return nil, err } return resp, nil } } } |
Basically, I only leverage the Hystrix function provided by go-kit, in order to provide fallback message and logging function.
Step 2: discover/main.go
In this step, I will configure Hystrix command for applying settings for a circuit. Then passing the existing Lorem endpoint to the Hystrix function.
|
1 2 3 4 |
// configure hystrix hystrix.ConfigureCommand("Lorem Request", hystrix.CommandConfig{Timeout: 1000}) loremEndpoint = lorem_hystrix.Hystrix("Lorem Request", "Service currently unavailable", logger)(loremEndpoint) |
Next, I will configure Hystrix stream handler and make it run on port 9000. This Hystrix stream handler is needed if we want to display the metrics in realtime. The display itself shows in the Hystrix Dashboard.
|
1 2 3 4 5 6 |
// configure the hystrix stream handler hystrixStreamHandler := hystrix.NewStreamHandler() hystrixStreamHandler.Start() go func() { errc <- http.ListenAndServe(net.JoinHostPort("", "9000"), hystrixStreamHandler) }() |
Step 3: Run It
Before running the examples, I need to launch consul agent and the other one is Hystrix dashboard.
|
1 2 3 4 5 |
# consul docker run --rm -p 8400:8400 -p 8500:8500 -p 8600:53/udp -h node1 progrium/consul -server -bootstrap -ui-dir /ui # hystrix docker run -p 8181:9002 --name hystrix-dashboard mlabouardy/hystrix-dashboard:latest |
For Hystrix Dashboard, I used docker image from mlabouardy/hystrix-dashboard. The dashboard will run on http://localhost:8181/hystrix.
Next, start three Lorem Service instances.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 |
# start three lorem services in port 7002 7003 7010. # Each runs in separate terminal cd $GOPATH/src/github.com/ru-rocker/gokit-playground go run lorem-hystrix/lorem-hystrix.d/main.go \ -consul.addr localhost -consul.port 8500 \ -advertise.addr 192.168.1.103 -advertise.port 7002 go run lorem-hystrix/lorem-hystrix.d/main.go \ -consul.addr localhost -consul.port 8500 \ -advertise.addr 192.168.1.103 -advertise.port 7003 go run lorem-hystrix/lorem-hystrix.d/main.go \ -consul.addr localhost -consul.port 8500 \ -advertise.addr 192.168.1.103 -advertise.port 7010 |
And the client as well.
|
1 2 3 4 |
# client cd $GOPATH/src/github.com/ru-rocker/gokit-playground go run lorem-hystrix/discover.d/main.go \ -consul.addr localhost -consul.port 8500 |
In order to show the circuit breaker works, as well as the fallback message, I will execute the request in a forever loop manner.
|
1 2 3 4 5 |
# execute in forever loop # each runs in separate terminal while true; do curl -XPOST -d'{"requestType":"word", "min":10, "max":10}' http://localhost:8080/sd-lorem; sleep 1; done; while true; do curl -XPOST -d'{"requestType":"sentence", "min":10, "max":10}' http://localhost:8080/sd-lorem; sleep 1; done; while true; do curl -XPOST -d'{"requestType":"paragraph", "min":10, "max":10}' http://localhost:8080/sd-lorem; sleep 1; done; |
Now you can try the circuit breaker by shutting down all three instances for example. After few seconds, start all the instances again. Similarly, you will see output as below:
|
1 2 3 4 5 6 |
# sample output {"message":"curiosarum"} {"message":"caecitatem"} {"fallback":"Service currently unavailable"} {"fallback":"Service currently unavailable"} {"message":"nonnullius"} |
Step 4: Configure Hystrix Dashboard
Onto Hystrix dashboard, the only configuration is to enter the Hystrix Stream Handler URL (mentioned in Step 2). In this case, I enter http://192.168.1.103:9000/ for the URL configuration. Then click Monitor Stream to start real time monitoring.
These are several screenshots from the dashboard after I played around by shutting down and/or starting up Lorem Sevice instances:
Summary
Circuit breaker is a design pattern to make a service resilient and self-healing. This approach will prevent cascading failures in a distributed environments because of its fault tolerance mechanism.
Go-kit itself already provide three libraries to support this pattern. Arguably, my decision to use Hystrix compare to other is because Hystrix has already shipped with its dashboard. Well, maybe I am wrong about it. Maybe the gobreaker or handybreaker have their dashboards too. I open to any discussion.
Alright, that’s all for now. You can check the sample on my github, under lorem-hystrix folder.
PS: whenever you want to know more about circuit breaker pattern, you can visit a very nice article from Martin Fowler.
References
- Circuit Breaker Pattern (https://martinfowler.com/bliki/CircuitBreaker.html)
- Hystrix Wiki (https://github.com/Netflix/Hystrix/wiki)
- Spring Boot Circuit Breaker Pattern (https://spring.io/guides/gs/circuit-breaker)
Hi,I tried to run the above experiment on mu machine,but when I was trying to kick-start the client with this command –
o run lorem-hystrix/discover.d/main.go \
> -consul.addr localhost -consul.port 8500
I received this error.I checked the folder github.com/go-kit/kit/sd/consul but could not find a subscriber file with the below function.
What should I do??
# command-line-arguments
lorem-hystrix/discover.d/main.go:74: undefined: consul.NewSubscriber
Hi
It seems the go-kit package has been updated and removed the NewSubscriber function. It is replaced by Instancer. I already push the latest source code into my repository.