Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

State Change Callback in circuit breaker #101

Open
wants to merge 6 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -95,6 +95,15 @@ hystrix.ConfigureCommand("my_command", hystrix.CommandConfig{

You can also use ```hystrix.Configure()``` which accepts a ```map[string]CommandConfig```.


### Enable State Change Callback
In your main.go, register the Callback handler for a command which will be called in a goroutine.

```go
callback.Register("my_command", func(name string, state callback.State) {
fmt.Println("Name ", name, " State ", state)
})
```
### Enable dashboard metrics

In your main.go, register the event stream HTTP handler on a port and launch it in a goroutine. Once you configure turbine for your [Hystrix Dashboard](https://github.com/Netflix/Hystrix/tree/master/hystrix-dashboard) to start streaming events, your commands will automatically begin appearing.
Expand Down
13 changes: 13 additions & 0 deletions go.mod
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
module github.com/afex/hystrix-go

go 1.15

require (
github.com/DataDog/datadog-go v4.0.1+incompatible
github.com/cactus/go-statsd-client/statsd v0.0.0-20200728222731-a2baea3bbfc6
github.com/rcrowley/go-metrics v0.0.0-20200313005456-10cdbea86bc0
github.com/smartystreets/goconvey v1.6.4
github.com/stretchr/testify v1.6.1 // indirect
)

replace github.com/afex/hystrix-go => github.com/ContinuumLLC/hystrix-go v1.0.1
31 changes: 31 additions & 0 deletions go.sum
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
github.com/DataDog/datadog-go v4.0.1+incompatible h1:6BF/mPNV2f+PJzpw5vxDK91Z3IfOM7fEmz5ptuL+0vo=
github.com/DataDog/datadog-go v4.0.1+incompatible/go.mod h1:LButxg5PwREeZtORoXG3tL4fMGNddJ+vMq1mwgfaqoQ=
github.com/cactus/go-statsd-client/statsd v0.0.0-20200728222731-a2baea3bbfc6 h1:arL5CcymFtWyyi5Xk44VpSxUfFfJqxCMbhZQ0lxLkDE=
github.com/cactus/go-statsd-client/statsd v0.0.0-20200728222731-a2baea3bbfc6/go.mod h1:l/bIBLeOl9eX+wxJAzxS4TveKRtAqlyDpHjhkfO0MEI=
github.com/davecgh/go-spew v1.1.0 h1:ZDRjVQ15GmhC3fiQ8ni8+OwkZQO4DARzQgrnXU1Liz8=
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/gopherjs/gopherjs v0.0.0-20181017120253-0766667cb4d1 h1:EGx4pi6eqNxGaHF6qqu48+N2wcFQ5qg5FXgOdqsJ5d8=
github.com/gopherjs/gopherjs v0.0.0-20181017120253-0766667cb4d1/go.mod h1:wJfORRmW1u3UXTncJ5qlYoELFm8eSnnEO6hX4iZ3EWY=
github.com/jtolds/gls v4.20.0+incompatible h1:xdiiI2gbIgH/gLH7ADydsJ1uDOEzR8yvV7C0MuV77Wo=
github.com/jtolds/gls v4.20.0+incompatible/go.mod h1:QJZ7F/aHp+rZTRtaJ1ow/lLfFfVYBRgL+9YlvaHOwJU=
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/rcrowley/go-metrics v0.0.0-20200313005456-10cdbea86bc0 h1:MkV+77GLUNo5oJ0jf870itWm3D0Sjh7+Za9gazKc5LQ=
github.com/rcrowley/go-metrics v0.0.0-20200313005456-10cdbea86bc0/go.mod h1:bCqnVzQkZxMG4s8nGwiZ5l3QUCyqpo9Y+/ZMZ9VjZe4=
github.com/smartystreets/assertions v0.0.0-20180927180507-b2de0cb4f26d h1:zE9ykElWQ6/NYmHa3jpm/yHnI4xSofP+UP6SpjHcSeM=
github.com/smartystreets/assertions v0.0.0-20180927180507-b2de0cb4f26d/go.mod h1:OnSkiWE9lh6wB0YB77sQom3nweQdgAjqCqsofrRNTgc=
github.com/smartystreets/goconvey v1.6.4 h1:fv0U8FUIMPNf1L9lnHLvLhgicrIVChEkdzIKYqbNC9s=
github.com/smartystreets/goconvey v1.6.4/go.mod h1:syvi0/a8iFYH4r/RixwvyeAJjdLS9QV7WQ/tjFTllLA=
github.com/stretchr/objx v0.1.0 h1:4G4v2dO3VZwixGIRoQ5Lfboy6nUhCyYzaqnIAPPhYs4=
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
github.com/stretchr/testify v1.6.1 h1:hDPOHmpOpP40lSULcqw7IrRb/u7w6RpDC9399XyoNd0=
github.com/stretchr/testify v1.6.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
golang.org/x/net v0.0.0-20190311183353-d8887717615a/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
golang.org/x/tools v0.0.0-20190328211700-ab21143f2384/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c h1:dUUwHk2QECo/6vqA44rthZ8ie2QXMNeKRTHCNY2nXvo=
gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
34 changes: 34 additions & 0 deletions hystrix/callback/callback.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
package callback

var circuitCallback map[string]stateFunc

//State is a type to hold Circuit-state this will be used while calling stateFunc on State change
type State string

const (
//Open is a state to indicate that Circuit state is Open
Open = "Open"
//Close is a state to indicate that Circuit state is Close
Close = "Close"
//AllowSingle is a state to indicate that Circuit state is AllowSingle or trying to Open Circuit

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AllowSingle is for trying to 'close' the circuit?

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes

AllowSingle = "Allow Single"
)

type stateFunc func(name string, state State)

func init() {
circuitCallback = make(map[string]stateFunc)
}

//Register adds callback for a circuit
func Register(name string, callbackFunc stateFunc) {
circuitCallback[name] = callbackFunc
}

//Invoke is a function to invoke Callback function in a goroutine on State change
func Invoke(name string, state State) {
callbackFunc, _ := circuitCallback[name]
if callbackFunc != nil {
go callbackFunc(name, state)
}
}
57 changes: 57 additions & 0 deletions hystrix/callback/callback_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
package callback

import (
"testing"
"time"

. "github.com/smartystreets/goconvey/convey"
)

func TestRegister(t *testing.T) {
Convey("Register a command", t, func() {
Register("Test-Command", func(name string, state State) {})

Convey("Read Callback function for Registered Command", func() {
callbackFunc, _ := circuitCallback["Test-Command"]
So(callbackFunc, ShouldNotBeNil)
})
Convey("Read Callback function for unknown Command", func() {
callbackFunc, _ := circuitCallback["Command"]
So(callbackFunc, ShouldBeNil)
})
})
}

func TestInvoke(t *testing.T) {
Convey("Register a command", t, func() {
var callbackInvoked bool
var callbackState State = Close
Register("TestInvokeCommand", func(name string, state State) {
callbackInvoked = true
callbackState = state
})

Invoke("TestInvokeCommand", Open)
time.Sleep(2 * time.Second)
Convey("Invoke Callback for Registered Command", func() {
So(callbackInvoked, ShouldBeTrue)
So(callbackState, ShouldEqual, Open)
})
})

Convey("Register a Invoke command", t, func() {
var callbackInvoked = false
var callbackState State = Close
Register("TestInvokeCommand", func(name string, state State) {
callbackInvoked = true
callbackState = state
})

Invoke("Command", Open)

Convey("Read Callback function for unknown Command", func() {
So(callbackInvoked, ShouldBeFalse)
So(callbackState, ShouldEqual, Close)
})
})
}
11 changes: 10 additions & 1 deletion hystrix/circuit.go
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,8 @@ import (
"sync"
"sync/atomic"
"time"

"github.com/afex/hystrix-go/hystrix/callback"
)

// CircuitBreaker is created for each ExecutorPool to track whether requests
Expand Down Expand Up @@ -128,6 +130,8 @@ func (circuit *CircuitBreaker) allowSingleTest() bool {
swapped := atomic.CompareAndSwapInt64(&circuit.openedOrLastTestedTime, openedOrLastTestedTime, now)
if swapped {
log.Printf("hystrix-go: allowing single test to possibly close circuit %v", circuit.Name)

callback.Invoke(circuit.Name, callback.AllowSingle)
}
return swapped
}
Expand All @@ -144,9 +148,11 @@ func (circuit *CircuitBreaker) setOpen() {
}

log.Printf("hystrix-go: opening circuit %v", circuit.Name)

circuit.openedOrLastTestedTime = time.Now().UnixNano()
circuit.open = true

callback.Invoke(circuit.Name, callback.Open)

}

func (circuit *CircuitBreaker) setClose() {
Expand All @@ -161,6 +167,9 @@ func (circuit *CircuitBreaker) setClose() {

circuit.open = false
circuit.metrics.Reset()

callback.Invoke(circuit.Name, callback.Close)

}

// ReportEvent records command metrics for tracking recent error rates and exposing data to the dashboard.
Expand Down
2 changes: 2 additions & 0 deletions hystrix/settings.go
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@ var (
DefaultLogger = NoopLogger{}
)

//Settings is used to tune circuit settings
type Settings struct {
Timeout time.Duration
MaxConcurrentRequests int
Expand Down Expand Up @@ -106,6 +107,7 @@ func getSettings(name string) *Settings {
return s
}

//GetCircuitSettings returns Circuit Settings for each command
func GetCircuitSettings() map[string]*Settings {
copy := make(map[string]*Settings)

Expand Down