Handle ZK connection state and show in the Dashboard

[mariusz-jachimowicz-83]

• Fix Zookeeper connection backoff #54
• Fix Improve dashboard ZK connection error message #49
• show connection status after dashboard open
• watch connection state
• update connection status in dashboard
• update connection status in console
• close all things properly - properly reloading during dev
• detect connection change very quickly
• try continuously reconnect after connection lost
• understand and solve problem with connection after ZK server restart

; when I restart ZK server
; and I continously call
(defn try-connect [client]
  (println "Trying connect ZK 5s ...")
  (.. client
      (blockUntilConnected 5 TimeUnit/SECONDS)))
; then I see continously
16-09-15 11:28:32 starter-clojure INFO [org.apache.zookeeper.ClientCnxn:975] - Opening socket connection to server 127.0.0.1/127.0.0.1:2188. Will not attempt to authenticate using SASL (unknown error)
16-09-15 11:28:32 starter-clojure INFO [org.apache.zookeeper.ClientCnxn:852] - Socket connection established to 127.0.0.1/127.0.0.1:2188, initiating session
16-09-15 11:28:32 starter-clojure DEBUG [org.apache.zookeeper.ClientCnxn:892] - Session establishment request sent on 127.0.0.1/127.0.0.1:2188
16-09-15 11:28:32 starter-clojure INFO [org.apache.zookeeper.ClientCnxn:1098] - Unable to read additional data from server sessionid 0x1572d95c4ec0006, likely server has closed socket, closing socket connection and attempting reconnect

# I don't understand why ZK is not creating new session

• handle connection lost from log-subscriber

16-09-16 13:43:01 starter-clojure WARN [onyx.log.zookeeper:291] - Send ConnectionLossException by channel
--- entry
#error {
 :cause KeeperErrorCode = ConnectionLoss for /onyx/da248462-5adc-45b8-a3c8-6fb5516a84c4/log/entry-0000000026
 :via
 [{:type org.apache.zookeeper.KeeperException$ConnectionLossException
   :message KeeperErrorCode = ConnectionLoss for /onyx/da248462-5adc-45b8-a3c8-6fb5516a84c4/log/entry-0000000026
   :at [org.apache.zookeeper.KeeperException create KeeperException.java 99]}]
 :trace
 [[org.apache.zookeeper.KeeperException create KeeperException.java 99]
  [org.apache.zookeeper.KeeperException create KeeperException.java 51]
  [org.apache.zookeeper.ZooKeeper exists ZooKeeper.java 1045]
  [org.apache.curator.framework.imps.ExistsBuilderImpl$3 call ExistsBuilderImpl.java 226]
  [org.apache.curator.framework.imps.ExistsBuilderImpl$3 call ExistsBuilderImpl.java 215]
  [org.apache.curator.RetryLoop callWithRetry RetryLoop.java 107]
  [org.apache.curator.framework.imps.ExistsBuilderImpl pathInForegroundStandard ExistsBuilderImpl.java 211]
  [org.apache.curator.framework.imps.ExistsBuilderImpl pathInForeground ExistsBuilderImpl.java 205]
  [org.apache.curator.framework.imps.ExistsBuilderImpl forPath ExistsBuilderImpl.java 168]
  [org.apache.curator.framework.imps.ExistsBuilderImpl forPath ExistsBuilderImpl.java 39]
  [onyx.log.curator$exists invokeStatic curator.clj 141]
  [onyx.log.curator$exists doInvoke curator.clj 136]
  [clojure.lang.RestFn invoke RestFn.java 425]
  [onyx.log.zookeeper$eval25996$fn__25998$fn__26000 invoke zookeeper.clj 279]
  [clojure.core.async$thread_call$fn__8522 invoke async.clj 434]
  [clojure.lang.AFn run AFn.java 22]
  [java.util.concurrent.ThreadPoolExecutor runWorker ThreadPoolExecutor.java 1142]
  [java.util.concurrent.ThreadPoolExecutor$Worker run ThreadPoolExecutor.java 617]
  [java.lang.Thread run Thread.java 745]]}

• handle no node exception
• restart deployments watch after reconnected

[mariusz-jachimowicz-83]

Work in progress
Browser notification

Console notification

[lbradstreet]

The changes so far are great! Thank you for this, it'll improve the usability a lot.

[MichaelDrogalis]

Fantastic!!

[lbradstreet]

Fantastic work. Works very well for me. 👍