IST is not interrupted properly during shutdown

[temeo]

Happens every now and then in #71 testing.

Copy from: https://bugs.launchpad.net/galera/+bug/1176852

People report:

May 3 00:46:08 localhost mysqld: 130503 0:46:08 [Note] WSREP: view((empty))
May 3 00:46:08 localhost mysqld: 130503 0:46:08 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
May 3 00:46:08 localhost mysqld: 130503 0:46:08 [Note] WSREP: gcomm: closed
May 3 00:46:08 localhost mysqld: 130503 0:46:08 [Note] WSREP: Flow-control interval: [64, 64]
May 3 00:46:08 localhost mysqld: 130503 0:46:08 [Note] WSREP: Received NON-PRIMARY.
May 3 00:46:08 localhost mysqld: 130503 0:46:08 [Note] WSREP: Shifting JOINER -> OPEN (TO: 4095760)
May 3 00:46:08 localhost mysqld: 130503 0:46:08 [Note] WSREP: Received self-leave message.
May 3 00:46:08 localhost mysqld: 130503 0:46:08 [Note] WSREP: Flow-control interval: [64, 64]
May 3 00:46:08 localhost mysqld: 130503 0:46:08 [Note] WSREP: Received SELF-LEAVE. Closing connection.
May 3 00:46:08 localhost mysqld: 130503 0:46:08 [Note] WSREP: Shifting OPEN -> CLOSED (TO: 4095760)
May 3 00:46:08 localhost mysqld: 130503 0:46:08 [Note] WSREP: RECV thread exiting 0: Success
May 3 00:46:08 localhost mysqld: 130503 0:46:08 [Note] WSREP: recv_thread() joined.
May 3 00:46:08 localhost mysqld: 130503 0:46:08 [Note] WSREP: Closing slave action queue.
May 3 02:19:11 localhost mysqld: 130503 2:19:11 [Note] WSREP: IST received: 92f5a26e-aeaf-11e2-0800-fcc64d722c65:4093185
May 3 02:19:11 localhost mysqld: 130503 2:19:11 [ERROR] WSREP: gcs/src/gcs.c:_join():800: Sending JOIN failed: -103 (Software caused connection abort).
May 3 02:19:11 localhost mysqld: 130503 2:19:11 [ERROR] WSREP: Failed to JOIN the cluster after SST
May 3 02:19:12 localhost mysqld: 130503 2:19:12 [Warning] WSREP: Failed to report last committed 4094167, -77 (File descriptor in bad state)
May 3 02:19:14 localhost mysqld: 130503 2:19:14 [Warning] WSREP: Failed to report last committed 4095153, -77 (File descriptor in bad state)

• the node won't shut down until IST is over - in this case 1.5 hours. IST should be easily interruptible and the node should be able to shutdown nicely.

[temeo]

This seems to require gcs/replicator interface refactoring. The issue here is that while applier threads are handling IST events, there is no thread that will handle the leave events.