resource synchronization might affect kubernetes API

[rg2011]

When kube-mgmt is synchronizing resources, it will restart syncronization whenever an update to OPA fails. Each time synchronization restarts, it will create a new ListWatch and ask the kubernetes API for a full resource list across all namespaces.

This can happen as often as once per second, because OPA errors always reset the backoff delay to the minimum (1 second currently):

kube-mgmt/pkg/data/generic.go

Line 93 in 01cab16

delay = backoffMin

So a faulty OPA or kube-mgmt service can hit the API servers quite hard, depending on the number of resources being watched.

Kube-mgmt should probably use an informer and resync from the informer cache instead.