Openstack Discoverer logs incorrect update

[stevesloka]

The openstack discoverer logs updates for services when no updates have happened. The following example show the lb being added, then shows an update on each cycle when it wasn't updated in openstack.

time="2018-05-08T13:10:17Z" level=info msg="Successfully handled: add service 'demo/lb1-2ad78ab3-aa94-43e4-a764-194a67601d16-openstack'"
time="2018-05-08T13:10:17Z" level=info msg="Successfully handled: add endpoints 'demo/lb1-2ad78ab3-aa94-43e4-a764-194a67601d16-openstack'"
time="2018-05-08T13:10:47Z" level=info msg="Successfully handled: update service 'demo/lb1-2ad78ab3-aa94-43e4-a764-194a67601d16-openstack'"
time="2018-05-08T13:11:17Z" level=info msg="Successfully handled: update service 'demo/lb1-2ad78ab3-aa94-43e4-a764-194a67601d16-openstack'"
time="2018-05-08T13:11:47Z" level=info msg="Successfully handled: update service 'demo/lb1-2ad78ab3-aa94-43e4-a764-194a67601d16-openstack'"
time="2018-05-08T13:12:19Z" level=info msg="Successfully handled: update service 'demo/lb1-2ad78ab3-aa94-43e4-a764-194a67601d16-openstack'"
time="2018-05-08T13:12:47Z" level=info msg="Successfully handled: update service 'demo/lb1-2ad78ab3-aa94-43e4-a764-194a67601d16-openstack'"
time="2018-05-08T13:13:21Z" level=info msg="Successfully handled: update service 'demo/lb1-2ad78ab3-aa94-43e4-a764-194a67601d16-openstack'"
time="2018-05-08T13:13:47Z" level=info msg="Successfully handled: update service 'demo/lb1-2ad78ab3-aa94-43e4-a764-194a67601d16-openstack'"
time="2018-05-08T13:14:17Z" level=info msg="Successfully handled: update service 'demo/lb1-2ad78ab3-aa94-43e4-a764-194a67601d16-openstack'"

[alexbrand]

While running performance tests, this issue was very prominent. The discoverer is enqueuing updates at a faster rate than the workers are dequeuing items (Related to #140). This results in memory consumption and an ever growing queue.

Memory:

Queue:

Bumping up the priority on this.

[alexbrand]

Currently, we are comparing the synced services/endpoints that are in Gimbal with the services/endpoints we produce after translation using reflect.DeepEquals.

The problem is that the resource we get from the Gimbal API Server includes creation timestamp, resource version, UID, etc. while the one we create during the translation process does not. This causes the equality comparison to return false, and thus the discoverer thinks it needs to perform an update.

One solution is to compare the fields we care about, instead of comparing the entire struct. The downside is that we create coupling where a change to the translation process will require a change to the diff process. For example, if we start translating a new field, we have to make sure we update the diff function to include that new field in the comparison. This is problematic because over time it could get out of sync, causing bugs in the diff process. I wonder what's the best way to mitigate this. Maybe a unit test?

With that said, I am not sure if there are other solutions.

[alexbrand]

Turns out I jumped the gun. We are already only comparing the fields that we care about. The actual problem has been documented in the PR (#149) with the fixes.