knative-operator goes to Error during remove event

[ca-scribner]

During application removal, knative operator enters error state with juju status:

knative-operator/0*  error     idle   10.1.208.139         hook failed: "remove"

and debug logs of:

2023-03-07 20:51:34 INFO juju-log The OpenTelemetry Collector may not be deployed yet.This may be temporary or due to a missing otel-collector relation.
2023-03-07 20:51:34 INFO juju.worker.uniter.operation runhook.go:146 ran "update-status" hook (via hook dispatching script: dispatch)
2023-03-07 20:52:01 INFO juju-log The OpenTelemetry Collector may not be deployed yet.This may be temporary or due to a missing otel-collector relation.
2023-03-07 20:52:02 INFO juju-log Rendering manifests
2023-03-07 20:52:05 INFO juju-log Reconcile completed successfully
2023-03-07 20:52:05 INFO juju.worker.uniter.operation runhook.go:146 ran "config-changed" hook (via hook dispatching script: dispatch)
2023-03-07 20:52:06 INFO juju-log The OpenTelemetry Collector may not be deployed yet.This may be temporary or due to a missing otel-collector relation.
2023-03-07 20:52:06 INFO juju-log Rendering manifests
2023-03-07 20:52:08 INFO juju-log Reconcile completed successfully
2023-03-07 20:52:09 INFO juju.worker.uniter.operation runhook.go:146 ran "config-changed" hook (via hook dispatching script: dispatch)
2023-03-07 20:53:13 INFO juju.worker.caasunitterminationworker worker.go:82 terminating due to SIGTERM
2023-03-07 20:53:13 WARNING juju.worker.uniter.operation leader.go:123 we should run a leader-deposed hook here, but we can't yet
2023-03-07 20:53:14 INFO juju-log The OpenTelemetry Collector may not be deployed yet.This may be temporary or due to a missing otel-collector relation.
2023-03-07 20:53:14 INFO juju.worker.uniter.operation runhook.go:146 ran "stop" hook (via hook dispatching script: dispatch)
^[[A^[[B2023-03-07 20:53:15 INFO juju-log The OpenTelemetry Collector may not be deployed yet.This may be temporary or due to a missing otel-collector relation.
2023-03-07 20:53:15 INFO juju-log Rendering manifests
2023-03-07 20:53:16 INFO juju-log Rendering manifests
2023-03-07 20:53:17 WARNING juju-log Log string greater than 131071. Splitting into multiple chunks: 
2023-03-07 20:53:17 WARNING juju-log Failed to delete resources: ... TRUNCATED ... with: deployments.apps "otel-collector" not found
2023-03-07 20:53:17 ERROR juju-log Uncaught exception while in charm code:
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-knative-operator-0/charm/venv/lightkube/core/generic_client.py", line 188, in raise_for_status
    resp.raise_for_status()
  File "/var/lib/juju/agents/unit-knative-operator-0/charm/venv/httpx/_models.py", line 745, in raise_for_status
    raise HTTPStatusError(message, request=request, response=self)
httpx.HTTPStatusError: Client error '404 Not Found' for url 'https://10.152.183.1/apis/apps/v1/namespaces/kubeflow/deployments/otel-collector'
For more information check: https://httpstatuses.com/404

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "./src/charm.py", line 287, in <module>
    main(KnativeOperatorCharm)
  File "/var/lib/juju/agents/unit-knative-operator-0/charm/venv/ops/main.py", line 438, in main
    _emit_charm_event(charm, dispatcher.event_name)
  File "/var/lib/juju/agents/unit-knative-operator-0/charm/venv/ops/main.py", line 150, in _emit_charm_event
    event_to_emit.emit(*args, **kwargs)
  File "/var/lib/juju/agents/unit-knative-operator-0/charm/venv/ops/framework.py", line 355, in emit
    framework._emit(event)  # noqa
  File "/var/lib/juju/agents/unit-knative-operator-0/charm/venv/ops/framework.py", line 856, in _emit
    self._reemit(event_path)
  File "/var/lib/juju/agents/unit-knative-operator-0/charm/venv/ops/framework.py", line 931, in _reemit
    custom_handler(event)
  File "./src/charm.py", line 282, in _on_remove
    raise e
  File "./src/charm.py", line 277, in _on_remove
    delete_many(
  File "/var/lib/juju/agents/unit-knative-operator-0/charm/venv/charmed_kubeflow_chisme/lightkube/batch/_many.py", line 97, in delete_many
    client.delete(res=obj.__class__, name=obj.metadata.name, namespace=namespace)
  File "/var/lib/juju/agents/unit-knative-operator-0/charm/venv/lightkube/core/client.py", line 73, in delete
    return self._client.request("delete", res=res, name=name, namespace=namespace)
  File "/var/lib/juju/agents/unit-knative-operator-0/charm/venv/lightkube/core/generic_client.py", line 245, in request
    return self.handle_response(method, resp, br)
  File "/var/lib/juju/agents/unit-knative-operator-0/charm/venv/lightkube/core/generic_client.py", line 196, in handle_response
    self.raise_for_status(resp)
  File "/var/lib/juju/agents/unit-knative-operator-0/charm/venv/lightkube/core/generic_client.py", line 190, in raise_for_status
    raise transform_exception(e)
lightkube.core.exceptions.ApiError: deployments.apps "otel-collector" not found
2023-03-07 20:53:17 ERROR juju.worker.uniter.operation runhook.go:140 hook "remove" (via hook dispatching script: dispatch) failed: exit status 1