knative-operator goes to Error during remove event

Start in src/charm.py around the _on_remove handler and the delete_many call shown in the traceback. Reproduce removal when the otel-collector deployment is already absent, then verify the remove hook completes successfully without treating the Kubernetes 404 as an uncaught failure.

During application removal, knative operator enters error state with juju status:

knative-operator/0*  error     idle   10.1.208.139         hook failed: "remove"

and debug logs of:

2023-03-07 20:51:34 INFO juju-log The OpenTelemetry Collector may not be deployed yet.This may be temporary or due to a missing otel-collector relation.
2023-03-07 20:51:34 INFO juju.worker.uniter.operation runhook.go:146 ran "update-status" hook (via hook dispatching script: dispatch)
2023-03-07 20:52:01 INFO juju-log The OpenTelemetry Collector may not be deployed yet.This may be temporary or due to a missing otel-collector relation.
2023-03-07 20:52:02 INFO juju-log Rendering manifests
2023-03-07 20:52:05 INFO juju-log Reconcile completed successfully
2023-03-07 20:52:05 INFO juju.worker.uniter.operation runhook.go:146 ran "config-changed" hook (via hook dispatching script: dispatch)
2023-03-07 20:52:06 INFO juju-log The OpenTelemetry Collector may not be deployed yet.This may be temporary or due to a missing otel-collector relation.
2023-03-07 20:52:06 INFO juju-log Rendering manifests
2023-03-07 20:52:08 INFO juju-log Reconcile completed successfully
2023-03-07 20:52:09 INFO juju.worker.uniter.operation runhook.go:146 ran "config-changed" hook (via hook dispatching script: dispatch)
2023-03-07 20:53:13 INFO juju.worker.caasunitterminationworker worker.go:82 terminating due to SIGTERM
2023-03-07 20:53:13 WARNING juju.worker.uniter.operation leader.go:123 we should run a leader-deposed hook here, but we can't yet
2023-03-07 20:53:14 INFO juju-log The OpenTelemetry Collector may not be deployed yet.This may be temporary or due to a missing otel-collector relation.
2023-03-07 20:53:14 INFO juju.worker.uniter.operation runhook.go:146 ran "stop" hook (via hook dispatching script: dispatch)
^[[A^[[B2023-03-07 20:53:15 INFO juju-log The OpenTelemetry Collector may not be deployed yet.This may be temporary or due to a missing otel-collector relation.
2023-03-07 20:53:15 INFO juju-log Rendering manifests
2023-03-07 20:53:16 INFO juju-log Rendering manifests
2023-03-07 20:53:17 WARNING juju-log Log string greater than 131071. Splitting into multiple chunks: 
2023-03-07 20:53:17 WARNING juju-log Failed to delete resources: ... TRUNCATED ... with: deployments.apps "otel-collector" not found
2023-03-07 20:53:17 ERROR juju-log Uncaught exception while in charm code:
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-knative-operator-0/charm/venv/lightkube/core/generic_client.py", line 188, in raise_for_status
    resp.raise_for_status()
  File "/var/lib/juju/agents/unit-knative-operator-0/charm/venv/httpx/_models.py", line 745, in raise_for_status
    raise HTTPStatusError(message, request=request, response=self)
httpx.HTTPStatusError: Client error '404 Not Found' for url 'https://10.152.183.1/apis/apps/v1/namespaces/kubeflow/deployments/otel-collector'
For more information check: https://httpstatuses.com/404

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "./src/charm.py", line 287, in <module>
    main(KnativeOperatorCharm)
  File "/var/lib/juju/agents/unit-knative-operator-0/charm/venv/ops/main.py", line 438, in main
    _emit_charm_event(charm, dispatcher.event_name)
  File "/var/lib/juju/agents/unit-knative-operator-0/charm/venv/ops/main.py", line 150, in _emit_charm_event
    event_to_emit.emit(*args, **kwargs)
  File "/var/lib/juju/agents/unit-knative-operator-0/charm/venv/ops/framework.py", line 355, in emit
    framework._emit(event)  # noqa
  File "/var/lib/juju/agents/unit-knative-operator-0/charm/venv/ops/framework.py", line 856, in _emit
    self._reemit(event_path)
  File "/var/lib/juju/agents/unit-knative-operator-0/charm/venv/ops/framework.py", line 931, in _reemit
    custom_handler(event)
  File "./src/charm.py", line 282, in _on_remove
    raise e
  File "./src/charm.py", line 277, in _on_remove
    delete_many(
  File "/var/lib/juju/agents/unit-knative-operator-0/charm/venv/charmed_kubeflow_chisme/lightkube/batch/_many.py", line 97, in delete_many
    client.delete(res=obj.__class__, name=obj.metadata.name, namespace=namespace)
  File "/var/lib/juju/agents/unit-knative-operator-0/charm/venv/lightkube/core/client.py", line 73, in delete
    return self._client.request("delete", res=res, name=name, namespace=namespace)
  File "/var/lib/juju/agents/unit-knative-operator-0/charm/venv/lightkube/core/generic_client.py", line 245, in request
    return self.handle_response(method, resp, br)
  File "/var/lib/juju/agents/unit-knative-operator-0/charm/venv/lightkube/core/generic_client.py", line 196, in handle_response
    self.raise_for_status(resp)
  File "/var/lib/juju/agents/unit-knative-operator-0/charm/venv/lightkube/core/generic_client.py", line 190, in raise_for_status
    raise transform_exception(e)
lightkube.core.exceptions.ApiError: deployments.apps "otel-collector" not found
2023-03-07 20:53:17 ERROR juju.worker.uniter.operation runhook.go:140 hook "remove" (via hook dispatching script: dispatch) failed: exit status 1