Non-blocking event recorder
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 30/100
Research direction
Start with runtime/events/recorder.go, especially the recorder entry point and the HTTP request around lines 211-220, to understand where reconciliation blocks. Compare the proposed goroutine and event-processor approaches, including ordering, retries, buffering, and deduplication. Done means event recording no longer makes reconcilers wait on a slow or unavailable webhook server.
Written by the indexing model from the issue text.
Description
The event recorder currently runs in a blocking fashion due to which it can affect the reconcilers when they emit an event and the event webhook server takes time to respond. The http client waits and retries on failure. This results in the reconciler to wait until the event is posted. Sometimes reconcilers time out waiting for the event emitting to complete, resulting in failed reconciliation. A non-blocking event recorder would help prevent the reconcilers to be affected by failure in event recording.
Following are some ideas to address this issue:
non-blocking request to the event webhook server
A quick and simple change to solve the immediate issue would be to make the http request to the event webhook server non-blocking by running it in a goroutine. This would unblock the reconciler but doesn't guarantee the ordering of the requests. If the webhook server is offline and the reconciliation is failing, the reconcilers retrying may create multiple of these goroutines which may keep retrying to post the event with back-off. Due to the variation in the back-off duration, when the webhook server becomes available, the events would be posted out of order. Since there's no de-duplication at the event source level, the webhook server will have to serve all the accumulated event requests and do de-duplication on its own. This may result in the creation of too many goroutines that are trying to do the same thing. But the goroutines can be configured to fail after certain number of attempts to ensure they get cleaned up.
per controller event processor
Another approach would be to introduce event processor per controller. The events package can provide some API to run an event processor, typically in the main.go file before setting up the reconcilers. The event recorder in the reconcilers would send event to the event processor through a buffered channel. The event processor would collect the events to be posted to the event webhook server, categorize them and process them based on certain strategies. Since all the events go through a central events processor, it can be used to add more functionalities in the event source. Order of the events can be maintained. Event de-duplication can be done at the source and spamming the event server can be avoided. If the event server isn't ready, the event processor can perform one health check and hold all the event processing. If the event buffer gets filled, it can drop certain events based on certain strategies. More interesting things can be done at the source of events centrally at the controller level.
This may be similar to the events notification broadcaster in kubernetes apimachiner https://github.com/kubernetes/apimachinery/blob/fd8a60496be5e4ce19d586738adb48ac6fa66ef9/pkg/watch/mux.go#L43 .
Some other variation of event processor could be to run event processor per reconciler or even per object and that'll create opportunities to handle the events in different ways. Like a tenant configuring the events related to their objects to be sent to their own event server which they manage.
- Dominant language
- Go
- Stars
- 63
- Forks
- 162
- Avg merge
- 9h 54m
- Merged PRs (30d)
- 29
Getting set up
- No Dockerfile or Docker Compose file
- No pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from fluxcd/pkg
-
ssa: wait errors drop the kstatus message, so health check failures don't say whyPossibly taken @schdief claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
Maintainers usually reply within 1 day
-
ssa/jsondiff: honor ignored fields before dry-run validationPossibly taken @lubronzhan claimed this 31 days ago. Open
Difficulty 4/5 3-5 days Newbie friendliness 30/100
Maintainers usually reply within 1 day
-
Difficulty 4/5 3-5 days Newbie friendliness 35/100
Maintainers usually reply within 1 day
-
Difficulty 3/5 1-2 days Newbie friendliness 35/100
Maintainers usually reply within 1 day
-
Improve `make release` to support major version bumpsPossibly taken @Iam-Karan-Suresh claimed this 175 days ago. Openarea/ci
fluxcd/pkg#1172 · 3 comments · 1 reaction · 1 assignee ·
Maintainers usually reply within 1 day
Similar issues
-
Pin and verify Security Insights schema in lintinsights targetPossibly taken @imshubham22apr-gif claimed this 5 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
gemaraproj/gemara#618 ·
Maintainers usually reply within 2 days
-
Difficulty 2/5 1-3 hours Newbie friendliness 64/100
a2aproject/a2a-go#597 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
Maintainers usually reply within 4 days
-
scopes: GetAllowedTopics, GetProtectedTopics, and GetScopedTopics include empty topic names and do not trim whitespace charactersPossibly taken @patelshivani2283-lab claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Maintainers usually reply within 1 day
-
area/networking kind/bug
Difficulty 2/5 1-3 hours Newbie friendliness 69/100
siderolabs/talos#14972 ·
Maintainers usually reply within 1 day