Support `chunk` option to enable "at-least-once" delivery
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 25/100
- Issue type
- Feature
- Clarity
- Needs clarification
- Activity status
- Stale
- Tech stack
- python
- Domain
- distributed-systems
Research direction
Start by reading the FluentSender implementation, especially its pendings buffer and emit() return-value behavior, then compare them with the Forward Protocol Specification v1 option section. Define how chunk support, retries, delivery results, and existing buffer_overflow_handler compatibility should work; done means the design and compatibility boundaries are agreed before implementation.
Written by the indexing model from the issue text.
Description
Background
According to the "Forward Protocol Specification v1", fluentd supports an option named chunk which enables at-least-once delivery of messages.
This option is very useful in cases where data loss is not acceptable.
https://github.com/fluent/fluentd/wiki/Forward-Protocol-Specification-v1#option
The problem
The current design of fluent-logger-python, however, makes it difficult to support this new option.
Specifically:
- Events are buffered inside
FluentSenderclass as a singlebytessequence (self.pendings). There is no efficient way to reconstruct a specific event from the buffer and resend it. - And this
bytessequence buffer is kinda API. So we cannot moddify the format in whichFluentSenderbuffers messages (at least, casually) or it will break many user-definedbuffer_overflow_handlers. - Also for now, we lack a handful of building blocks for supporting the "at-least-once" semantics. For example, there is no reliable mechanism for users to tell if a message has been delivered successfully [^]
So we need to ...
The bottom line is, we need to apply some architectural changes to make this library support the (newly-introduced) "at-least-once" semantics. Of course, we need to do it without breaking many existing programs.
What do you think about this? Or is there already a plan to make this library compliant with the v1 specification?
[^] Yes, FluentSender.emit() is supposed to notify this via its return value. But even if the method returns False, the message might be delivered anyway through the pending buffer, and this "retry" part is totally opaque to users.
- Dominant language
- Python
- Stars
- 457
- Forks
- 138
- PR merge metrics
- No merged PRs in 30d
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from fluent/fluent-logger-python
-
support of `asctime` Open
Difficulty 3/5 1-2 days Newbie friendliness 48/100
fluent/fluent-logger-python#210 ·
-
Difficulty 3/5 1-2 days Newbie friendliness 25/100
fluent/fluent-logger-python#196 ·
-
Difficulty 3/5 1-2 days Newbie friendliness 42/100
fluent/fluent-logger-python#195 · 1 comment ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 45/100
fluent/fluent-logger-python#194 ·
-
Difficulty 4/5 3-5 days Newbie friendliness 25/100
fluent/fluent-logger-python#193 · 3 comments ·
All issues in fluent/fluent-logger-python
Similar issues
-
essnmx good first issue
Difficulty 1/5 Under an hour Newbie friendliness 95/100
-
[Feature] 奇物选择添加优先级 Open
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
syfoud/Simulated_Scepter#174 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
Giskard-AI/giskard-oss#2840 · 1 comment ·
-
A claim comment carrying the issue number is silently declined while the workflow reports success Openarea: repo bug perceived difficulty: 2
Difficulty 2/5 1-3 hours Newbie friendliness 70/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
yeti-platform/yeti#1380 ·