Ensure all state is shared across NiFi 2 cluster members
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 20/100
- Issue type
- Feature
- Clarity
- Needs clarification
- Activity status
- Stale
- Tech stack
- kubernetes
- Domain
- distributed-systems
Research direction
Start by reviewing NIFI-7246 and the linked issues 809 and 811 to identify which NiFi state remains local to cluster members. Then discuss the feasibility with NiFi maintainers and investigate the proposed NiFi fork; done means documenting the blockers or achieving shared state that supports load-balanced members without sticky sessions.
Written by the indexing model from the issue text.
Description
Part of https://github.com/stackabletech/nifi-operator/issues/809.
Why?
This opens the door for fixing the Oauth2 implementation which solves problems for many (including us).
Current state
Although it is termed a NiFi cluster, cluster members are not exactly replicas, and hold local state.
One example of local state is a Site-to-Site flowfile transfer to another cluster. Currently, to get the status of a transfer, you must connect to the node that you initiated the transfer on.
This might be the reason for the following:
- A JWT issuer per cluster memeber (tokens cannot be used on other cluster members)
- SNI hostname checks
Future state
All cluster state should be shared to cluster members so that cluster members can actually be replicas, and therefore sit behind a load balancer without needing sticky-sessions.
Recommended approach
- Try to find any reason why some state cannot be shared across cluster members.
- Start a conversation with NiFi maintainers to gauge interest in accepting this fix upstream.
- Fork NiFi and try to share the currently unshared state across cluster members.
Unknowns
- Which other state is local only? Perhaps the answer can be found in NIFI-7246
- Dominant language
- Rust
- Stars
- 48
- Forks
- 12
- Avg merge
- 11h 26m
- Merged PRs (30d)
- 10
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from stackabletech/nifi-operator
-
Difficulty 3/5 1-2 days Newbie friendliness 68/100
stackabletech/nifi-operator#956 ·
-
Difficulty 4/5 3-5 days Newbie friendliness 45/100
stackabletech/nifi-operator#879 ·
-
customer-request type/bug
Difficulty 3/5 1-2 days Newbie friendliness 42/100
stackabletech/nifi-operator#842 · 1 comment ·
-
size/M type/security
Difficulty 5/5 Over a week Newbie friendliness 25/100
stackabletech/nifi-operator#811 · 1 comment ·
-
epic size/XL type/security
Difficulty 5/5 Over a week Newbie friendliness 15/100
stackabletech/nifi-operator#809 ·
All issues in stackabletech/nifi-operator
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
TheLarkInn/aipm#2413 ·
-
documentation
Difficulty 1/5 Under an hour Newbie friendliness 90/100
alexgorbatchev/simple-ptt#15 ·
-
tooling
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
-
todo:ticket
Difficulty 2/5 1-3 hours Newbie friendliness 70/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
taikoxyz/taiko-mono#22168 · 1 comment ·