Dataflow Listener (bigger project)
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 12/100
- Issue type
- Feature
- Clarity
- Mostly clear
- Activity status
- Stale
- Tech stack
- python
- Domain
- stream-processing
Research direction
Start by reading logger/listener/listener.py to see the current reader/transform/writer model, then listen.py and run_logger.py to see how configs are parsed and run. The sketched DataflowNode class in the issue is only a starting point, and the design (threads or processes, config syntax) is not settled. Work here needs a maintainer decision first, so this is not a good first contribution.
Written by the indexing model from the issue text.
Description
This would be an alternative to the Listener class in logger/listener/listener.py. The current listener implements a parallel-reader, serial-transform, parallel-writer processing model that is inefficient in terms of not allowing computed values to be used in different processing streams. A pure dataflow model would consist of a set of nodes, each of which defined an input queue, a function (whether it be reader/transform/writer/whatever) and a list of nodes into whose queues it should place its result.
In addition to a stand-alone dataflow runner to parallel listen.py, we would enhance the config file syntax so that run_logger.py could recognize, parse and execute dataflow configs as well as the existing listener configs.
Depending on design (or invocation), the processing of dataflow each node could be a separate thread, or even process (because the multiprocessing module supports interprocess Queues) with an abstract wrapper that looked something like what's below:
# An very-initial first sketch of dataflow
class DataflowNode:
def __init__(self,
name='', # convenient display name
component=None, # the reader/transform/writer itself
method=None, # the method we call to process input
use_input_queue=False, # does method take input from queue?
extra_args=None, # additional args the method might need
threadsafe=False, # is the method threadsafe?
max_queue_size=0): # how big to let input queue get
pass
def successors(self):
pass
def add_successor(self, node):
pass
def push_queue(self, record):
"""Externally-visible method to add a record to this node's input queue."""
pass
def run(self):
"""Start the dataflow loop."""
pass
def quit(self):
"""Shut down nicely."""
pass
############################
def _generate(self):
"""Get next record from queue (if necessary) and do whatever we need
to generate our next output record."""
pass
def _pop_queue(self):
"""Internal method to get the next record from our input queue."""
pass
def _push_to_successors(self, record):
pass
- Dominant language
- Python
- Stars
- 57
- Forks
- 30
- Avg merge
- 3h 14m
- Merged PRs (30d)
- 56
Getting set up
- No Dockerfile or Docker Compose file
- No pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from OceanDataTools/openrvdas
-
enhancement
Difficulty 2/5 1-3 hours Newbie friendliness 67/100
OceanDataTools/openrvdas#760 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 35/100
OceanDataTools/openrvdas#758 · 1 comment ·
Maintainers usually reply within 1 day
-
enhancement
Difficulty 5/5 Over a week Newbie friendliness 25/100
OceanDataTools/openrvdas#748 · 1 comment ·
Maintainers usually reply within 1 day
-
enhancement
Difficulty 4/5 3-5 days Newbie friendliness 58/100
OceanDataTools/openrvdas#743 ·
Maintainers usually reply within 1 day
-
enhancement
Difficulty 4/5 3-5 days Newbie friendliness 35/100
OceanDataTools/openrvdas#721 ·
Maintainers usually reply within 1 day
All issues in OceanDataTools/openrvdas
Similar issues
-
automated endpoint-dead needs-curation
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
almanac-data/climate-almanac#138 ·
Maintainers usually reply within 1 day
-
level/subtask type/enhancement
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
wazuh/wazuh-virtual-machines#1091 ·
Maintainers usually reply within 1 day
-
Python: Bug: ChatMessageContent.from_element drops AudioContent and BinaryContent into raw TextContentPossibly taken @halcyon-past claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
microsoft/semantic-kernel#14824 · 1 comment ·
Maintainers usually reply within 3 days
-
help wanted
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
stack314/django-neolith#142 · 1 comment ·
Maintainers usually reply within 1 day
-
Difficulty 1/5 Under an hour Newbie friendliness 82/100
LedgerHQ/app-bitcoin-legacy#306 ·