Add structural `Sequence` and `Independent` composite models

Open
  • Is something wrong?
#229 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
30/100
Issue type
Feature
Clarity
Needs clarification
Activity status
Quiet
Tech stack
python
Domain
backend, distributed-systems

Research direction

No files or tests are named. Start by reading the existing CallableModel and GraphEvaluator implementations to understand composition and evaluation boundaries. Before implementation, resolve the open questions around naming, result shape, and Independent failure semantics; done means nested Sequence and Independent models express ordering without embedding evaluator-specific scheduling.

Written by the indexing model from the issue text.

Description

Add two composite CallableModels for expressing the structural shape of a workflow.

  • Sequence holds an ordered list of child CallableModels and declares that they must execute one after another. Useful when ordering matters because of external side effects — for example, an ETL stage that must finish writing before the next stage reads, or a migration that must complete before a backfill.
  • Independent holds a list of child CallableModels and declares that they have no ordering relationship to one another. Whether they actually execute concurrently, in parallel across processes, distributed across workers, or simply one after another is left to the evaluator.

Data dependencies between steps are expressed the existing ccflow way: by composing models so that one contains another. These two primitives only declare ordering constraints, and can be freely nested (a Sequence of Independents of Sequences, etc.) to build arbitrary DAGs. The resulting graph is itself a CallableModel.

Execution policy — concurrency, scheduling, distribution, retries — is the evaluator's responsibility. The same Sequence/Independent graph should run unchanged under the existing GraphEvaluator, a future Ray-based evaluator, a future Celery-based evaluator, etc.

Open questions
  • Naming. Independent captures the structural property accurately but is less immediately recognizable than something like Parallel. Worth a final naming pass before settling.
  • Result shape. Should the composite return a list keyed by position, a dict keyed by child name, or both via a configurable option? Names are friendlier for downstream consumers but require unique identifiers per child.
  • Failure semantics. For Independent, should a single child failure prevent siblings from being scheduled (fail-fast) or let them complete and report all errors? For Sequence, fail-fast is the obvious default but worth confirming. In either case the actual cancellation/scheduling behavior is the evaluator's call; the composite only declares the policy.
Dominant language
Python
Stars
44
Forks
7
Avg merge
18h 20m
Merged PRs (30d)
14

Getting set up

  • No Dockerfile or Docker Compose file
  • Has a pull request template
  • No contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from Point72/ccflow

All issues in Point72/ccflow

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.