Refactor fetching tasks to process

Open
#75 1 comment 0 reactions 1 assignee View on GitHub

@emyller is already working on this.

Since May 19, 2025.

Assessment

This issue has not been assessed yet.

Description

tech-debt

A couple issues have been observed around the database function responsible for retrieving and locking tasks to process.

1. Result inconsistency.

Especially when running tests, somewhat often the database query will return no results, leading to flaky test failures, e.g.

    def test_run_task_runs_task_and_creates_task_run_object_when_success(
        database: str,
        dummy_task: TaskHandler[[str, str]],
    ) -> None:
        # Given
        task = Task.create(dummy_task.task_identifier, scheduled_for=timezone.now())
        task.save(using=database)
    
        # When
        task_runs = run_tasks(database)
    
        # Then
>       assert cache.get(DEFAULT_CACHE_KEY)
E       AssertionError: assert None
E        +  where None = get('foo')
E        +    where get = <django.utils.connection.ConnectionProxy object at 0x107958080>.get

tests/unit/task_processor/test_unit_task_processor_processor.py:93: AssertionError

We became suspicious of a time zone issue so we forced UTC in both test databases as an experiment to confirm the bug. While the change led to tests passing consistently, eventually the same exception has randomly arose again.

We're still not sure what causes this. Time might be involved.

2. Locked tasks

Since the database function will select N tuples and lock them in-query, and if the associated Django queryset is not exhausted — e.g. because of an exception, we'll result in a state with tasks locked but never processed.

Related code:

https://github.com/Flagsmith/flagsmith-common/blob/ce8106418772767dcfd682bf8b95fd1d3c7c3bdc/src/task_processor/processor.py#L36-L48

The current implementation wraps running tasks within a try/except block which should prevent dangling tasks. Still, the design could be improved, and perhaps future-proofed, by using the Django ORM to somehow achieve a similar effect and efficiency.

Dominant language
Python
Stars
2
Forks
4
Avg merge
2d 9h
Merged PRs (30d)
10

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from Flagsmith/flagsmith-common

All issues in Flagsmith/flagsmith-common

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.