Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

Refactor how we check job status

Open
  • Is something wrong?
#166 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
25/100
Issue type
Refactor
Clarity
Needs clarification
Activity status
Stale
Tech stack
python
Domain
backend, distributed-systems

Research direction

The issue names check_status, submit_pending_jobs, JobSubmitter, Git.clone, command_line_tools, and the LSF client at batch_systems/lsf_client/lsf_client.py. Start by mapping job status transitions and queue and retry behavior across those entry points. Done would require an agreed scope, defined concurrency and failure semantics, and tests for the listed edge cases.

Written by the indexing model from the issue text.

Description

I think we need to ensure that we're covering all edge cases as we move a job from CREATING to COMPLETED.

A couple of ideas/questions:

  • There should only be one instance of check_status running at a time. When there are high loads check_status can hit >5 minutes, but the beat is every 1 minute.
  • We need a better way to restart celery. What's been happening is that the Worker will not shutdown gracefully and leave a PID so that it can no longer be started. The beat will restart and continue to queue jobs. When the worker is finally restarted, all of the beat jobs will run at once. This means that check_status will run N times, milliseconds apart, same with submit_pending_jobs, etc.
  • Separate queues for check_status/submit job workers
  • a status should only be updated in a single place
  • Ideally for FAILED we'd just have the decorator catch exceptions
  • ensure jobs that shouldn't run concurrently, can never do so (put them in their own queue with a single worker? memcache lock?)
  • Maybe implement a state machine for status changes to ensure statuses go in the order we're expecting.
  • Need a better way to handle when a git clone fails,Git.clone only checks to see if the directory exists, it doesn't verify the contents. A task will retry if git clone fails and then it'll submit a job with an empty git directory
  • JobSubmitter only requires the external_id to check for status, we should just call LSFClient instead
  • Move logic for what happens when a status is changed to the model. E.g. if a Job goes from RUNNING to COMPLETED, we can update the finished_date in the save function similar to how it's done in Beagle -- so the business logic in the task is easier to read
  • Write tests to cover all edge cases.
  • avoid functions with side effects like update_message
  • I'm not sure what command_line_tools does
  • Batch bjobs here: https://github.com/mskcc/ridgeback/blob/8b4891708dbaeca36146f0c319461f22a7681908/batch_systems/lsf_client/lsf_client.py#L205
  • How do we prevent Voyager from slowing down when there are a lot of jobs?
Dominant language
Python
Stars
1
Forks
2
No merged PRs in 30d

Getting set up

  • Ships a Dockerfile or Docker Compose file
  • No pull request template
  • No contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from mskcc/ridgeback

All issues in mskcc/ridgeback

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.