Some abort jobs are failing on flower

Open
  • Is something wrong?
#149 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
3/5
Estimated time
1-2 days
Newbie friendliness
25/100
Issue type
Bug
Clarity
Needs clarification
Activity status
Stale
Tech stack
git, python
Domain
backend

Research direction

Start with ridgeback/toil_orchestrator/tasks.py at abort_job, then trace submitter/jobsubmitter.py through submit, _command_line, _dump_app_inputs, and resolve. Reproduce or inspect the failing abort task and determine why cloning leaves an existing non-empty cwl-commandlinetools directory; done means abort jobs no longer fail for this condition.

Written by the indexing model from the issue text.

Description

bug

http://voyager:4001/task/3d11b7cb-4e22-4ffe-942d-00c8a936cfc3

Args: ('613713e3-5a53-4e82-8bf7-433110d52009',)

Traceback (most recent call last): File "/juno/work/ci/prod_ridgeback/venv/lib/python3.7/site-packages/celery/app/trace.py", line 412, in trace_task R = retval = fun(*args, **kwargs) File "/juno/work/ci/prod_ridgeback/venv/lib/python3.7/site-packages/celery/app/trace.py", line 704, in protected_call return self.run(*args, **kwargs) File "/srv/services/ridgeback/code/ridgeback/toil_orchestrator/tasks.py", line 145, in abort_job self.retry(exc=e, countdown=10) File "/juno/work/ci/prod_ridgeback/venv/lib/python3.7/site-packages/celery/app/task.py", line 706, in retry raise_with_context(exc) File "/srv/services/ridgeback/code/ridgeback/toil_orchestrator/tasks.py", line 116, in abort_job external_job_id, job_store_dir, job_work_dir, job_output_dir = submitter.submit() File "/srv/services/ridgeback/code/ridgeback/submitter/jobsubmitter.py", line 68, in submit command_line = self._command_line() File "/srv/services/ridgeback/code/ridgeback/submitter/jobsubmitter.py", line 151, in _command_line app_location, inputs_location = self._dump_app_inputs() File "/srv/services/ridgeback/code/ridgeback/submitter/jobsubmitter.py", line 99, in _dump_app_inputs app_location = self.app.resolve(self.job_work_dir) File "/srv/services/ridgeback/code/ridgeback/submitter/jobsubmitter.py", line 42, in resolve git.Git(location).clone(self.github, '--branch', self.version, '--recurse-submodules') File "/juno/work/ci/prod_ridgeback/venv/lib/python3.7/site-packages/git/cmd.py", line 542, in return lambda *args, **kwargs: self._call_process(name, *args, **kwargs) File "/juno/work/ci/prod_ridgeback/venv/lib/python3.7/site-packages/git/cmd.py", line 1005, in _call_process return self.execute(call, **exec_kwargs) File "/juno/work/ci/prod_ridgeback/venv/lib/python3.7/site-packages/git/cmd.py", line 822, in execute raise GitCommandError(command, status, stderr_value, stdout_value) git.exc.GitCommandError: Cmd('git') failed due to: exit code(128) cmdline: git clone https://github.com/mskcc/cwl-commandlinetools --branch master --recurse-submodules stderr: 'fatal: destination path 'cwl-commandlinetools' already exists and is not an empty directory.'

Dominant language
Python
Stars
1
Forks
2
No merged PRs in 30d

Getting set up

  • Ships a Dockerfile or Docker Compose file
  • No pull request template
  • No contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from mskcc/ridgeback

All issues in mskcc/ridgeback

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.