Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

Import tasks getting stuck mid execution

Open
  • Is something wrong?
#19 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
25/100
Issue type
Bug
Clarity
Needs clarification
Activity status
Stale
Tech stack
aws, go, rabbitmq
Domain
backend, distributed-systems

Research direction

Start with the 10-minute timeout in task/runner.go, then trace the import task's progress reporting, piping, and S3 upload paths. Reproduce or instrument the reported stuck-task cases, including the large and regular asset examples. Done means the cause is identified and import tasks no longer remain silent and are not repeatedly re-executed after RabbitMQ redelivery.

Written by the indexing model from the issue text.

Description

We have some weird cases of tasks getting stuck mid execution. These tasks just keep getting
re-executed, since the RabbitMQ will eventually timeout (after 30m) waiting on an ACK and just
terminates the connection with the client (nacking all messages it had in-flight). The task is then
re-executed as if nothing happened [1].

[1] This is also another bug that we should address. Should just fail if the task had already tried
running before and just disappeared, which we can already tell from the metadata in the API. This
is not the root cause though so we still need to investigate and fix the stuck tasks.

No logs that indicate what is wrong, but I have a light suspicion on either:

  • the "progress reporting" logic
  • the "piping" logic in the import task which sends a stream both to ffprobe and to the storage
  • the S3 upload client

On the first tasks I found this error, they were actually importing large stream recordings which take
12+ minutes to download on a good connection, due to the on-demand MP4 generation bottleneck.
It was already weird since we have a hard timeout of 10 minutes so the task runner
should have just failed the task, instead of gone silent.

Right now I just found an even weirder case though. It was from a regular "import" task, which is not
importing a recording but actually just another asset as a test that the user was making. This is the
task:

{
    "id": "51ea2a1e-618e-452d-a024-7c5a0ace266f",
    "type": "import",
    "params": {
        "import": {
            "url": "https://livepeercdn.com/asset/REDACTED/video"
        }
    },
    "status": {
        "phase": "running",
        "progress": 0.649,
        "updatedAt": 1651269956139
    },
    "userId": "REDACTED",
    "createdAt": 1650886712179,
    "outputAssetId": "4582de3b-ead3-4ffe-8b6d-b130f61290a1"
}

The asset has around 5GB and takes less than a minute to download from a good connection, so there's
no clear reason of why the task-runner is getting stuck.

Dominant language
Go
Stars
3
Forks
1
No merged PRs in 30d

Getting set up

  • Ships a Dockerfile or Docker Compose file
  • No pull request template
  • No contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from livepeer/task-runner

All issues in livepeer/task-runner

Similar issues

More Go issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.