tar source plugin fails to download long urls

Open
#2,034 2 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
3/5
Estimated time
1-2 days
Newbie friendliness
72/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Quiet
Tech stack
python
Domain
build-system

Research direction

Start with src/buildstream/downloadablefilesource.py around mirror-directory computation and inspect utils.url_directory_name, then reproduce the failure with bst source fetch harry-curl.bst using the long URL from the issue. Done means the tar source can fetch that URL without triggering a file-name-too-long error.

Written by the indexing model from the issue text.

Description

bug

The following element file:

kind: import

sources:
- kind: tar
  # works
  # url: curl:download/curl-8.14.1.tar.gz#000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
  # fails
  url: curl:download/curl-8.14.1.tar.gz#0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
  ref: 6766ada7101d292b42b8b15681120acd68effa4a9660935853cf6d61f0d984d4

(with the curl alias defined in the project.conf as curl: https://curl.se/)

Fails to download:

$ bst source fetch harry-curl.bst

...

10:40:55 00:00:00 1525c594    fetch:harry-curl.bst BUG     Fetch

    An unhandled exception occured:
    
    Traceback (most recent call last):
      File "<...>/buildstream/src/buildstream/_scheduler/jobs/job.py", line 350, in child_action
        result = self.child_process()  # pylint: disable=assignment-from-no-return
                 ^^^^^^^^^^^^^^^^^^^^
      File "<...>/buildstream/src/buildstream/_scheduler/jobs/elementjob.py", line 81, in child_process
        return self._action_cb(self._element)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "<...>/buildstream/src/buildstream/_scheduler/queues/fetchqueue.py", line 75, in _fetch_not_original
        element._fetch(fetch_original=False)
      File "<...>/buildstream/src/buildstream/element.py", line 2252, in _fetch
        self.__sources.fetch()
      File "<...>/buildstream/src/buildstream/_elementsources.py", line 221, in fetch
        self.fetch_sources()
      File "<...>/buildstream/src/buildstream/_elementsources.py", line 250, in fetch_sources
        self._fetch_source(source)
      File "<...>/buildstream/src/buildstream/_elementsources.py", line 431, in _fetch_source
        source._fetch()
      File "<...>/buildstream/src/buildstream/source.py", line 1421, in _fetch
        self.__do_fetch()
      File "<...>/buildstream/src/buildstream/source.py", line 1927, in __do_fetch
        new_source.fetch(**kwargs)
      File "<...>/buildstream/src/buildstream/downloadablefilesource.py", line 341, in fetch
        sha256 = self._ensure_mirror(
                 ^^^^^^^^^^^^^^^^^^^^
      File "<...>/buildstream/src/buildstream/downloadablefilesource.py", line 399, in _ensure_mirror
        os.makedirs(self._mirror_dir, exist_ok=True)
      File "<frozen os>", line 225, in makedirs
    OSError: [Errno 36] File name too long: '<...>/.cache/buildstream/sources/tar/curl_download_curl_8_14_1_tar_gz_0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000'


Fetch failure on element: harry-curl.bst

I believe buildstream computes the mirror directory here https://github.com/apache/buildstream/blob/6558f9628aa812b8f31aac236e812d37f2be0b04/src/buildstream/downloadablefilesource.py#L277 using utils.url_directory_name which can generate filenames longer than 255 characters.

This URL is a ridiculous one but in an private project I have a file I need to download (with a very long url) where I cannot control the url I download from.

Dominant language
Python
Stars
144
Forks
45
Avg merge
20h 38m
Merged PRs (30d)
6

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from apache/buildstream

All issues in apache/buildstream

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.