tar source plugin fails to download long urls
Nobody has claimed this yet.
Assessment
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Newbie friendliness
- 72/100
- Issue type
- Bug
- Clarity
- Clearly specified
- Activity status
- Quiet
- Tech stack
- python
- Domain
- build-system
Research direction
Start with src/buildstream/downloadablefilesource.py around mirror-directory computation and inspect utils.url_directory_name, then reproduce the failure with bst source fetch harry-curl.bst using the long URL from the issue. Done means the tar source can fetch that URL without triggering a file-name-too-long error.
Written by the indexing model from the issue text.
Description
The following element file:
kind: import
sources:
- kind: tar
# works
# url: curl:download/curl-8.14.1.tar.gz#000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
# fails
url: curl:download/curl-8.14.1.tar.gz#0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
ref: 6766ada7101d292b42b8b15681120acd68effa4a9660935853cf6d61f0d984d4
(with the curl alias defined in the project.conf as curl: https://curl.se/)
Fails to download:
$ bst source fetch harry-curl.bst
...
10:40:55 00:00:00 1525c594 fetch:harry-curl.bst BUG Fetch
An unhandled exception occured:
Traceback (most recent call last):
File "<...>/buildstream/src/buildstream/_scheduler/jobs/job.py", line 350, in child_action
result = self.child_process() # pylint: disable=assignment-from-no-return
^^^^^^^^^^^^^^^^^^^^
File "<...>/buildstream/src/buildstream/_scheduler/jobs/elementjob.py", line 81, in child_process
return self._action_cb(self._element)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "<...>/buildstream/src/buildstream/_scheduler/queues/fetchqueue.py", line 75, in _fetch_not_original
element._fetch(fetch_original=False)
File "<...>/buildstream/src/buildstream/element.py", line 2252, in _fetch
self.__sources.fetch()
File "<...>/buildstream/src/buildstream/_elementsources.py", line 221, in fetch
self.fetch_sources()
File "<...>/buildstream/src/buildstream/_elementsources.py", line 250, in fetch_sources
self._fetch_source(source)
File "<...>/buildstream/src/buildstream/_elementsources.py", line 431, in _fetch_source
source._fetch()
File "<...>/buildstream/src/buildstream/source.py", line 1421, in _fetch
self.__do_fetch()
File "<...>/buildstream/src/buildstream/source.py", line 1927, in __do_fetch
new_source.fetch(**kwargs)
File "<...>/buildstream/src/buildstream/downloadablefilesource.py", line 341, in fetch
sha256 = self._ensure_mirror(
^^^^^^^^^^^^^^^^^^^^
File "<...>/buildstream/src/buildstream/downloadablefilesource.py", line 399, in _ensure_mirror
os.makedirs(self._mirror_dir, exist_ok=True)
File "<frozen os>", line 225, in makedirs
OSError: [Errno 36] File name too long: '<...>/.cache/buildstream/sources/tar/curl_download_curl_8_14_1_tar_gz_0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000'
Fetch failure on element: harry-curl.bst
I believe buildstream computes the mirror directory here https://github.com/apache/buildstream/blob/6558f9628aa812b8f31aac236e812d37f2be0b04/src/buildstream/downloadablefilesource.py#L277 using utils.url_directory_name which can generate filenames longer than 255 characters.
This URL is a ridiculous one but in an private project I have a file I need to download (with a very long url) where I cannot control the url I download from.
- Dominant language
- Python
- Stars
- 144
- Forks
- 45
- Avg merge
- 20h 38m
- Merged PRs (30d)
- 6
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from apache/buildstream
-
logging
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
apache/buildstream#2150 · 3 comments ·
-
question
Difficulty 2/5 1-3 hours Newbie friendliness 64/100
apache/buildstream#2143 · 2 comments · 1 reaction ·
-
Difficulty 3/5 1-2 days Newbie friendliness 55/100
apache/buildstream#2194 ·
-
Modification time of an element with an open workspace propagates into consumer element's sandbox Openbug
Difficulty 3/5 1-2 days Newbie friendliness 58/100
apache/buildstream#2186 · 2 comments · 2 reactions ·
-
Difficulty 3/5 1-2 days Newbie friendliness 55/100
apache/buildstream#2184 · 1 comment ·
All issues in apache/buildstream
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 86/100
EleutherAI/lm-evaluation-harness#4207 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
ClickHouse/clickhouse-connect#1057 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 70/100
open-telemetry/sig-end-user#406 ·
-
bug ci good first issue
Difficulty 2/5 1-3 hours Newbie friendliness 88/100