Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

Crash when using remote execution on a large element

Open
  • Is something wrong?
#1,961 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
48/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Quiet
Tech stack
grpc, python
Domain
build-system, infrastructure

Research direction

Start with buildstream/_cas/cascache.py, especially CasCache.add_objects(), _fetch_tree(), and pull_tree(), then follow the remote execution path through sandbox/_sandboxremote.py. Reproduce a large-element remote build and inspect the gRPC message-size failure; done means the build completes without RESOURCE_EXHAUSTED for large directory data.

Written by the indexing model from the issue text.

Description

bug remote execution

Trying to build a large element (and after raising limit on the number of open files to work around https://github.com/apache/buildstream/issues/1842), I got another crash

    An unhandled exception occured:
    
    Traceback (most recent call last):
      File "/home/abderrahimkitouni/.local/pipx/venvs/buildstream/lib/python3.11/site-packages/buildstream/_scheduler/jobs/job.py", line 346, in child_action
        result = self.child_process()  # pylint: disable=assignment-from-no-return
                 ^^^^^^^^^^^^^^^^^^^^
      File "/home/abderrahimkitouni/.local/pipx/venvs/buildstream/lib/python3.11/site-packages/buildstream/_scheduler/jobs/elementjob.py", line 81, in child_process
        return self._action_cb(self._element)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/home/abderrahimkitouni/.local/pipx/venvs/buildstream/lib/python3.11/site-packages/buildstream/_scheduler/queues/buildqueue.py", line 55, in _assemble_element
        element._assemble()
      File "/home/abderrahimkitouni/.local/pipx/venvs/buildstream/lib/python3.11/site-packages/buildstream/element.py", line 1687, in _assemble
        collect = self.assemble(sandbox)  # pylint: disable=assignment-from-no-return
                  ^^^^^^^^^^^^^^^^^^^^^^
      File "/home/abderrahimkitouni/.local/pipx/venvs/buildstream/lib/python3.11/site-packages/buildstream/buildelement.py", line 315, in assemble
        with sandbox.batch(root_read_only=True, label="Running commands"):
      File "/usr/lib/python3.11/contextlib.py", line 144, in __exit__
        next(self.gen)
      File "/home/abderrahimkitouni/.local/pipx/venvs/buildstream/lib/python3.11/site-packages/buildstream/sandbox/sandbox.py", line 265, in batch
        batch.execute()
      File "/home/abderrahimkitouni/.local/pipx/venvs/buildstream/lib/python3.11/site-packages/buildstream/sandbox/_sandboxreapi.py", line 234, in execute
        self.sandbox._run_with_flags(
      File "/home/abderrahimkitouni/.local/pipx/venvs/buildstream/lib/python3.11/site-packages/buildstream/sandbox/sandbox.py", line 374, in _run_with_flags
        return self._run(command, flags=flags, cwd=cwd, env=env)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/home/abderrahimkitouni/.local/pipx/venvs/buildstream/lib/python3.11/site-packages/buildstream/sandbox/_sandboxreapi.py", line 101, in _run
        action_result = self._execute_action(action, flags)  # pylint: disable=assignment-from-no-return
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/home/abderrahimkitouni/.local/pipx/venvs/buildstream/lib/python3.11/site-packages/buildstream/sandbox/_sandboxremote.py", line 302, in _execute_action
        cascache.pull_tree(casremote, tree_digest)
      File "/home/abderrahimkitouni/.local/pipx/venvs/buildstream/lib/python3.11/site-packages/buildstream/_cas/cascache.py", line 251, in pull_tree
        digest = self._fetch_tree(remote, digest)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/home/abderrahimkitouni/.local/pipx/venvs/buildstream/lib/python3.11/site-packages/buildstream/_cas/cascache.py", line 617, in _fetch_tree
        dirdigests = self.add_objects(buffers=dirbuffers)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/home/abderrahimkitouni/.local/pipx/venvs/buildstream/lib/python3.11/site-packages/buildstream/_cas/cascache.py", line 357, in add_objects
        response = local_cas.CaptureFiles(request)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/home/abderrahimkitouni/.local/pipx/venvs/buildstream/lib/python3.11/site-packages/grpc/_channel.py", line 1181, in __call__
        return _end_unary_response_blocking(state, call, False, None)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/home/abderrahimkitouni/.local/pipx/venvs/buildstream/lib/python3.11/site-packages/grpc/_channel.py", line 1006, in _end_unary_response_blocking
        raise _InactiveRpcError(state)  # pytype: disable=not-instantiable
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
    	status = StatusCode.RESOURCE_EXHAUSTED
    	details = "CLIENT: Received message larger than max (7309916 vs. 4194304)"
    	debug_error_string = "UNKNOWN:Error received from peer unix:/tmp/buildstreample8mkmu/cas/casserver-l6ur9wu9.sock {grpc_message:"CLIENT: Received message larger than max (7309916 vs. 4194304)", grpc_status:8, created_time:"2024-10-09T14:46:23.774345481+01:00"}"
    >

This is again a problem with the current implementation of CasCache.add_objects().

A see two ways to trackle this (and https://github.com/apache/buildstream/issues/1842):

  • Use BatchUpdateBlobs to send the data to buildbox-casd, composing batches of appropriate size. Since all the data is already in memory.
  • Use root_directory_digest instead of tree_digest from the output directories of an action result (we may need to set output_directory_format and servers are not required to support it)
Dominant language
Python
Stars
151
Forks
49
Avg merge
3d 5h
Merged PRs (30d)
6

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from apache/buildstream

All issues in apache/buildstream

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.