Question: Any way to reduce memory for spider example

Open
#389 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
25/100
Issue type
Feature
Clarity
Needs clarification
Activity status
Stale
Tech stack
python, pytorch
Domain
machine-learning, observability

Research direction

Start with the README and configuration under examples/spider, then reproduce the CUDA out-of-memory failure using the reported 24 GB GPU settings. Review the run output to separate memory-related errors from duplicate logging, and identify the configuration or entry points that control each behavior. Done means a documented, reproducible configuration that runs within the available memory and a clear way to reduce unhelpful logs.

Written by the indexing model from the issue text.

Description

examples question v0.x verl

Hi, new to agent-lightning, I'm now trying to run the rl example: https://github.com/microsoft/agent-lightning/tree/v0.2.2/examples/spider

In the README, it says this example requires at least one 40GB GPU, but I only have one 24G, asking if there's some configurations to make it less memory-consuming?
I have tried to change to "gpu_memory_utilization": 0.4 and reduce some batch sizes in the config from 8 to 2, but still get the OOM error

File "/home/arda/miniforge3/envs/kai-agentic/lib/python3.12/site-packages/torch/optim/adam.py", line 181, in _init_group
    state["exp_avg_sq"] = torch.zeros_like(
                          ^^^^^^^^^^^^^^^^^
torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 180.00 MiB. GPU 0 has a total capacity of 23.52 GiB of which 117.31 MiB is free. Including non-PyTorch memory, this process has 23.25 GiB memory in use. Of the allocated memory 30.63 GiB is allocated by PyTorch, with 231.19 MiB allocated in private pools (e.g., CUDA Graphs), and 64.51 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)

Also, there's too much log during the run, which makes me confused to find the key logs, any way to hide useless/duplicate logs?

Image

Thanks so much for the help in advance!

Dominant language
Python
Stars
18.4k
Forks
1.6k
Avg merge
2d 8h
Merged PRs (30d)
27

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from microsoft/agent-lightning

All issues in microsoft/agent-lightning

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.