[Transformers v5] IsaacForConditionalGeneration

This is a sub-issue forming part of the work in https://github.com/vllm-project/vllm/issues/38379, please read the description of this issue before beginning to work on this one.

Which test is failing?

This one has lots of issues beyond this test, it should be upstreamed so that the code is correct and stays maintained

$ pytest tests/models/test_initialization.py::test_can_initialize_large_subset[IsaacForConditionalGeneration]
...
AttributeError: 'str' object has no attribute '__args__'

Related tests:

[2026-03-27T01:46:05Z] FAILED models/multimodal/generation/test_common.py::test_single_image_models[isaac-test_case35] - ImportError: cannot import name 'SlidingWindowCache' from 'transformers.cache_utils' (/usr/local/lib/python3.12/dist-packages/transformers/cache_utils.py)
[2026-03-27T01:46:05Z] FAILED models/multimodal/generation/test_common.py::test_single_image_models[isaac-test_case36] - ImportError: cannot import name 'SlidingWindowCache' from 'transformers.cache_utils' (/usr/local/lib/python3.12/dist-packages/transformers/cache_utils.py)
[2026-03-27T01:46:05Z] FAILED models/multimodal/generation/test_common.py::test_single_image_models[isaac-test_case37] - ImportError: cannot import name 'SlidingWindowCache' from 'transformers.cache_utils' (/usr/local/lib/python3.12/dist-packages/transformers/cache_utils.py)
[2026-03-27T01:46:05Z] FAILED models/multimodal/generation/test_common.py::test_multi_image_models[isaac-test_case35] - ImportError: cannot import name 'SlidingWindowCache' from 'transformers.cache_utils' (/usr/local/lib/python3.12/dist-packages/transformers/cache_utils.py)
[2026-03-27T01:46:05Z] FAILED models/multimodal/generation/test_common.py::test_multi_image_models[isaac-test_case36] - ImportError: cannot import name 'SlidingWindowCache' from 'transformers.cache_utils' (/usr/local/lib/python3.12/dist-packages/transformers/cache_utils.py)
[2026-03-27T01:46:05Z] FAILED models/multimodal/generation/test_common.py::test_multi_image_models[isaac-test_case37] - ImportError: cannot import name 'SlidingWindowCache' from 'transformers.cache_utils' (/usr/local/lib/python3.12/dist-packages/transformers/cache_utils.py)

How to configure my environment?

It's very important that you install both vLLM and Transformers from source so that your test results reflect the current state of both libraries.

# Or your fork
git clone https://github.com/huggingface/transformers.git
git clone https://github.com/vllm-project/vllm.git

cd vllm
VLLM_USE_PRECOMPILED=1 uv pip install -e .
uv pip install -e ../transformers