models.list -32603 in Kubernetes

Start at session.send_and_wait and trace the bundled CLI subprocess RPC handler used for models.list. Reproduce the failure in the provided python:3.11-slim Kubernetes setup, then compare its authentication, environment, filesystem, and IPC context with local execution. Done means model listing works in the container without breaking the documented callbacks or session flow.

GitHub Copilot SDK Bug Report: models.list -32603 in Kubernetes

Summary

GitHub Copilot SDK (Python, v1.0.1) fails with JSON-RPC Error -32603: Failed to list models when running in Kubernetes containers, despite:

• ✅ Valid GitHub token with copilot scope
• ✅ Network connectivity to all required endpoints
• ✅ Successful session creation
• ✅ Works perfectly in local development environment
• ✅ Direct REST API calls to /models endpoint succeed

Environment

• SDK: github-copilot-sdk 1.0.1 (Python)
• Runtime: Docker container (python:3.11-slim)
• Platform: Kubernetes (AWS EKS, python:3.11-slim base image)
• Token: Valid GitHub token (gho_* format) with copilot scope verified via curl to API

Error Details

Query execution fails with:
  JSON-RPC Error -32603: Request models.list failed with message: Failed to list models

Stack trace indicates error occurs in:
  - session.send_and_wait({"prompt": ...})
  - SDK's bundled CLI subprocess RPC handler
  - NOT in SDK Python code

Reproduction

Local (Works ✓)

# Windows local development
python -m uvicorn app.main:app --host 0.0.0.0 --port 8080
curl -X POST http://localhost:8080/prompt \
  -H "Content-Type: application/json" \
  -d '{"prompt":"Say hello","model":"gpt-5.3-codex"}'
# → Returns assistant response successfully

Kubernetes (Fails ✗)

# Same code in K8s pod
kubectl exec -it <pod> -- python -c "
import asyncio
from copilot import CopilotClient
async def test():
    async with CopilotClient(github_token='gho_...', use_logged_in_user=False) as client:
        async with await client.create_session(model='gpt-5.3-codex') as session:
            response = await session.send_and_wait({'prompt': 'hello'})
asyncio.run(test())
"
# → RuntimeError: Failed to list models (JSON-RPC -32603)

Key Observations

1. Session creation succeeds - CopilotClient.create_session() completes without error
2. Error happens during query execution - session.send_and_wait() fails
3. Error is from bundled CLI subprocess - not SDK Python code
4. All other RPC calls work - session lifecycle, message sending succeed
5. Direct API calls work - curl to api.githubcopilot.com/models returns 200 OK
6. Token is valid - verified with curl to api.github.com/user

Attempted Workarounds (All Failed)

Root Cause Hypothesis

The SDK spawns a bundled Copilot CLI subprocess for internal JSON-RPC calls. This subprocess:

1. Works locally - has proper user context and environment
2. Fails in K8s - CLI subprocess cannot authenticate or reach API due to:
   • Different user context (subprocess run as different UID?)
   • Missing environment variables
   • File system access restrictions
   • Socket/IPC communication issues
   • Home directory/config path issues

The on_list_models callback only helps with external calls; internal RPC calls during query execution bypass this handler.

Requested Actions

1. Acknowledge - Confirm this is a known limitation in container environments
2. Document - Add warning to SDK docs about K8s/container deployment issues
3. Fix - Implement one of:
   • Make CLI subprocess authentication environment-aware
   • Allow disabling internal model validation via COPILOT_CLI_SKIP_MODEL_VALIDATION
   • Pre-authenticate CLI subprocess in SDK initialization
   • Make on_list_models handler apply to ALL internal list_models calls