[OHE] Per-User LLM Gateway Attribution in Bifrost

Open
#615 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
35/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Active
Tech stack
python

Research direction

Start by tracing LLM Profile extra_headers and the existing forward_client_headers_to_llm_api mechanism, then identify where authenticated user and organization context is available for interactive and automated requests. Done means supported variables are validated and resolved per request, forwarded through the gateway, and covered for the stated attribution and unresolved-variable requirements.

Written by the indexing model from the issue text.

Description

Problem

Enterprise customers often route OpenHands inference through an existing LLM gateway like Bifrost for centralized governance, routing, cost management, and access control.

Today, OpenHands typically authenticates to the external gateway using a single shared virtual key configured by an administrator. As a result, the downstream gateway sees all OpenHands inference as originating from the same identity.

This creates two problems:

  1. Limited attribution and governance
    • Gateway administrators cannot easily attribute token usage or cost to individual OpenHands users.
    • Existing per-user budgets, policies, rate limits, and reporting configured in the gateway cannot be applied.
  2. Limited gateway personalization
    • Some gateways use the incoming identity or virtual key to determine which resources a user can access, including models, routing policies, or MCP servers.
    • When all OpenHands traffic shares the same identity, these user-specific policies cannot be preserved.

OpenHands already supports forwarding arbitrary extra_headers from an LLM Profile through its built-in LiteLLM gateway. When header forwarding is enabled, those headers reach the downstream gateway.

However, these headers are currently static values defined on an LLM Profile. Supporting per-user attribution therefore requires creating and maintaining a separate profile for each user.

Goal of this ticket

GOAL: Generate solution options for supporting Per-user Usage and Budget attribution in Bifrost

This should allow an external LLM gateway to apply its existing:

  • Usage and cost attribution
  • User-level budgets and rate limits
  • Routing and access policies
  • MCP or tool entitlements
  • Audit and reporting policies

Solution Options

Option 1: Dynamic extra_headers added to LiteLLM and passed into Bifrost

Extend LLM Profile extra_headers to support OpenHands context variables that are resolved dynamically for each inference request (e.g., {{user.id}})

Bifrost extra headers: https://docs.getbifrost.ai/providers/request-options#extra-headers-x-bf-eh

Example configuration:

{
  "x-bf-eh-user-id": "{{user.id}}"
}

Pros:

  • Provides per-user usage attribution in Bifrost (against a shared Virtual Key)

Cons:

  • Does not enforce user-level budgets given all usage is still against a shared Virtual Key
Option 2: Support an Org-setting that allows users to bring their own LLM Key

Pros:

  • Provides a way for both usage tracking and budget enforcement to occur on the user's specific LLM Key

Cons:

  • Hard to scale — every user needs to copy/paste their own Virtual Key manually given to them from an Admin
  • LLM Profiles and Org Settings have historically been brittle
Dominant language
Python
Stars
9
Forks
17
Avg merge
1d 7h
Merged PRs (30d)
396

Getting set up

  • Ships a Dockerfile or Docker Compose file
  • Has a pull request template
  • No contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from OpenHands/enterprise

All issues in OpenHands/enterprise

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.