[BUG] Chat Completions reports vLLM cache writes as input tokens
Maintainers usually reply within 1 day
A pull request for this has already been merged.
- #1267 by @merlin-croain — merged
Assessment
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Newbie friendliness
- 95/100
Research direction
Start in lib/ruby_llm/protocols/chat_completions/chat.rb at cache_write_tokens, then run the supplied repro.rb with bundle exec ruby -Ilib repro.rb. Add or update the Chat Completions usage spec so the vLLM created_cache_tokens value is recognized. Done means usage reports 27 input tokens, 6528 cache-read tokens, and 2560 cache-write tokens.
Written by the indexing model from the issue text.
Description
Basic checks
- I searched existing issues - this hasn't been reported
- I can reproduce this consistently
- This is a RubyLLM bug, not my application code
What's broken?
vLLM reports prompt cache writes in Chat Completions usage as prompt_tokens_details.created_cache_tokens (vLLM schema). cache_write_tokens (chat.rb:193) does not read that key, so a vLLM reply gets cache_write: 0 and the written tokens stay in input.
How to reproduce
Save this as repro.rb in the repo root on main and run bundle exec ruby -Ilib repro.rb. It needs no API key. The usage numbers come from a real vLLM reply.
require 'ruby_llm'
require 'webmock'
include WebMock::API
WebMock.enable!
RubyLLM.configure do |config|
config.openai_api_key = 'test'
config.openai_api_base = 'http://vllm.test/v1'
end
stub_request(:post, 'http://vllm.test/v1/chat/completions').to_return(
headers: { 'Content-Type' => 'application/json' },
body: {
model: 'glm-5.3',
choices: [{ message: { role: 'assistant', content: '4' }, finish_reason: 'stop' }],
usage: { prompt_tokens: 9115, completion_tokens: 38,
prompt_tokens_details: { cached_tokens: 6528, created_cache_tokens: 2560 } }
}.to_json
)
chat = RubyLLM.chat(model: 'glm-5.3', provider: :openai, protocol: :chat_completions, assume_model_exists: true)
p chat.ask('2 + 2?').tokens.to_h
Expected behavior
{input_tokens: 27, output_tokens: 38, cache_read_tokens: 6528, cache_write_tokens: 2560}
What actually happened
{input_tokens: 2587, output_tokens: 38, cache_read_tokens: 6528, cache_write_tokens: 0}
I have a fix with a spec ready: one more fallback key in cache_write_tokens.
Environment
- Ruby version: ruby 3.4.4
- RubyLLM version: 2.1.0 in our app; reproduced on
mainata32aa84 - Provider: a vLLM server through the Chat Completions protocol, stubbed with WebMock above
- OS: macOS
- Dominant language
- Ruby
- Stars
- 4.5k
- Forks
- 523
- Avg merge
- 13h 36m
- Merged PRs (30d)
- 239
Getting set up
- No Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from crmne/ruby_llm
-
enhancement
Difficulty 4/5 3-5 days Newbie friendliness 50/100
crmne/ruby_llm#1314 · 1 comment · 1 reaction ·
Maintainers usually reply within 1 day
Similar issues
-
L: javascript
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
dependabot/dependabot-core#16744 ·
Maintainers usually reply within 1 day
-
area:core-lib P2 size:S
Difficulty 2/5 1-3 hours Newbie friendliness 77/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
Maintainers usually reply within 1 day
-
bug P2
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
Maintainers usually reply within 1 day