[BUG] `Tool#ask` does not check that a stored answer belongs to the question it asks now
Los mantenedores suelen responder en 1 día
Nadie ha tomado este issue todavía.
Evaluación
- Dificultad
- 4/5
- Tiempo estimado
- 3-5 días
- Aptitud para principiantes
- 55/100
- Tipo de issue
- Error
- Claridad
- Bien especificado
- Estado de actividad
- Activo
- Stack tecnológico
- ruby
- Área
- backend-api-design
Línea de trabajo
Start with lib/ruby_llm/tool.rb and inspect how Tool#ask retrieves stored requests by position. Use the two reproductions in the issue to check skipped questions and changed choices; the report does not name a test file. Done means a stored answer is not returned when the current question differs, and the tool asks again as proposed.
Escrito por el modelo de indexación a partir del texto del issue.
Descripción
Basic checks
- I searched existing issues - this hasn't been reported
- I can reproduce this consistently
- This is a RubyLLM bug, not my application code
What's broken?
When a question is answered, execute runs again from the top, and ask finds the stored answer by position only: the first ask of the run gets the answer at index 0, the second the answer at index 1 (lib/ruby_llm/tool.rb):
requests = Array(state['input']&.[]('requests'))
return Question.from_h(requests[index]).asked_answer(fields) if requests[index]&.[]('response')
The stored question is not compared with the question asked now. So ask has no protection against a stale answer: if a later run asks a different question at the same position, the tool gets the answer the person gave to the other question. Nothing is raised, and nothing reports the mismatch.
I know the limits of this report. The tools guide says "A call can ask several questions, one after another, and each run asks them in the same order", and the first case below breaks that rule. The second case keeps the order: only the choices change. In both, the rule is easy to break without noticing (one if around an ask, or choices built from data that changes), and the result is silent: a person's "no" reaches the tool as "yes". The stored question lies next to the answer, so RubyLLM can see the mismatch.
How to reproduce
No network: the script replaces the provider's complete with two prepared replies. Save it as issue_repro.rb in a checkout of main and run ruby -I lib issue_repro.rb.
require 'ruby_llm'
RubyLLM.configure { |config| config.anthropic_api_key = 'test' }
MODEL = 'claude-haiku-4-5'
DRAFT = { present: true }
SEATS = %w[window aisle].dup
# Case 1: a question is skipped on a later run.
class CleanDeck < RubyLLM::Tool
description 'Cleans up a deck'
def execute
keep = ask('Keep the draft?', choices: %w[yes no]) if DRAFT[:present]
wipe = ask('Delete every card?', choices: %w[yes no])
"keep the draft: #{keep.inspect}, delete every card: #{wipe.inspect}"
end
end
# Case 2: one question, in the same place; its choices come from data.
class PickSeat < RubyLLM::Tool
description 'Picks a seat'
def execute
seat = ask('Which seat?', choices: SEATS)
"seat: #{seat.inspect}, offered now: #{SEATS.inspect}"
end
end
# A chat whose model calls +tool+ once, then says "Done".
def chat_calling(tool)
chat = RubyLLM::Chat.new(model: MODEL)
call = RubyLLM::ToolCall.new(id: 'call_1', name: tool.new.name, arguments: {})
replies = [
RubyLLM::Message.new(role: :assistant, content: '', tool_calls: { 'call_1' => call }),
RubyLLM::Message.new(role: :assistant, content: 'Done', model: MODEL, input_tokens: 1, output_tokens: 1)
]
chat.provider.define_singleton_method(:complete) { |*, **, &_block| replies.shift }
chat.with_tools(tool).ask('Go')
chat
end
def answer(chat, value)
question = chat.pending_questions.first
puts "asked: #{question.message} #{question.fields.first.choices.inspect} -> the person answers #{value}"
question.answer(value)
end
def result(chat) = chat.messages.find { |message| message.role == :tool }.content
chat = chat_calling(CleanDeck)
answer(chat, 'yes')
chat.complete
answer(chat, 'no')
DRAFT[:present] = false # the draft is published elsewhere while the person decides
chat.complete
puts "tool result: #{result(chat)}\n\n"
chat = chat_calling(PickSeat)
answer(chat, 'aisle')
SEATS.replace(%w[front back]) # the seat map changes while the person decides
chat.complete
puts "tool result: #{result(chat)}"
Expected behavior
A tool does not get an answer that the person gave to another question.
- Case 1: the tool does not get "yes" for "Delete every card?". The person answered "no" to that question.
- Case 2: the tool does not get "aisle" when it offers "front" and "back".
What actually happened
asked: Keep the draft? ["yes", "no"] -> the person answers yes
asked: Delete every card? ["yes", "no"] -> the person answers no
tool result: keep the draft: nil, delete every card: "yes"
asked: Which seat? ["window", "aisle"] -> the person answers aisle
tool result: seat: "aisle", offered now: ["front", "back"]
Case 1: on the third run "Keep the draft?" is skipped, "Delete every card?" becomes the first ask, and it gets the answer stored at index 0.
Case 2: the answer was checked against the choices of the first run, and is returned for the choices of the second.
Two more cases give the same kind of result. A run that asks another question at the same position ("Keep the draft?", then "Delete every card in the deck?") gets "yes". A run that asks for fields where the first run asked for a choice gets {answer: "aisle"}, with none of the declared fields.
Answers do not cross between tool calls: two calls in one round, answered in reverse order, each get their own answer.
Proposed solution
Compare the question before ask returns its answer. ask has every argument it needs to build the question it asks now, and the stored question keeps the same data (params: the message, the schema of the answer, the URL). Return the stored answer, or the stored decline, only if the two are equal. If they differ, replace the stored questions from that position on with the question asked now, under a new id, and pause on it.
In case 1, "Delete every card?" is asked once more on the third run. In case 2, the person picks between "front" and "back".
Ask again or raise: this is a choice. I propose to ask again, because a changed question is one the person did not answer yet (a price went up while they were away). An error is the alternative: it shows unstable tool code faster and can never ask twice.
What this does not do, and what it costs:
- It is a guard, not an identity. Two questions with equal text and equal choices about different things still match.
- A tool whose wording differs on every run while the meaning stays the same (the current time in the text, a sentence written by a model, choices in another order) is asked again on every run. Today it finishes.
- The block of fields runs on every run, to build the schema to compare. Today it does not run for an answered question.
- The signature of
askdoes not change. Theparamsof all three kinds of question are equal to themselves after a JSON round trip, so I expect the comparison to hold for a persisted chat; I did not run one. - The change stays inside
Tool#ask. Questions from an MCP server and a Task returned after a question keep their behavior. A tool that calls another tool that asks is a different problem and is out of scope.
If you agree with the comparison, I can send a pull request that adds it.
Environment
- Ruby version: 4.0.9
- RubyLLM version: 2.2.0,
mainat5d041358 - Provider: Anthropic, stubbed (the script replaces
complete) - An in-memory chat. I did not run an
acts_as_chatrecord; it goes through the sameTool#ask. - OS: macOS (Darwin 25.6.0)
- Lenguaje dominante
- Ruby
- Estrellas
- 4.5k
- Forks
- 537
- Merge medio
- 7 h 30 min
- PR fusionados (30 d)
- 378
Preparar el entorno
- Sin Dockerfile ni archivo de Docker Compose
- Tiene una plantilla de pull request
- Leer la guía de contribución
Primeros pasos
- Lee el issue completo y luego la guía de contribución del proyecto.
- Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
- Haz un fork del repositorio y trabaja en una rama.
- Abre un pull request que haga referencia al número del issue.
Más de crmne/ruby_llm
-
bug
Dificultad 2/5 1-3 horas Aptitud para principiantes 78/100
Los mantenedores suelen responder en 1 día
-
enhancement
Dificultad 4/5 3-5 días Aptitud para principiantes 22/100
crmne/ruby_llm#1674 · 1 comentario ·
Los mantenedores suelen responder en 1 día
-
[BUG] FileStore::Directory follows links to folders inside its own folderPosiblemente ocupada @yorzi la tomó hace 1 día. Abiertobug
Dificultad 3/5 1-2 días Aptitud para principiantes 30/100
crmne/ruby_llm#1673 · 1 comentario ·
Los mantenedores suelen responder en 1 día
-
[BUG] FileStore::Directory#list returns nothing when a file vanishes mid-listingPosiblemente ocupada @yorzi la tomó hace 1 día. Abiertobug
Dificultad 2/5 1-3 horas Aptitud para principiantes 32/100
Los mantenedores suelen responder en 1 día
-
Body logging is tied to the logger's debug level, and streaming debug lines ignore the context loggerPosiblemente ocupada @josh-sanders la tomó hace 1 día. Abiertobug
Dificultad 3/5 Medio día Aptitud para principiantes 55/100
crmne/ruby_llm#1669 · 2 comentarios ·
Los mantenedores suelen responder en 1 día
Todos los issues de crmne/ruby_llm
Issues similares
-
Bug Ruby
Dificultad 2/5 1-3 horas Aptitud para principiantes 76/100
getsentry/sentry-ruby#3211 · 1 comentario ·
Los mantenedores suelen responder en 2 días
-
bug
Dificultad 2/5 1-3 horas Aptitud para principiantes 72/100
Los mantenedores suelen responder en 1 día
-
enhancement
Dificultad 2/5 1-3 horas Aptitud para principiantes 66/100
internetee/registry#2984 ·
Los mantenedores suelen responder en 6 días
-
interest-zsv
Dificultad 2/5 1-3 horas Aptitud para principiantes 62/100
hitobito/hitobito_zsv#6 ·
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 82/100
ecosyste-ms/issues#862 ·
Los mantenedores suelen responder en 1 día