[BUG] `Tool#ask` does not check that a stored answer belongs to the question it asks now

Abierto
#1,682 1 comentario 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
4/5
Tiempo estimado
3-5 días
Aptitud para principiantes
55/100
Tipo de issue
Error
Claridad
Bien especificado
Estado de actividad
Activo
Stack tecnológico
ruby

Línea de trabajo

Start with lib/ruby_llm/tool.rb and inspect how Tool#ask retrieves stored requests by position. Use the two reproductions in the issue to check skipped questions and changed choices; the report does not name a test file. Done means a stored answer is not returned when the current question differs, and the tool asks again as proposed.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

bug
Basic checks
  • I searched existing issues - this hasn't been reported
  • I can reproduce this consistently
  • This is a RubyLLM bug, not my application code
What's broken?

When a question is answered, execute runs again from the top, and ask finds the stored answer by position only: the first ask of the run gets the answer at index 0, the second the answer at index 1 (lib/ruby_llm/tool.rb):

requests = Array(state['input']&.[]('requests'))
return Question.from_h(requests[index]).asked_answer(fields) if requests[index]&.[]('response')

The stored question is not compared with the question asked now. So ask has no protection against a stale answer: if a later run asks a different question at the same position, the tool gets the answer the person gave to the other question. Nothing is raised, and nothing reports the mismatch.

I know the limits of this report. The tools guide says "A call can ask several questions, one after another, and each run asks them in the same order", and the first case below breaks that rule. The second case keeps the order: only the choices change. In both, the rule is easy to break without noticing (one if around an ask, or choices built from data that changes), and the result is silent: a person's "no" reaches the tool as "yes". The stored question lies next to the answer, so RubyLLM can see the mismatch.

How to reproduce

No network: the script replaces the provider's complete with two prepared replies. Save it as issue_repro.rb in a checkout of main and run ruby -I lib issue_repro.rb.

require 'ruby_llm'

RubyLLM.configure { |config| config.anthropic_api_key = 'test' }
MODEL = 'claude-haiku-4-5'

DRAFT = { present: true }
SEATS = %w[window aisle].dup

# Case 1: a question is skipped on a later run.
class CleanDeck < RubyLLM::Tool
  description 'Cleans up a deck'

  def execute
    keep = ask('Keep the draft?', choices: %w[yes no]) if DRAFT[:present]
    wipe = ask('Delete every card?', choices: %w[yes no])
    "keep the draft: #{keep.inspect}, delete every card: #{wipe.inspect}"
  end
end

# Case 2: one question, in the same place; its choices come from data.
class PickSeat < RubyLLM::Tool
  description 'Picks a seat'

  def execute
    seat = ask('Which seat?', choices: SEATS)
    "seat: #{seat.inspect}, offered now: #{SEATS.inspect}"
  end
end

# A chat whose model calls +tool+ once, then says "Done".
def chat_calling(tool)
  chat = RubyLLM::Chat.new(model: MODEL)
  call = RubyLLM::ToolCall.new(id: 'call_1', name: tool.new.name, arguments: {})
  replies = [
    RubyLLM::Message.new(role: :assistant, content: '', tool_calls: { 'call_1' => call }),
    RubyLLM::Message.new(role: :assistant, content: 'Done', model: MODEL, input_tokens: 1, output_tokens: 1)
  ]
  chat.provider.define_singleton_method(:complete) { |*, **, &_block| replies.shift }
  chat.with_tools(tool).ask('Go')
  chat
end

def answer(chat, value)
  question = chat.pending_questions.first
  puts "asked: #{question.message} #{question.fields.first.choices.inspect} -> the person answers #{value}"
  question.answer(value)
end

def result(chat) = chat.messages.find { |message| message.role == :tool }.content

chat = chat_calling(CleanDeck)
answer(chat, 'yes')
chat.complete
answer(chat, 'no')
DRAFT[:present] = false # the draft is published elsewhere while the person decides
chat.complete
puts "tool result: #{result(chat)}\n\n"

chat = chat_calling(PickSeat)
answer(chat, 'aisle')
SEATS.replace(%w[front back]) # the seat map changes while the person decides
chat.complete
puts "tool result: #{result(chat)}"
Expected behavior

A tool does not get an answer that the person gave to another question.

  • Case 1: the tool does not get "yes" for "Delete every card?". The person answered "no" to that question.
  • Case 2: the tool does not get "aisle" when it offers "front" and "back".
What actually happened
asked: Keep the draft? ["yes", "no"] -> the person answers yes
asked: Delete every card? ["yes", "no"] -> the person answers no
tool result: keep the draft: nil, delete every card: "yes"

asked: Which seat? ["window", "aisle"] -> the person answers aisle
tool result: seat: "aisle", offered now: ["front", "back"]

Case 1: on the third run "Keep the draft?" is skipped, "Delete every card?" becomes the first ask, and it gets the answer stored at index 0.

Case 2: the answer was checked against the choices of the first run, and is returned for the choices of the second.

Two more cases give the same kind of result. A run that asks another question at the same position ("Keep the draft?", then "Delete every card in the deck?") gets "yes". A run that asks for fields where the first run asked for a choice gets {answer: "aisle"}, with none of the declared fields.

Answers do not cross between tool calls: two calls in one round, answered in reverse order, each get their own answer.

Proposed solution

Compare the question before ask returns its answer. ask has every argument it needs to build the question it asks now, and the stored question keeps the same data (params: the message, the schema of the answer, the URL). Return the stored answer, or the stored decline, only if the two are equal. If they differ, replace the stored questions from that position on with the question asked now, under a new id, and pause on it.

In case 1, "Delete every card?" is asked once more on the third run. In case 2, the person picks between "front" and "back".

Ask again or raise: this is a choice. I propose to ask again, because a changed question is one the person did not answer yet (a price went up while they were away). An error is the alternative: it shows unstable tool code faster and can never ask twice.

What this does not do, and what it costs:

  • It is a guard, not an identity. Two questions with equal text and equal choices about different things still match.
  • A tool whose wording differs on every run while the meaning stays the same (the current time in the text, a sentence written by a model, choices in another order) is asked again on every run. Today it finishes.
  • The block of fields runs on every run, to build the schema to compare. Today it does not run for an answered question.
  • The signature of ask does not change. The params of all three kinds of question are equal to themselves after a JSON round trip, so I expect the comparison to hold for a persisted chat; I did not run one.
  • The change stays inside Tool#ask. Questions from an MCP server and a Task returned after a question keep their behavior. A tool that calls another tool that asks is a different problem and is out of scope.

If you agree with the comparison, I can send a pull request that adds it.

Environment
  • Ruby version: 4.0.9
  • RubyLLM version: 2.2.0, main at 5d041358
  • Provider: Anthropic, stubbed (the script replaces complete)
  • An in-memory chat. I did not run an acts_as_chat record; it goes through the same Tool#ask.
  • OS: macOS (Darwin 25.6.0)
Lenguaje dominante
Ruby
Estrellas
4.5k
Forks
537
Merge medio
7 h 30 min
PR fusionados (30 d)
378

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de crmne/ruby_llm

Todos los issues de crmne/ruby_llm

Issues similares

Más issues de Ruby

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.