Reviser can't resolve references named only in the query

Abierto
  • ¿Algo no está bien?
#77 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

@leothomas ya está trabajando en esto.

Desde el 11/11/2026.

  • #80 de @leothomas — abierto

Evaluación

Dificultad
4/5
Tiempo estimado
3-5 días
Aptitud para principiantes
35/100
Tipo de issue
Nueva funcionalidad
Claridad
Bien especificado
Estado de actividad
Estancado
Stack tecnológico
python
Área
machine-learning

Línea de trabajo

Read the Reviser.run and run_batch entry points, then trace how FactReasoner and the three baselines call them. Add the query to the prompt with a few-shot example and enforce the stated rule against introducing non-function words absent from the unit, response, and query. Check that references resolve while revisions that add unsupported claims leave the unit unchanged.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

Problem

The Reviser only sees the unit and the response. When the response never names its subject (common for chat answers, which lean on the question), the Reviser can't resolve the reference, and the atom stays unverifiable:

Q: Tell me about the James Webb Space Telescope.
A: It launched on December 25, 2021. It observes primarily in the infrared.
→ "It launched on December 25, 2021."   (unchanged on main)

The one-shot AtomizerReviser already takes the query for this reason, but it needs the LoRA adapter on vLLM.

Proposal

Pass the query to Reviser.run / run_batch as an optional argument, from FactReasoner and the three baselines, and add it to the prompt with a few-shot example.

One risk turned up in testing: with the question in front of it, a model may answer it rather than resolve a reference ("Who discovered penicillin?" + "He discovered it in 1928." → "Alexander Fleming discovered penicillin in 1928.", a claim the response never made). The prompt alone didn't prevent this with llama3.1-8B. So the proposal also rejects any revision that adds a non-function word found in neither the unit, response nor query, and keeps the unit as-is instead.

PR to follow.

Lenguaje dominante
Python
Estrellas
45
Forks
14
Merge medio
5 h 1 min
PR fusionados (30 d)
9

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de IBM/FactReasoner

Todos los issues de IBM/FactReasoner

Issues similares

Más issues de Python

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.