Evaluate minimal reasoning for OpenAI PDF remediation latency

Abierto
  • ¿Algo no está bien?
#101 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
4/5
Tiempo estimado
3-5 días
Aptitud para principiantes
48/100
Tipo de issue
Nueva funcionalidad
Claridad
Bastante claro
Estado de actividad
Tranquilo
Stack tecnológico
csharp
Área
ai, backend-api-design

Línea de trabajo

Start by locating the migration branch's Responses API calls for PDF title generation, table classification, character encoding repair, and ActualText repair. Build a repeatable timing and quality comparison using the listed reasoning levels, then document task-specific recommendations and add tests if minimal reasoning becomes the default.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

enhancement

Context

While migrating PDF remediation calls from Chat Completions to the Responses API, we ran a small real-API timing smoke test against gpt-5-mini to compare low vs minimal reasoning effort for lightweight remediation tasks.

The current migration branch uses low reasoning effort for Responses API calls to preserve the original intent of reducing latency while keeping reasoning enabled.

Smoke test results

Measured with gpt-5-mini, timing only the awaited API service call. Prompts were simple title generation and table classification samples.

Previous baseline
  • Chat Completions with medium reasoning was reportedly averaging about 7s per call.
Responses API with low reasoning

A 5-call smoke run across title and table classification showed:

  • Title service: avg 2.23s, median 1.84s, range 1.69s to 3.38s
  • Table classification: avg 2.40s, median 2.07s, range 2.01s to 3.75s
  • Combined: avg 2.31s, median 2.07s

A later 3-iteration comparison run showed:

  • Title low avg: 3.26s
  • Table classification low avg: 2.27s
Responses API with minimal reasoning

Same 3-iteration comparison run:

  • Title minimal avg: 0.92s
  • Table classification minimal avg: 1.70s

Sample outputs looked acceptable in the tiny smoke test:

  • Minimal title: UC Davis PDF Accessibility Remediation Guide
  • Minimal table classification: data_table, confidence 0.9
Responses API with none reasoning

none is not supported by gpt-5-mini. The API returned:

Unsupported value: 'none' is not supported with the 'gpt-5-mini' model. Supported values are: 'minimal', 'low', 'medium', and 'high'.

Investigation goals

  • Decide whether text-only remediation calls should use minimal reasoning instead of low reasoning.
  • Evaluate latency and quality for at least:
    • PDF title generation
    • table classification
    • character encoding repair
    • ActualText repair
  • Keep image alt text separate unless/until alt-text quality evals support changing it.
  • Confirm whether any deployed model choices support none, and whether none is appropriate for any remediation task.

Acceptance criteria

  • A repeatable timing/eval script or test harness exists for comparing reasoning effort levels.
  • Results include latency distributions and representative quality samples.
  • Recommendation is documented for each OpenAI-backed remediation task.
  • If minimal reasoning is adopted, defaults are changed with tests covering the selected effort level.
Lenguaje dominante
C#
Estrellas
1
Forks
0
Sin PR fusionados en 30 d

Preparar el entorno

Abrir en Codespaces

Inicia el contenedor de desarrollo del proyecto en tu navegador, con tu propia cuenta de GitHub.

  • Sin Dockerfile ni archivo de Docker Compose
  • Sin plantilla de pull request
  • Sin guía de contribución

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de ucdavis/readable

Todos los issues de ucdavis/readable

Issues similares

Más issues de C#

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.