Report credits an inaccurate problem restatement as a strength

Aberta
  • Algo errado?
#613 2 comentários 0 reações 0 responsáveis Ver no GitHub

Mantenedores costumam responder em até 1 dia

@jserv já está trabalhando nisso.

Desde 7/11/2026.

Avaliação

Dificuldade
4/5
Tempo estimado
3-5 dias
Facilidade para iniciantes
48/100
Tipo de issue
Bug
Clareza
Claramente especificada
Status de atividade
Ativa
Stack de tecnologia
rust
Domínio
ai, backend-api-design

Direção de pesquisa

Inspect the report generation path and the scoring and communication feedback prompts or rubrics referenced by the report: bundle 44, live prompt 33, report prompt 25, rubric 1, and report schema 4. Compare how the scripted restatement is evaluated with the scenario contract and its stated pitfall, then trace where Repeat and communication strengths are produced. Done means inaccurate or incomplete restatements lower Repeat and are reported as communication improvements.

Escrita pelo modelo de indexação a partir do texto da issue.

Descrição

Observed behavior

In a coding-only Editor interview on Single Resale Window (LeetCode 121), ended after Repeat and Example, the candidate's only restatement misstated the output (a price instead of the profit) and never stated that the sale must come after the purchase. The report still scored Repeat 90/100 in the framework assessment, and communicationFeedback.strengths[0] reads: "You clearly restated the problem requirements, including the input format and the goal of maximizing profit."

The same report's scenario contract requires i < j, and its first common pitfall is "Allowing a sell before the buy", so the report had the rule the restatement omitted.

Impact: a candidate who leaves out the key constraint is told their restatement was clear, so the report reinforces the gap instead of naming it.

Expected behavior

A restatement earns credit only for the inputs, output and rules it actually states. A misstated output, or an omitted contract rule that the brief lists as a pitfall, is a gap: it should lower Repeat and appear under communication improvements, not strengths.

Reproduction
  1. Start a coding-only Editor interview and select Single Resale Window.
  2. When Jim asks for a restatement, say: "The input is an integer array that represents the price every day. I need to find out which day is the best day to buy and which day is the best day to sell. To make the profit maximum. And I need to return the price. The maximum price."
  3. Give one example: decreasing prices 6, 5, 4, 3, 2, 1 return 0.
  4. End the interview and generate the report.
  5. Repeat scores 90/100 and communicationFeedback.strengths says the problem requirements were clearly restated.

Observed twice so far, on bundle 14 and bundle 44. The report is model-generated, so how often it happens is not yet measured.

CodeTrial version

codetrial 0.1.0 (cfcd576)

Environment

OS: Windows 11 with WSL2 (Ubuntu), server built from source
Browser: Google Chrome
Language: C++
Interview loop: Coding only, Editor
Problem: single-resale-window (LeetCode 121)
REACTO or STAR step: report (the claim concerns Repeat)
Reconnect: none

Sanitized evidence

Report header: bundle 44; live prompt 33; report prompt 25; rubric 1; report schema 4.

  • Phase scores: Repeat 90 / 100, Example 85 / 100.
  • communicationFeedback.strengths[0]: "You clearly restated the problem requirements, including the input format and the goal of maximizing profit."
  • Scenario contract in the same report: "bestSingleFlip(prices) returns the maximum of prices[j] - prices[i] over all day pairs with i < j, or 0 if no pair gives a positive difference (including a single day)."

The quoted restatement is the scripted line from a test session run to reproduce this, not a real candidate's interview.

Public data confirmation
  • I removed credentials, session tokens, candidate recordings, and personal interview content from this public report.
Linguagem predominante
Rust
Estrelas
223
Forks
148
Merge médio
1d 7h
PRs com merge (30d)
225

Preparar o ambiente

Primeiros passos

  1. Leia a issue inteira e depois o guia de contribuição do projeto.
  2. Comente na issue dizendo que vai assumir — evita que duas pessoas façam o mesmo trabalho.
  3. Faça um fork do repositório e trabalhe em uma branch.
  4. Abra um pull request que referencie o número da issue.

Mais de sysprog21/codetrial

Todas as issues de sysprog21/codetrial

Issues semelhantes

Mais issues de Rust

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.