Report credits an inaccurate problem restatement as a strength

Offen
  • Stimmt etwas nicht?
#613 2 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen

Maintainer antworten meist innerhalb von 1 Tag

@jserv arbeitet bereits daran.

Seit 07.11.2026.

Bewertung

Schwierigkeit
4/5
Geschätzter Aufwand
3-5 Tage
Anfängerfreundlichkeit
48/100
Issue-Typ
Bug
Klarheit
Klar beschrieben
Aktivitätsstatus
Aktiv
Tech-Stack
rust
Bereich
ai, backend-api-design

Rechercherichtung

Inspect the report generation path and the scoring and communication feedback prompts or rubrics referenced by the report: bundle 44, live prompt 33, report prompt 25, rubric 1, and report schema 4. Compare how the scripted restatement is evaluated with the scenario contract and its stated pitfall, then trace where Repeat and communication strengths are produced. Done means inaccurate or incomplete restatements lower Repeat and are reported as communication improvements.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Beschreibung

Observed behavior

In a coding-only Editor interview on Single Resale Window (LeetCode 121), ended after Repeat and Example, the candidate's only restatement misstated the output (a price instead of the profit) and never stated that the sale must come after the purchase. The report still scored Repeat 90/100 in the framework assessment, and communicationFeedback.strengths[0] reads: "You clearly restated the problem requirements, including the input format and the goal of maximizing profit."

The same report's scenario contract requires i < j, and its first common pitfall is "Allowing a sell before the buy", so the report had the rule the restatement omitted.

Impact: a candidate who leaves out the key constraint is told their restatement was clear, so the report reinforces the gap instead of naming it.

Expected behavior

A restatement earns credit only for the inputs, output and rules it actually states. A misstated output, or an omitted contract rule that the brief lists as a pitfall, is a gap: it should lower Repeat and appear under communication improvements, not strengths.

Reproduction
  1. Start a coding-only Editor interview and select Single Resale Window.
  2. When Jim asks for a restatement, say: "The input is an integer array that represents the price every day. I need to find out which day is the best day to buy and which day is the best day to sell. To make the profit maximum. And I need to return the price. The maximum price."
  3. Give one example: decreasing prices 6, 5, 4, 3, 2, 1 return 0.
  4. End the interview and generate the report.
  5. Repeat scores 90/100 and communicationFeedback.strengths says the problem requirements were clearly restated.

Observed twice so far, on bundle 14 and bundle 44. The report is model-generated, so how often it happens is not yet measured.

CodeTrial version

codetrial 0.1.0 (cfcd576)

Environment

OS: Windows 11 with WSL2 (Ubuntu), server built from source
Browser: Google Chrome
Language: C++
Interview loop: Coding only, Editor
Problem: single-resale-window (LeetCode 121)
REACTO or STAR step: report (the claim concerns Repeat)
Reconnect: none

Sanitized evidence

Report header: bundle 44; live prompt 33; report prompt 25; rubric 1; report schema 4.

  • Phase scores: Repeat 90 / 100, Example 85 / 100.
  • communicationFeedback.strengths[0]: "You clearly restated the problem requirements, including the input format and the goal of maximizing profit."
  • Scenario contract in the same report: "bestSingleFlip(prices) returns the maximum of prices[j] - prices[i] over all day pairs with i < j, or 0 if no pair gives a positive difference (including a single day)."

The quoted restatement is the scripted line from a test session run to reproduce this, not a real candidate's interview.

Public data confirmation
  • I removed credentials, session tokens, candidate recordings, and personal interview content from this public report.
Vorherrschende Sprache
Rust
Sterne
223
Forks
148
Ø Merge
1 T. 7 Std.
Gemergte PRs (30 T.)
225

Entwicklungsumgebung

Erste Schritte

  1. Lesen Sie das ganze Issue und danach den Beitragsleitfaden des Projekts.
  2. Schreiben Sie ins Issue, dass Sie es übernehmen — das erspart doppelte Arbeit.
  3. Forken Sie das Repository und arbeiten Sie in einem Branch.
  4. Öffnen Sie einen Pull Request, der die Issue-Nummer nennt.

Mehr aus sysprog21/codetrial

Alle Issues in sysprog21/codetrial

Ähnliche Issues

Weitere Issues zu Rust

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.