Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

Evaluation: Update improve-prompt to v2

Open
#273 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
68/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Active
Tech stack
typescript
Domain
api, backend

Research direction

Start with app/api/evaluations/[id]/improve-prompt/route.ts and trace its callback_url resolution, then inspect /api/webhooks/prompt-improvement and the available PromptRecommendationJobPublic shape. Verify the judged-run guard and v2 request, preserve the callback body shape, handle recommendation_type: "prompt", and confirm v1 runs return 422 not_a_judge_run.

Written by the indexing model from the issue text.

Description

enhancement

Is your feature request related to a problem?
The improve-prompt currently uses v1, which relies on cosine scores. This limits the ability to provide users with a more comprehensive recommendation based on better metrics available in v2.

Describe the solution you'd like

  • Redirect app/api/evaluations/[id]/improve-prompt/route.ts to POST /api/v2/evaluations/{evaluation_id}/improve-prompt
  • Maintain existing callback_url resolution and /api/webhooks/prompt-improvement receiver with unchanged body shape
  • Implement handling for PromptRecommendationJobPublic callback: recommendation_type: "prompt"
  • Restrict action to judged runs; return 422 not_a_judge_run for v1 runs
Original issue

Is your feature request related to a problem?
Improve-prompt proxies to v1, which drafts from cosine scores. v2 drafts from the three-metric judge trace (score + reasoning per metric) and returns a typed recommendation, so the console gets a better prompt and a reason for it.

Describe the solution you'd like

  • Point app/api/evaluations/[id]/improve-prompt/route.ts at POST /api/v2/evaluations/{evaluation_id}/improve-prompt
  • Keep the existing callback_url resolution and /api/webhooks/prompt-improvement receiver — the body shape is unchanged
  • Handle the PromptRecommendationJobPublic callback: recommendation_type: "prompt" (widening to a union later)
  • Only offer the action on judged runs; surface 422 not_a_judge_run if it's attempted on a v1 run

Parent: #265

Dominant language
TypeScript
Stars
1
Forks
0
Avg merge
19h 4m
Merged PRs (30d)
6

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from ProjectTech4DevAI/kaapi-frontend

All issues in ProjectTech4DevAI/kaapi-frontend

Similar issues

More TypeScript issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.