Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

[P2] Verify actual model identity before reporting a same-model comparison

Closed
  • Is something wrong?
#4 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
3/5
Estimated time
1-2 days
Newbie friendliness
55/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
ollama, python
Domain
backend, testing

Research direction

Start with scripts/evaluate.py lines 14–43 and trace how it records model settings for the direct Ollama baseline and the running PatchPitch server response. Check the app's health or metadata and response meta.model, then find the offline or mock checks that can cover a configuration mismatch. Done means mismatched runs are rejected or marked non-comparable, with actual model and available digest or configuration evidence recorded.

Written by the indexing model from the issue text.

Description

Finding

Reviewed commit: 0549e5b1e6d88aaa46a0e580dcaad5b177446997.

scripts/evaluate.py describes a same-model experiment, but it records its own process's MODEL and OPTIONS for both variants. The baseline calls Ollama directly; the structured variant calls a separately running PatchPitch server.

Those two processes can have different model settings. For example:

  • PatchPitch is started with PATCHPITCH_MODEL=gemma3:12b.
  • The evaluation shell uses the default gemma3:4b.
  • Both top-level records say gemma3:4b, although the structured response's meta.model identifies gemma3:12b.

This configuration risk is established by the code. It is not evidence that the owner's retained experiment actually used mismatched models.

Questions for the existing results

  1. Did the baseline and structured runs use the same actual model, digest, and generation options?
  2. Do each raw response and the running app's health/metadata agree with the recorded top-level model?
  3. Were prompt/input changes and failed attempts retained when selecting the final example?

Acceptance if the comparison remains in the submission

  • Check the running app configuration before making the experiment's inference calls.
  • Record the actual model reported by each response and available model digest/configuration evidence.
  • Reject or clearly mark mismatched pairs as non-comparable; do not present them as a same-model comparison.
  • Cover configuration mismatch with an offline/mock check.
  • Keep human quality review and measured timing separate from structural validity.
  • Avoid claiming a controlled benchmark or measured improvement from unreviewed/tuned samples.

Priority: after the core P1 fixes. This is not required to demonstrate the product; a submission can omit the comparative claim and describe the observed live run honestly. No additional model service is needed.

Dominant language
Python
Stars
1
Forks
0
Avg merge
25m
Merged PRs (30d)
5

Getting set up

This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from drevendev/PatchPitch

All issues in drevendev/PatchPitch

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.