pm_eval_model crashes with TypeError after the evaluation when --out_fname is not given (regression from #617)

Open Beginner friendly
  • Is something wrong?
#675 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

@Siddhanthguptaa is already working on this.

Since Nov 7, 2026.

  • #676 by @Siddhanthguptaa — open

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
72/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
python, pytorch
Domain
machine-learning, testing

Research direction

Start in perceptionmetrics/cli/eval_model.py at the command’s output handling after model.eval() returns. Check the existing tests for segmentation and detection output handling, then add coverage for invoking the command without --out_fname. Done means both return types complete successfully without that option while the existing output behavior remains covered.

Written by the indexing model from the issue text.

Description

Problem

pm_eval_model declares --out_fname as optional, but since #617 the end of the command does os.makedirs(os.path.dirname(os.path.abspath(out_fname)), exist_ok=True) without checking for None. A run without --out_fname (for example one that only wants per-sample outputs through --predictions_outdir) now spends the whole evaluation, writes the per-sample predictions, and then ends with:

TypeError: expected str, bytes or os.PathLike object, not NoneType

In v3.1.0 the same call finished normally: the last line was results.to_csv(out_fname), which writes nothing when the path is None. The regression comes from b10bf7d (#617, the fix for #563) and is not in any release yet.

Reproduction

from click.testing import CliRunner
from perceptionmetrics import cli
from perceptionmetrics.cli.eval_model import eval_model

# any model/dataset: the error happens after model.eval() returns
# (model, ontology and cfg files must exist; fake get_model/get_dataset keep it short)
result = CliRunner().invoke(eval_model, ["segmentation", "image",
                            "--model", "model.pt",
                            "--model_ontology", "ontology.json",
                            "--model_cfg", "cfg.json"])
print(repr(result.exception))

On master (4a1ae41) this prints TypeError('expected str, bytes or os.PathLike object, not NoneType') for both the segmentation (DataFrame) and detection (dict) return types.

Proposed fix

Only create the directory and write the CSV when --out_fname is given. This keeps what #617 added (directory creation and the detection dict handling) and restores the v3.1.0 behavior for runs without --out_fname. I am not proposing to make the option required: it was made optional on purpose when --predictions_outdir was introduced, and eval_preds / computational_cost are separate commands.

I would like to work on this. It is a two-line change plus tests, and I have it ready locally, so I can open a small PR if that works for you.

Dominant language
Python
Stars
114
Forks
116
Avg merge
1d 7h
Merged PRs (30d)
11

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from JdeRobot/PerceptionMetrics

All issues in JdeRobot/PerceptionMetrics

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.