Published `best_model_avg.pth` does not reproduce the hold-short IoU in Table 3

Start by reproducing the reported validation results with best_model_avg.pth from the Hugging Face repository and compare them with Table 3, focusing on hold-short IoU. Read the training code around best_model_avg.pth and best_model_worstcase.pth to determine which checkpoint produced the table. Done means identifying and documenting the checkpoint or evaluation difference that explains the mismatch.

Hi,

My name is Henry and I'm a rising senior in high school. This summer I reproduced the segmentation stage of your taxiing localization paper and I ran into something I couldn't figure out on my own and wanted to ask about.

Using best_model_avg.pth from your Hugging Face repo on the shipped validation split, I get centerline 0.7275 and pavement 0.9824 which are both within half a percent of Table 3. My confusion was because hold-short comes out at 0.3505 instead of 0.6645.

I checked whether it was the decision threshold, the split, or how IoU gets averaged, and none of those explain it. The checkpoint is only finding about 40% of the hold-short pixels.

Was Table 3 made with this checkpoint, or a different one? I noticed the training code also saves best_model_worstcase.pth, which isn't on Hugging Face.

Happy to send the full numbers if that's useful. Thanks for releasing the code
and the dataset, I really enjoyed the reading and the opportunity to give it a try.

Best,

Henry