huawei-noah/vega

CARS Accuracy Metric Question

Aperta

#248 aperta il 8 giu 2022

 (1 commento) (0 reazioni) (1 assegnatario)Python (177 fork)auto 404
good first issue

Metriche repository

Star
 (848 stelle)
Metriche merge PR
 (Metriche PR in attesa)

Descrizione

I am using a custom dataset with CARS and it runs fine, but I am confused because it shows a different accuracy at the end of the search part than during the search itself. During the 500 epochs of searching it for example finds this NN with 89% Val accuracy: 2022-06-08 03:07:34.605 INFO ############## KEEP ############ 2022-06-08 03:07:34.609 INFO KEEP Valid_acc for invidual 1 0.890625, size 0.111890, genotype Genotype(normal=[['max_pool_3x3', 2, 0], ['max_pool_3x3', 2, 1], ['max_pool_3x3', 3, 0], ['max_pool_3x3', 3, 1], ['max_pool_3x3', 4, 2], ['max_pool_3x3', 4, 0], ['avg_pool_3x3', 5, 0], ['avg_pool_3x3', 5, 1]], normal_concat=range(2, 6), reduce=[['sep_conv_5x5', 2, 0], ['max_pool_3x3', 2, 1], ['avg_pool_3x3', 3, 0], ['skip_connect', 3, 1], ['max_pool_3x3', 4, 1], ['max_pool_3x3', 4, 0], ['avg_pool_3x3', 5, 0], ['max_pool_3x3', 5, 1]], reduce_concat=range(2, 6))

but then at the end of the search it shows the best NN had 84% accuracy: 022-06-08 03:11:18.300 INFO worker id [0], epoch [500/500], train step [10/17], loss [ 0.004, 0.100], lr [ 0.0010004], time pre batch [0.582s] , total mean time per batch [0.569s] 2022-06-08 03:11:23.176 INFO Finished the unified trainer successfully. 2022-06-08 03:11:23.191 INFO Update Success. step_name=nas, worker_id=0 2022-06-08 03:11:23.194 INFO Best values: [{'worker_id': 2, 'performance': {'accuracy': 0.84375}}] 2022-06-08 03:11:23.218 INFO Clean worker folder /content/tasks/0608.012736.396/workers/nas.

Is this because it is picking the best NNs based on the Val score, and then at the end showing what the Test accuracy was for it?

Guida contributor