[BUG] Is there certain configuration on ImageNet-A? #2456
Description
Describe the bug
Thank you for the awesome library.
When I tried "validate.py" on the validation set of ImageNet with resnet50d.ra2_in1k, I obtained appropriate performance such as the following.
- Acc@1 81.362 (18.638) Acc@5 95.742 (4.258)
However, when I tried the same validation on the "ImageNet-A" dataset with resnet50d.ra2_in1k, I obtained top-1 accuracy of 0.08, which is significantly lower than your results reported in results-imagenet-a.csv.
Similarly, deit_tiny_patch16_224.fb_in1k exhibits top-1 accuracy of 0.160. I expected 7.213% as described in your results.
I just downloaded ImageNet-A dataset, and unzipped it as is, which has 200 folders that correspond to each class.
Is there a certain configuration required on ImageNet-A to obtain your results?
I attach my result below.
python validate.py --data-dir ~/imagenet-a/ --model resnet50d --pretrained
Validating in torch.float32. AMP not enabled.
Loading pretrained weights from Hugging Face hub (timm/resnet50d.ra2_in1k)
Model resnet50d created, param count: 25576264
Data processing configuration for current model + dataset:
input_size: (3, 288, 288)
interpolation: bicubic
mean: (0.485, 0.456, 0.406)
std: (0.229, 0.224, 0.225)
crop_pct: 0.95
crop_mode: center
Test: [ 0/30] Time: 3.142s (3.142s, 81.48/s) Loss: 7.2620 (7.2620) Acc@1: 0.000 ( 0.000) Acc@5: 1.172 ( 1.172)
Test: [ 10/30] Time: 0.390s (0.815s, 314.25/s) Loss: 7.4709 (7.2446) Acc@1: 0.000 ( 0.178) Acc@5: 0.000 ( 0.994)
Test: [ 20/30] Time: 0.391s (0.722s, 354.43/s) Loss: 7.7462 (7.3696) Acc@1: 0.000 ( 0.112) Acc@5: 0.000 ( 0.670)
- Acc@1 0.080 (99.920) Acc@5 0.493 (99.507)
--result
{
"model": "resnet50d",
"top1": 0.08,
"top1_err": 99.92,
"top5": 0.4933,
"top5_err": 99.5067,
"param_count": 25.58,
"img_size": 288,
"crop_pct": 0.95,
"interpolation": "bicubic"
}
To Reproduce
Steps to reproduce the behavior:
See the above.
Expected behavior
See the above.