Inference on AVA and JHMDB Needs Maintenance and Necessary Files

[DanLuoNEU]

For the version I am using,

AVA2.1 inference needs several modifications:

1. tubelet-transformer/datasets/ava_frame.py

   Line 135 in f610c97

   video_frame_list = sorted(glob(video_frame_path + '/*.jpg'))

For function loadvideo, the function should be reading images with the video name.
video_frame_list = sorted(glob(video_frame_path + vid + '/*.jpg'))

2. Change the path here for the annotations.
   

   tubelet-transformer/evaluates/evaluate_ava.py

   Line 36 in f610c97

   f = open("/xxx/datasets/ava_val_excluded_timestamps_v2.1.csv")

3. The fixes above would get the number listed in the README table. But there would still be a tensorboard error "EOFerror". Add lines after

   tubelet-transformer/eval_tuber_ava.py

   Line 48 in f610c97

if cfg.DDP_CONFIG.GPU_WORLD_RANK == 0:
        writer.close()

AVA2.2 Inference

per_class [0.49119732        nan 0.32108856 0.58690862 0.1453127  0.25250868
 0.05269343 0.55119903 0.47336599 0.58118356 0.83511073 0.85809156
 0.4264426  0.79215918 0.7533182         nan 0.61339698        nan
        nan 0.04726829        nan 0.16529978        nan 0.23965087
        nan 0.04494236 0.306021   0.55275175 0.36725148 0.07057226
        nan        nan        nan 0.12159738        nan 0.03173127
 0.02196539 0.2641557         nan        nan 0.67544085        nan
 0.00367732        nan 0.01473403 0.03833153 0.03002702 0.37160171
 0.53368705        nan 0.21649021 0.1374056         nan 0.29578147
        nan 0.03978733 0.10253565 0.03219929 0.33915299 0.01752664
 0.28362901 0.3223239  0.14873739 0.52285939 0.14770317 0.11950478
 0.44886859 0.17733113 0.06789831 0.27917222        nan 0.46795067
 0.06238106 0.71983267        nan 0.05018591 0.31590126 0.09531384
 0.8376019  0.70844574]
{'PascalBoxes_Precision/mAP@0.5IOU': 0.30985340450933535, 'PascalBoxes_PerformanceByCategory/AP@0.5IOU/bend/bow (at the waist)': 0.4911973183134509, 'PascalBoxes_PerformanceByCategory/AP@0.5IOU/crouch/kneel': 0.3210885611841083, 'PascalBoxes_PerformanceByCategory/AP@0.5IOU/dance': 0.5869086163647963, 'PascalBoxes_PerformanceByCategory/AP@0.5IOU/fall down': 0.14531270272554303, 'PascalBoxes_PerformanceByCategory/AP@0.5IOU/get up': 0.25250867821227696, 'PascalBoxes_PerformanceByCategory/AP@0.5IOU/jump/leap': 0.05269343043207558, 'PascalBoxes_PerformanceByCategory/AP@0.5IOU/lie/sleep': 0.5511990313327797, 'PascalBoxes_PerformanceByCategory/AP@0.5IOU/martial art': 0.47336599427812304, 'PascalBoxes_PerformanceByCategory/AP@0.5IOU/run/jog': 0.5811835550049768, 'PascalBoxes_PerformanceByCategory/AP@0.5IOU/sit': 0.8351107282724392, 'PascalBoxes_PerformanceByCategory/AP@0.5IOU/stand': 0.8580915605931295, 'PascalBoxes_PerformanceByCategory/AP@0.5IOU/swim': 0.42644259946642094, 'PascalBoxes_PerformanceByCategory/AP@0.5IOU/walk': 0.7921591772441756, 'PascalBoxes_PerformanceByCategory/AP@0.5IOU/answer phone': 0.7533181965878357, 'PascalBoxes_PerformanceByCategory/AP@0.5IOU/carry/hold (an object)': 0.613396976906247, 'PascalBoxes_PerformanceByCategory/AP@0.5IOU/climb (e.g., a mountain)': 0.047268291513739374, 'PascalBoxes_PerformanceByCategory/AP@0.5IOU/close (e.g., a door, a box)': 0.16529978105316412, 'PascalBoxes_PerformanceByCategory/AP@0.5IOU/cut': 0.239650870599096, 'PascalBoxes_PerformanceByCategory/AP@0.5IOU/dress/put on clothing': 0.04494235744272522, 'PascalBoxes_PerformanceByCategory/AP@0.5IOU/drink': 0.30602100382076136, 'PascalBoxes_PerformanceByCategory/AP@0.5IOU/drive (e.g., a car, a truck)': 0.5527517520577403, 'PascalBoxes_PerformanceByCategory/AP@0.5IOU/eat': 0.3672514840844659, 'PascalBoxes_PerformanceByCategory/AP@0.5IOU/enter': 0.07057225556756908, 'PascalBoxes_PerformanceByCategory/AP@0.5IOU/hit (an object)': 0.12159737681929804, 'PascalBoxes_PerformanceByCategory/AP@0.5IOU/lift/pick up': 0.03173127096825363, 'PascalBoxes_PerformanceByCategory/AP@0.5IOU/listen (e.g., to music)': 0.021965385905557883, 'PascalBoxes_PerformanceByCategory/AP@0.5IOU/open (e.g., a window, a car door)': 0.2641556990694153, 'PascalBoxes_PerformanceByCategory/AP@0.5IOU/play musical instrument': 0.6754408509957595, 'PascalBoxes_PerformanceByCategory/AP@0.5IOU/point to (an object)': 0.0036773150722066972, 'PascalBoxes_PerformanceByCategory/AP@0.5IOU/pull (an object)': 0.01473402768023624, 'PascalBoxes_PerformanceByCategory/AP@0.5IOU/push (an object)': 0.038331529680086275, 'PascalBoxes_PerformanceByCategory/AP@0.5IOU/put down': 0.03002701544153771, 'PascalBoxes_PerformanceByCategory/AP@0.5IOU/read': 0.3716017145811048, 'PascalBoxes_PerformanceByCategory/AP@0.5IOU/ride (e.g., a bike, a car, a horse)': 0.5336870531261757, 'PascalBoxes_PerformanceByCategory/AP@0.5IOU/sail boat': 0.21649020512834088, 'PascalBoxes_PerformanceByCategory/AP@0.5IOU/shoot': 0.13740559748226708, 'PascalBoxes_PerformanceByCategory/AP@0.5IOU/smoke': 0.2957814682780021, 'PascalBoxes_PerformanceByCategory/AP@0.5IOU/take a photo': 0.03978732762876234, 'PascalBoxes_PerformanceByCategory/AP@0.5IOU/text on/look at a cellphone': 0.10253564997258985, 'PascalBoxes_PerformanceByCategory/AP@0.5IOU/throw': 0.03219929211064902, 'PascalBoxes_PerformanceByCategory/AP@0.5IOU/touch (an object)': 0.33915299353156436, 'PascalBoxes_PerformanceByCategory/AP@0.5IOU/turn (e.g., a screwdriver)': 0.017526643108955034, 'PascalBoxes_PerformanceByCategory/AP@0.5IOU/watch (e.g., TV)': 0.28362901476702795, 'PascalBoxes_PerformanceByCategory/AP@0.5IOU/work on a computer': 0.322323903124391, 'PascalBoxes_PerformanceByCategory/AP@0.5IOU/write': 0.1487373880589133, 'PascalBoxes_PerformanceByCategory/AP@0.5IOU/fight/hit (a person)': 0.5228593870747025, 'PascalBoxes_PerformanceByCategory/AP@0.5IOU/give/serve (an object) to (a person)': 0.14770317484649234, 'PascalBoxes_PerformanceByCategory/AP@0.5IOU/grab (a person)': 0.11950477963584528, 'PascalBoxes_PerformanceByCategory/AP@0.5IOU/hand clap': 0.44886858836133026, 'PascalBoxes_PerformanceByCategory/AP@0.5IOU/hand shake': 0.17733112595251085, 'PascalBoxes_PerformanceByCategory/AP@0.5IOU/hand wave': 0.06789830556787521, 'PascalBoxes_PerformanceByCategory/AP@0.5IOU/hug (a person)': 0.27917221591712854, 'PascalBoxes_PerformanceByCategory/AP@0.5IOU/kiss (a person)': 0.4679506698404774, 'PascalBoxes_PerformanceByCategory/AP@0.5IOU/lift (a person)': 0.062381058259554645, 'PascalBoxes_PerformanceByCategory/AP@0.5IOU/listen to (a person)': 0.7198326661128859, 'PascalBoxes_PerformanceByCategory/AP@0.5IOU/push (another person)': 0.050185914377705816, 'PascalBoxes_PerformanceByCategory/AP@0.5IOU/sing to (e.g., self, a person, a group)': 0.31590125934914154, 'PascalBoxes_PerformanceByCategory/AP@0.5IOU/take (an object) from (a person)': 0.09531383956904724, 'PascalBoxes_PerformanceByCategory/AP@0.5IOU/talk to (e.g., self, a person, a group)': 0.8376018955287321, 'PascalBoxes_PerformanceByCategory/AP@0.5IOU/watch (a person)': 0.7084457445779531}
mAP: 0.30985

[DanLuoNEU]

For JHMDB Inference

• here should be
  from models.tuber_jhmdb import build_model

• modify
  

  tubelet-transformer/models/tuber_jhmdb.py

  Line 20 in f610c97

  from models.transformer.transformer_refined import build_transformer

  
  as
  from models.transformer.transformer import build_transformer

• Needs 'JHMDB-GT.pkl'
  Found the script to download according to the direction in dataset part.
  Update:

  1. nope, that link only has Annotations, Frames and OF, but not with the file above.
  2. Get the pickle file from MOC

• Provided pretrained DETR model has different embedded query input dimensions as the model built and pretrained JHMDB model

modify the loading detr part according to the built model embed_query input dimensions to avoid this problem

tubelet-transformer/utils/model_utils.py

Line 25 in f610c97

pretrained_dict.update({k: v[:query_size]})

pretrained_dict.update({k: v[:query_size]})
if query_size == model.module.query_embed.weight.shape[0]: continue 
if v.shape[0] < model.module.query_embed.weight.shape[0]: # In case the pretrained model does not align
  query_embed_zeros=torch.zeros(model.module.query_embed.weight.shape)
  pretrained_dict.update({k: query_embed_zeros})
else:
  pretrained_dict.update({k: v[:model.module.query_embed.weight.shape[0]]})

Got different mAP as the table shows

per_class [0.96529908 0.4870422  0.81740977 0.64671594 0.99981187 0.48678173
 0.72522214 0.70157535 0.99132313 0.99332738 0.92539198 0.63780982
 0.6607778  0.89695387 0.78694818 0.42965094 0.26324953 0.94429166
 0.27346689 0.68134081 0.87238637        nan        nan        nan]
{'PascalBoxes_Precision/mAP@0.5IOU': 0.7231798302410739, 'PascalBoxes_PerformanceByCategory/AP@0.5IOU/Basketball': 0.9652990848728149, 'PascalBoxes_PerformanceByCategory/AP@0.5IOU/BasketballDunk': 0.4870421987013735, 'PascalBoxes_PerformanceByCategory/AP@0.5IOU/Biking': 0.8174097664543525, 'PascalBoxes_PerformanceByCategory/AP@0.5IOU/CliffDiving': 0.6467159401389935, 'PascalBoxes_PerformanceByCategory/AP@0.5IOU/CricketBowling': 0.9998118686054533, 'PascalBoxes_PerformanceByCategory/AP@0.5IOU/Diving': 0.48678173366600064, 'PascalBoxes_PerformanceByCategory/AP@0.5IOU/Fencing': 0.7252221388068574, 'PascalBoxes_PerformanceByCategory/AP@0.5IOU/FloorGymnastics': 0.7015753486207187, 'PascalBoxes_PerformanceByCategory/AP@0.5IOU/GolfSwing': 0.9913231289322941, 'PascalBoxes_PerformanceByCategory/AP@0.5IOU/HorseRiding': 0.9933273801597415, 'PascalBoxes_PerformanceByCategory/AP@0.5IOU/IceDancing': 0.9253919821730238, 'PascalBoxes_PerformanceByCategory/AP@0.5IOU/LongJump': 0.637809816668955, 'PascalBoxes_PerformanceByCategory/AP@0.5IOU/PoleVault': 0.6607777957457814, 'PascalBoxes_PerformanceByCategory/AP@0.5IOU/RopeClimbing': 0.8969538737505489, 'PascalBoxes_PerformanceByCategory/AP@0.5IOU/SalsaSpin': 0.7869481765834933, 'PascalBoxes_PerformanceByCategory/AP@0.5IOU/SkateBoarding': 0.42965094009542815, 'PascalBoxes_PerformanceByCategory/AP@0.5IOU/Skiing': 0.26324952994810963, 'PascalBoxes_PerformanceByCategory/AP@0.5IOU/Skijet': 0.9442916605769802, 'PascalBoxes_PerformanceByCategory/AP@0.5IOU/SoccerJuggling': 0.27346688938240526, 'PascalBoxes_PerformanceByCategory/AP@0.5IOU/Surfing': 0.681340807090747, 'PascalBoxes_PerformanceByCategory/AP@0.5IOU/TennisSwing': 0.8723863740884812, 'PascalBoxes_PerformanceByCategory/AP@0.5IOU/TrampolineJumping': nan, 'PascalBoxes_PerformanceByCategory/AP@0.5IOU/VolleyballSpiking': nan, 'PascalBoxes_PerformanceByCategory/AP@0.5IOU/WalkingWithDog': nan}
mAP: 0.72318

[CKK-coder]

For JHMDB Inference

• here should be
  from models.tuber_jhmdb import build_model

• modify
  

  tubelet-transformer/models/tuber_jhmdb.py

  Line 20 in f610c97

  from models.transformer.transformer_refined import build_transformer

  as
  from models.transformer.transformer import build_transformer

• Needs 'JHMDB-GT.pkl'
  Found the script to download according to the direction in dataset part.
  Update:

  1. nope, that link only has Annotations, Frames and OF, but not with the file above.
  2. Get the pickle file from MOC

• Provided pretrained DETR model has different embedded query input dimensions as the model built and pretrained JHMDB model

modify the loading detr part according to the built model embed_query input dimensions to avoid this problem

tubelet-transformer/utils/model_utils.py

Line 25 in f610c97

pretrained_dict.update({k: v[:query_size]})

pretrained_dict.update({k: v[:query_size]})
if query_size == model.module.query_embed.weight.shape[0]: continue 
if v.shape[0] < model.module.query_embed.weight.shape[0]: # In case the pretrained model does not align
  query_embed_zeros=torch.zeros(model.module.query_embed.weight.shape)
  pretrained_dict.update({k: query_embed_zeros})
else:
  pretrained_dict.update({k: v[:model.module.query_embed.weight.shape[0]]})

Got different mAP as the table shows

per_class [0.96529908 0.4870422  0.81740977 0.64671594 0.99981187 0.48678173
 0.72522214 0.70157535 0.99132313 0.99332738 0.92539198 0.63780982
 0.6607778  0.89695387 0.78694818 0.42965094 0.26324953 0.94429166
 0.27346689 0.68134081 0.87238637        nan        nan        nan]
{'PascalBoxes_Precision/mAP@0.5IOU': 0.7231798302410739, 'PascalBoxes_PerformanceByCategory/AP@0.5IOU/Basketball': 0.9652990848728149, 'PascalBoxes_PerformanceByCategory/AP@0.5IOU/BasketballDunk': 0.4870421987013735, 'PascalBoxes_PerformanceByCategory/AP@0.5IOU/Biking': 0.8174097664543525, 'PascalBoxes_PerformanceByCategory/AP@0.5IOU/CliffDiving': 0.6467159401389935, 'PascalBoxes_PerformanceByCategory/AP@0.5IOU/CricketBowling': 0.9998118686054533, 'PascalBoxes_PerformanceByCategory/AP@0.5IOU/Diving': 0.48678173366600064, 'PascalBoxes_PerformanceByCategory/AP@0.5IOU/Fencing': 0.7252221388068574, 'PascalBoxes_PerformanceByCategory/AP@0.5IOU/FloorGymnastics': 0.7015753486207187, 'PascalBoxes_PerformanceByCategory/AP@0.5IOU/GolfSwing': 0.9913231289322941, 'PascalBoxes_PerformanceByCategory/AP@0.5IOU/HorseRiding': 0.9933273801597415, 'PascalBoxes_PerformanceByCategory/AP@0.5IOU/IceDancing': 0.9253919821730238, 'PascalBoxes_PerformanceByCategory/AP@0.5IOU/LongJump': 0.637809816668955, 'PascalBoxes_PerformanceByCategory/AP@0.5IOU/PoleVault': 0.6607777957457814, 'PascalBoxes_PerformanceByCategory/AP@0.5IOU/RopeClimbing': 0.8969538737505489, 'PascalBoxes_PerformanceByCategory/AP@0.5IOU/SalsaSpin': 0.7869481765834933, 'PascalBoxes_PerformanceByCategory/AP@0.5IOU/SkateBoarding': 0.42965094009542815, 'PascalBoxes_PerformanceByCategory/AP@0.5IOU/Skiing': 0.26324952994810963, 'PascalBoxes_PerformanceByCategory/AP@0.5IOU/Skijet': 0.9442916605769802, 'PascalBoxes_PerformanceByCategory/AP@0.5IOU/SoccerJuggling': 0.27346688938240526, 'PascalBoxes_PerformanceByCategory/AP@0.5IOU/Surfing': 0.681340807090747, 'PascalBoxes_PerformanceByCategory/AP@0.5IOU/TennisSwing': 0.8723863740884812, 'PascalBoxes_PerformanceByCategory/AP@0.5IOU/TrampolineJumping': nan, 'PascalBoxes_PerformanceByCategory/AP@0.5IOU/VolleyballSpiking': nan, 'PascalBoxes_PerformanceByCategory/AP@0.5IOU/WalkingWithDog': nan}
mAP: 0.72318

Thank you for your correction.Do you find any code about video map inference. I want to reproduce the video map of UCF101-24.

[FransHk]

Thanks for taking your time to write this, helped me greatly. It's a shame that the codebase for this model is such a mess as-is.

[hemengfan2002]

Thank you very much for your advice. However, I still encountered some problems, when I used the trained model （TubeR_CSN152_JHMDB.pth）to inference, the map was 0. And when I used the trained model（TubeR_CSN152_JHMDB.pth）and retrained it, it worked well. But training from scratch is also poor, training from scratch will cause problems like the code below.

Train
Epoch: [1][460/2839] lr: 5e-05 data_time: 0.025, batch time: 0.657 class_error: 11.761, loss: 7.271, loss_bbox: 0.085, loss_giou: 0.161, loss_ce: 0.356, loss_ce_b: 0.000
Epoch: [1][461/2839] lr: 5e-05 data_time: 0.031, batch time: 0.661 class_error: 11.736, loss: 7.273, loss_bbox: 0.085, loss_giou: 0.161, loss_ce: 0.356, loss_ce_b: 0.000

eval
Epoch: [0][9138/9139] data_time: 0.003, batch time: 0.069 class_error: 99.354, loss: 15.582, loss_bbox: 0.143, loss_giou: 0.233, loss_ce: 1.298
Epoch: [0][9139/9139] data_time: 0.003, batch time: 0.068 class_error: 99.354, loss: 15.582, loss_bbox: 0.143, loss_giou: 0.233, loss_ce: 1.298

per_class_len 24 per_class [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.02090471 0. nan nan nan

Like the situation encountered with issue 19, I want to know why this situation occurred and want to resolve it. hope you can reply.