Evaluation results are inconsistent during training and after saving the trained model

[YUXIN-commit]

Hi MONAI Team,
First of all, thumbs up for your great work. I am facing the issue of inconsistent results of the SwinUNETR model while evaluating during the training and testing phases.
During training, I am getting good results but when I saved the pre-trained SwinUNETR model I am getting very bad results. I didn't change anything, just download your GitHub repo and trained the SwinUNETR model but the results are inconsistent. Here I am uploading the screenshots when I have trained the model for 100 epochs. Please help me to figure out the issue. Waiting for your response. Thank you in advance.

training results:

when I saved the pre-trained SwinUNETR model I am getting very bad results:

[YUXIN-commit]

I encountered another issue, which is 'Too many false positives on test-data' when using test.py. Has anyone encountered this problem before? Could you please let me know how to resolve it? "
Let me know if you'd like me to refine it further!

As shown in the image:

[ras-lyg]

Hello, I've encountered the same issue. When I downloaded the author’s pre-trained weights to test on the BTCV dataset, the results were very good. However, when I used my own brain supplement dataset for training, the testing performance was very poor despite achieving high accuracy during the training process. Have you managed to resolve this issue? Looking forward to your reply.

[YUXIN-commit]

Hello, I've encountered the same issue. When I downloaded the author’s pre-trained weights to test on the BTCV dataset, the results were very good. However, when I used my own brain supplement dataset for training, the testing performance was very poor despite achieving high accuracy during the training process. Have you managed to resolve this issue? Looking forward to your reply.

Hello, I have solved the issue of overly fragmented test datasets, which was mainly due to the strict requirements for sliding window inference size. You can adjust the roi_size and overlap to resolve this problem. Additionally, I found that setting include_background to False in DiceMetric during training makes the performance closer to the test performance.

[ras-lyg]

Thank you for your response. However, it seems that our issues are somewhat different. I am using SwinUNETR to train on other datasets to obtain weights, and then using the author's test.py for testing. Despite achieving an accuracy of 0.865 during training, the testing performance is not good, with many classes having a Dice score of only 0.

[YUXIN-commit]

Thank you for your response. However, it seems that our issues are somewhat different. I am using SwinUNETR to train on other datasets to obtain weights, and then using the author's test.py for testing. Despite achieving an accuracy of 0.865 during training, the testing performance is not good, with many classes having a Dice score of only 0.

I feel the same way, but I think the high Dice score during training is due to the include_background setting being set to True, which includes the background class (class 0) in the calculations, leading to a higher Dice score

[Yikol]

Hello, I've encountered the same issue. When I downloaded the author’s pre-trained weights to test on the BTCV dataset, the results were very good. However, when I used my own brain supplement dataset for training, the testing performance was very poor despite achieving high accuracy during the training process. Have you managed to resolve this issue? Looking forward to your reply.

Hello, after training with my own dataset, I tested it on the test dataset and encountered an error saying 'Error(s) in loading state_dict for UNETR'. It states that the network architecture during training is different from the one used during testing, particularly the last layer. During training, I had 10 output classes, but the error mentions that there are 14 output classes.Have you encountered this issue before?

[hhuxuchang]

@ras-lyg hello, have you solved this problems, any solutions?

[ras-lyg]

是的，我已经解决了，问题出在测试程序上  裴训龙 ***@***.***  

…

------------------ 原始邮件 ------------------ 发件人: ***@***.***>; 发送时间: 2025年6月16日(星期一) 晚上7:24 收件人: ***@***.***>; 抄送: ***@***.***>; ***@***.***>; 主题: Re: [Project-MONAI/research-contributions] Evaluation results are inconsistent during training and after saving the trained model (Issue #405) 胡昌留下评论（项目-MONAI/研究贡献#405） @ras-lyg你好，你解决这个问题了吗，有什么解决办法吗？ - 直接回复这封邮件，在GitHub上查看它，或者取消订阅. ***@***.*** >

[hhuxuchang]

是的，我已经解决了，问题出在测试程序上 

裴训龙
@.***

 
…

------------------ 原始邮件 ------------------
发件人: @.>;
发送时间: 2025年6月16日(星期一) 晚上7:24
收件人: @.>;
抄送: @.>; @.>;
主题: Re: [Project-MONAI/research-contributions] Evaluation results are inconsistent during training and after saving the trained model (Issue #405)

胡昌留下评论（项目-MONAI/研究贡献#405）

@ras-lyg你好，你解决这个问题了吗，有什么解决办法吗？

直接回复这封邮件，在GitHub上查看它，或者取消订阅.
@.*** >

方便告知是哪里的问题吗，Thanks

[ras-lyg]

方便的话，请添加我的Q：381306473   裴训龙 ***@***.***  

…

------------------ 原始邮件 ------------------ 发件人: ***@***.***>; 发送时间: 2025年7月3日(星期四) 晚上6:13 收件人: ***@***.***>; 抄送: ***@***.***>; ***@***.***>; 主题: Re: [Project-MONAI/research-contributions] Evaluation results are inconsistent during training and after saving the trained model (Issue #405) hhuxuchang left a comment (Project-MONAI/research-contributions#405) 是的，我已经解决了，问题出在测试程序上  裴训龙 @.***   …

------------------ 原始邮件 ------------------ 发件人: @.>; 发送时间: 2025年6月16日(星期一) 晚上7:24 收件人: @.>; 抄送: @.>; @.>; 主题: Re: [Project-MONAI/research-contributions] Evaluation results are inconsistent during training and after saving the trained model (Issue #405) 胡昌留下评论（项目-MONAI/研究贡献#405） @ras-lyg你好，你解决这个问题了吗，有什么解决办法吗？ 直接回复这封邮件，在GitHub上查看它，或者取消订阅. @.*** > 方便告知是哪里的问题吗，Thanks — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: ***@***.***>