Propose fix perceptual loss sqrt nan - #8414
Conversation
|
Thanks for the update, the changes looks fine to me. |
There was a problem hiding this comment.
This file should go into an appropriate subdirectory in the tests directory. We've changed the directory structure there recently so probably tests/losses.
| # Create tensor | ||
| x = torch.zeros(2, 3, 10, 10, requires_grad=True) | ||
|
|
||
| optimizer = optim.Adam([x], lr=0.01) |
There was a problem hiding this comment.
I don't think the optimizer is needed for this test?
| x = torch.zeros(2, 3, 10, 10, requires_grad=True) | ||
|
|
||
| optimizer = optim.Adam([x], lr=0.01) | ||
| x_scaled = x * scale |
There was a problem hiding this comment.
Since x is all 0, x_scaled is always going to be 0 unless you're expected float imprecision to create values here. If so, I would add a comment to mention this.
There was a problem hiding this comment.
I don't understand the point of this test with regards to the next one; instead of a zeros tensor, couldn't it be a random one which will be then multiplied by a really small number?
virginiafdez
left a comment
virginiafdez
left a comment
There was a problem hiding this comment.
The changes look good to me. I'd modify one of the tests, but the rest is fine.
| x = torch.zeros(2, 3, 10, 10, requires_grad=True) | ||
|
|
||
| optimizer = optim.Adam([x], lr=0.01) | ||
| x_scaled = x * scale |
There was a problem hiding this comment.
I don't understand the point of this test with regards to the next one; instead of a zeros tensor, couldn't it be a random one which will be then multiplied by a really small number?
virginiafdez
left a comment
virginiafdez
left a comment
There was a problem hiding this comment.
Besides my comment about the point of one of the tests, I think this PR can be merged, as long as the errors happening on the automatic tests are fixed.
|
Hi @cvbourne, could you please help resolve the DCO issue and also help take a look at the failed pipeline? Thanks. |
|
Note Currently processing new changes in this PR. This may take a few minutes, please wait... ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Team Run ID: 📒 Files selected for processing (1)
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Team Run ID: 📒 Files selected for processing (1)
🚧 Files skipped from review as they are similar to previous changes (1)
Included review availability: Your plan provides up to 8 included reviews per hour; 6 remain after this review. 📝 WalkthroughWalkthrough
Estimated code review effort: 2 (Simple) | ~12 minutes Merge Risk: ⚪ Minimal · up to Tensor normalization now remains stable for zero and very small inputs, preventing NaN gradients in perceptual-loss usage. The targeted gradient-stability coverage supports merge readiness. 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
♻️ Duplicate comments (1)
tests/test_perceptual_loss_stability.py (1)
36-39:⚠️ Potential issue | 🟡 MinorZero tensor negates scale parameter.
torch.zeros * scaleis always zeros. To test small values, use random tensor:Proposed fix
- x = torch.zeros(2, 3, 10, 10, requires_grad=True) - - optimizer = optim.Adam([x], lr=0.01) - x_scaled = x * scale + x = torch.randn(2, 3, 10, 10, requires_grad=True) + x_scaled = x * scaleThis also addresses the unused
nameparameter warning (ARG002) since parameterized tests require the name argument for test identification.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@tests/test_perceptual_loss_stability.py` around lines 36 - 39, The test currently creates x = torch.zeros(...) then computes x_scaled = x * scale which is always zero; replace the zero tensor with a small random tensor (e.g., torch.randn(...) * small_factor or torch.empty(...).normal_(mean=0, std=small_value)) so scaling actually affects values, and ensure the parameterized test's name parameter is used (or include it in the test id) to avoid the unused-name warning; update references to x, x_scaled, optimizer in the test_perceptual_loss_stability test accordingly.
🧹 Nitpick comments (2)
tests/test_perceptual_loss_stability.py (1)
17-17: Remove unused import.
torch.optimimport can be removed when optimizer is deleted.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@tests/test_perceptual_loss_stability.py` at line 17, Remove the now-unused import "import torch.optim as optim" from tests/test_perceptual_loss_stability.py because the optimizer was removed; search for the symbol "optim" and delete the import statement so the file no longer contains an unused dependency.monai/losses/perceptual.py (1)
277-279: Missing docstring.Per coding guidelines, add a docstring describing parameters (
x,eps) and return value.Proposed docstring
def normalize_tensor(x: torch.Tensor, eps: float = 1e-8) -> torch.Tensor: + """Normalize tensor across channel dimension. + + Args: + x: Input tensor with shape (B, C, ...). + eps: Small constant for numerical stability. + + Returns: + Channel-normalized tensor with same shape as input. + """ norm_factor = torch.sqrt(torch.sum(x**2, dim=1, keepdim=True) + eps) - return x / (norm_factor + eps) + return x / norm_factor🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@monai/losses/perceptual.py` around lines 277 - 279, Add a proper docstring to the normalize_tensor function that documents parameters and return value: describe param x (torch.Tensor) as the input tensor with channel dimension at dim=1, param eps (float) as the small numerical epsilon used to stabilize the norm computation, and describe the return as a torch.Tensor containing the input scaled by the L2 norm per-channel (with shape preserved). Also mention behavior with shapes and that eps prevents division by zero and the dtype of the returned tensor matches the input.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@monai/losses/perceptual.py`:
- Around line 277-279: The normalize_tensor function currently applies eps twice
(inside the sqrt and again added to norm_factor), so change normalize_tensor to
only add eps inside the sqrt: compute norm_factor = torch.sqrt(torch.sum(x**2,
dim=1, keepdim=True) + eps) and return x / norm_factor (remove the extra + eps
in the denominator) to avoid double-counting the epsilon while keeping numerical
stability.
---
Duplicate comments:
In `@tests/test_perceptual_loss_stability.py`:
- Around line 36-39: The test currently creates x = torch.zeros(...) then
computes x_scaled = x * scale which is always zero; replace the zero tensor with
a small random tensor (e.g., torch.randn(...) * small_factor or
torch.empty(...).normal_(mean=0, std=small_value)) so scaling actually affects
values, and ensure the parameterized test's name parameter is used (or include
it in the test id) to avoid the unused-name warning; update references to x,
x_scaled, optimizer in the test_perceptual_loss_stability test accordingly.
---
Nitpick comments:
In `@monai/losses/perceptual.py`:
- Around line 277-279: Add a proper docstring to the normalize_tensor function
that documents parameters and return value: describe param x (torch.Tensor) as
the input tensor with channel dimension at dim=1, param eps (float) as the small
numerical epsilon used to stabilize the norm computation, and describe the
return as a torch.Tensor containing the input scaled by the L2 norm per-channel
(with shape preserved). Also mention behavior with shapes and that eps prevents
division by zero and the dtype of the returned tensor matches the input.
In `@tests/test_perceptual_loss_stability.py`:
- Line 17: Remove the now-unused import "import torch.optim as optim" from
tests/test_perceptual_loss_stability.py because the optimizer was removed;
search for the symbol "optim" and delete the import statement so the file no
longer contains an unused dependency.
ℹ️ Review info
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Cache: Disabled due to data retention organization setting
Knowledge base: Disabled due to Reviews -> Disable Knowledge Base setting
📒 Files selected for processing (2)
monai/losses/perceptual.pytests/test_perceptual_loss_stability.py
|
Hi @cvbourne if you can come back to this one and address the issues raised by us and Coderabbit, we'd still like to merge this one. Thanks! |
Signed-off-by: Eric Kerfoot <17726042+ericspod@users.noreply.github.com>
Signed-off-by: Eric Kerfoot <17726042+ericspod@users.noreply.github.com>
|
Note GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer. |
|
I think this PR has been superseded now so closing. |
4 participants
Fixes # 8412
Description
This PR fixes a numerical stability issue in the PerceptualLoss implementation where the
normalize_tensorfunction can produce NaN gradients when the input values are very small.Types of changes
./runtests.sh -f -u --net --coverage../runtests.sh --quick --unittests --disttests.