shifter init fix - #247
Conversation
pollytur
left a comment
There was a problem hiding this comment.
@NathanSoeding thanks a lot for the PR!
see the comments to address
Ideally please add a few unit tests to check that initialization now works correctly for both MLP(Shifter) and class MLPShifter(ModuleDict) to make sure we are not missing any other bugs :)
|
@NathanSoeding any updates on it? It would be great if we could finalise it soon :) |
|
I committed the changes |
There was a problem hiding this comment.
Pull request overview
This PR fixes MLP shifter weight initialization by ensuring the initialization routine is actually invoked and by correctly initializing nn.Linear layer weights (instead of incorrectly checking Parameter objects as nn.Linear). It also adds a regression test to ensure Xavier initialization is used.
Changes:
- Call
MLP.initialize()fromMLP.__init__and correctly apply Xavier init tonn.Linearlayers (plus zero-initialize biases). - Remove the previous (incorrect) parameter-based initialization logic.
- Add a unit test that patches
xavier_normal_and asserts it is called once per Linear layer.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
neuralpredictors/layers/shifters/mlp.py |
Fixes and triggers MLP shifter initialization; changes init behavior and removes container-level init method. |
test/test_shifter_init.py |
Adds a regression test verifying Xavier initialization is applied for each shift layer. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| import torch | ||
| from torch import nn | ||
| from unittest.mock import patch |
There was a problem hiding this comment.
and also maybe add a sweep that you sent several values of the MLP layers and not just hardcoded 3 - in experanto there should be examples how to parametric tests with pytest like here https://github.com/sensorium-competition/experanto/blob/main/tests/test_sequence_interpolator.py#L14-L16
If pytest is not a part of neural predictors - ignore this comment and just remove the unused inputs
Have you run the tests locally?
|
@pollytur I removed unused library imports and added added the module dict initialization function which calls the individual shifter initialize() functions |
shifter initialization function was never called and it checked if parameters are instances of nn.Linear which is always wrong because they are tensors.