Remove or move permute after mean #3235

Job	Run time
test-xnnpack / test-backend-linux (xnnpack, operators) / linux-job	14m 55s
test-xnnpack / test-backend-linux (xnnpack, models) / linux-job	52m 42s
test-xnnpack / test-backend-macos	0s
test-xnnpack / package-golden-artifacts	3m 44s
	1h 11m 21s

Provide feedback