Question about the weight on the hidden benchmark

[sherlock55341]

I noticed that the weights of PPA in the hidden cases is much smaller than those in the public cases.
I wonder if it means the tns, wns and power is less important in the hidden cases.

[sherlock55341]

I think there may be some typos in the table of results of all the testcases.
Will you fix it?

[liangrj2014]

Hi, what typos do you refer to?

[sherlock55341]

Hi, what typos do you refer to?

Some of TNS is positive and power is negative.
In ariane (blind), wns is much better than the wns_{ref}, but the tns is the same.

[liangrj2014]

Thanks for catching the typos! I've just fixed them. Most of them were incorrect signs that occurred during copying.

Regarding the ariane (blind), I just confirmed that the numbers are correct. They might look weird since we only evaluated the solutions of a few teams and the reference results might be biased

[liangrj2014]

Regarding the impact of TNS/WNS and power on the final scores in the hidden test cases, I still observe that the timing and power scores of different teams significantly influence the overall rankings.

[sherlock55341]

Regarding the impact of TNS/WNS and power on the final scores in the hidden test cases, I still observe that the timing and power scores of different teams significantly influence the overall rankings.

If the weights are updated, will you re-run the router to input the new weights?

[liangrj2014]

The weights assigned to the hidden test cases are final and will remain unchanged.

[sherlock55341]

The weights assigned to the hidden test cases are final and will remain unchanged.

I see. Thanks for your help. The case Ariane has a big gap of w1, w4 ratio in visible and blind version, which makes me confused.

[liangrj2014]

Apologies for the confusion! The weight settings were determined in a somewhat ad-hoc manner as we have not yet found a principled approach for assigning them. However, to ensure a fair evaluation, the current weight configuration is final.