A lot of warning for tf.function retracing

[ymd-h]

From TF 2.3, input signature of tf.function graph has changed and our unit tests produces a lot of retracing warnings.
https://www.tensorflow.org/guide/function#tracing_semantics

One of the example warnings is following; (I add line break for readability)

WARNING:tensorflow:6 out of the last 11 calls to <function VPG._get_action_body at 0x7f5d92de6940> triggered tf.function retracing.
Tracing is expensive and the excessive number of tracings could be due to 
(1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. 
For (1), please define your @tf.function outside of the loop. 
For (2), @tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. 
For (3), please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for  more details.

I don't think we do (1), so I think candidates are (2) and (3).

I will make patch for (3).
((2) is not so simple and requires some investigation for usage.)