Reduce use of worker maintenance toggling #8
Description
Right now the scale down operation uses the worker maintenance toggle very liberally. In psuedocode it looks approx like:
IF scale down criteria is met THEN
put all online workers in maintenance
IF there are enough online workers to scale down THEN
find _n_ candidates to stop based on idleness and/or billing hour usage
IF stop candidate(s) found THEN
issue stop command to candidate(s)
release all non-stopped workers from maintenance state
This leads to a lot of unnecessary maintenance toggling because the majority of the time, even if the scale down criteria is met, there are additional conditions that prevent the stopping of any workers (none idle or only used a small portion of their billing hour).
A better approach would be to identify candidates to stop as soon as the scale down criteria is met. Then for each candidate, put it in maintenance, verify that it is still idle, and stop it if it is.