[Do not merge] Ensure OnStart/OnStop hooks can only be called once

[sywhang]

This is the same as #931.

Leaving this open for us to work on #945 and then merge.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 98.72%. Comparing base (2f5adb6) to head (caffadb).
⚠️ Report is 201 commits behind head on master.

Additional details and impacted files

@@           Coverage Diff           @@
##           master     #950   +/-   ##
=======================================
  Coverage   98.72%   98.72%           
=======================================
  Files          30       30           
  Lines        1332     1337    +5     
=======================================
+ Hits         1315     1320    +5     
  Misses         11       11           
  Partials        6        6           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.