Design: ThinkingBox x Agent Lightning stateful MCP training example

[hamedrabah]

Ownership question

Would maintainers prefer this integration to live in:

1. microsoft/agent-lightning as examples/thinkingbox/;
2. microsoft/thinkingbox as an Agent Lightning recipe; or
3. a small split across both repositories?

Proposed thin slice

ThinkingBox already exposes executable scalar rewards through TestResult.reward and identifies decode/test infrastructure errors with is_system_error. Agent Lightning already supplies a rollout-scoped OpenAI-compatible endpoint plus explicit succeeded/failed rollout states.

I propose one bundled cloud_drive example that:

• points ThinkingBox's existing AOAISession(endpoint_url=...) at $AGL_OPENAI_BASE_URL/chat/completions;
• emits an Agent Lightning reward only for a valid TestResult;
• treats decode errors, test-system errors, timeouts, and exhausted gateway failures as failed rollouts, never as zero reward;
• includes a credential-free mocked smoke test;
• reports pass@1, all-k reliability (pass^k), and consecutive-run/long-horizon reliability;
• documents local execution first, with Kubernetes and the full ThinkingBox dataset as optional follow-ups.

This should require no ThinkingBox core change for the initial version.

Acceptance criteria

• The mocked path covers gateway -> ThinkingBox agent -> MCP tool -> executable test -> reward event.
• A valid assertion failure can produce reward 0; infrastructure failures produce a failed rollout and no reward event.
• The local recipe runs the bundled scenario without external data or credentials.

If Agent Lightning should own this, I can send a focused PR with the example, tests, and documentation.

[hzy46]

Hi Hamed Rabah (@hamedrabah) both options are ok for us. If you prefer to put in our example, we will put a label in that example that you are the maintainer.