Skip to content

Commit 80b40a3

Browse files
AbirAbbasclaude
andauthored
feat: configurable reasoning endpoint + provider regression test suite (#3)
* test: add characterization suite locking documented provider behavior Pins the README happy-path before the provider config is refactored, so any regression is caught: clean-env defaults, documented model/env overrides, the OpenRouter media call contract (TTS/image/video + per-beat fallbacks), pure render helpers, the missing-key gates, and the SDK runtime patches. No real network calls — providers are faked and media bytes synthetic; ffmpeg-dependent tests skip if the binaries are absent. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * ci: run pytest and ruff on push and pull_request Installs ffmpeg, pins Python 3.11 to match the Dockerfile, lints the test suite, and runs the full pytest suite. Lint scope is tests/ for now; src/ carries pre-existing ruff debt tracked separately. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: allow custom reasoning endpoint via REEL_AF_API_BASE/REEL_AF_API_KEY Reasoning .ai() calls now resolve their key and base URL from optional env vars, falling back to OpenRouter when unset. REEL_AF_API_KEY falls back to OPENROUTER_API_KEY; an empty REEL_AF_API_BASE (Docker ${VAR:-}) falls back to the OpenRouter default rather than clobbering it. Purely additive — with both vars unset the flow is byte-identical to before. Media (TTS/image/ video) still routes through OpenRouter, so OPENROUTER_API_KEY stays required. New vars plumbed through docker-compose and .env.example; tests lock the override, precedence, fallback, and empty-string cases. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: document the bring-your-own reasoning endpoint Adds an advanced section near the bottom of the README explaining how to point reasoning at an OpenAI-compatible endpoint (local vLLM/Ollama, a self-hosted gateway), framed as the non-default path, with the two caveats (media stays on OpenRouter; use the openai/ prefix). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 parent d1367f1 commit 80b40a3

12 files changed

Lines changed: 778 additions & 2 deletions

.env.example

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,18 @@ REEL_AF_MODEL=openrouter/deepseek/deepseek-v4-pro
1717
# REEL_AF_MODEL=openrouter/qwen/qwen3-235b-a22b-instruct
1818
# REEL_AF_MODEL=openrouter/google/gemini-2.5-pro
1919

20+
# --- Advanced: bring your own reasoning endpoint (optional) ---
21+
# Leave both unset to use OpenRouter (the default). To run reasoning against
22+
# any OpenAI-compatible endpoint — a local vLLM/Ollama server, a self-hosted
23+
# gateway, or another aggregator — set the model to that server's id and point
24+
# these at it. Media (TTS/image/video) still uses OpenRouter, so keep
25+
# OPENROUTER_API_KEY set.
26+
# REEL_AF_MODEL=openai/your-local-model
27+
# REEL_AF_API_BASE=http://localhost:8000/v1
28+
# REEL_AF_API_KEY=sk-local-or-anything
29+
# REEL_AF_API_BASE=
30+
# REEL_AF_API_KEY=
31+
2032
# --- Media models ---
2133
# TTS: Gemini Flash supports 200+ inline tone tags.
2234
REEL_AF_TTS_MODEL=google/gemini-3.1-flash-tts-preview

.github/workflows/ci.yml

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
name: CI
2+
3+
on:
4+
push:
5+
branches: [main]
6+
pull_request:
7+
8+
jobs:
9+
test:
10+
runs-on: ubuntu-latest
11+
steps:
12+
- uses: actions/checkout@v4
13+
14+
# ffmpeg/ffprobe are a hard runtime requirement (karaoke render path);
15+
# the render tests exercise the real binaries.
16+
- name: Install ffmpeg
17+
run: sudo apt-get update && sudo apt-get install -y --no-install-recommends ffmpeg
18+
19+
- name: Install uv
20+
uses: astral-sh/setup-uv@v5
21+
22+
- name: Pin Python (matches the Dockerfile)
23+
run: uv python install 3.11
24+
25+
# Lint the test suite. NOTE: src/ carries some pre-existing ruff debt
26+
# tracked separately; scope here is the code this CI guards.
27+
- name: Lint tests
28+
run: uv run --extra dev ruff check tests/
29+
30+
- name: Run tests
31+
run: uv run --extra dev python -m pytest tests/ -q

README.md

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -260,6 +260,32 @@ Voice, pacing, and tone are picked in code (`render/tts.py:_VOICE_BY_TONE` and t
260260

261261
---
262262

263+
## Bring your own model (advanced)
264+
265+
OpenRouter is the recommended path — one key, every model, zero config. But if you'd rather run **reasoning** against your own OpenAI-compatible endpoint — a local [vLLM](https://github.com/vllm-project/vllm) / [Ollama](https://ollama.com/) server, a self-hosted gateway, or a different aggregator — you can, without touching code:
266+
267+
```bash
268+
REEL_AF_MODEL=openai/your-model-id # how your endpoint names the model
269+
REEL_AF_API_BASE=http://localhost:8000/v1 # your OpenAI-compatible base URL
270+
REEL_AF_API_KEY=sk-local-or-anything # your endpoint's key
271+
```
272+
273+
Leave `REEL_AF_API_BASE` / `REEL_AF_API_KEY` unset and everything points at OpenRouter exactly as before — this is strictly additive, the default flow is unchanged.
274+
275+
| Env var | Default | What it controls |
276+
|---|---|---|
277+
| `REEL_AF_API_BASE` | OpenRouter (`https://openrouter.ai/api/v1`) | Base URL for reasoning `.ai()` calls. Empty = OpenRouter. |
278+
| `REEL_AF_API_KEY` | falls back to `OPENROUTER_API_KEY` | Key sent to the reasoning endpoint. |
279+
280+
Two caveats:
281+
282+
- **Media still uses OpenRouter.** TTS, image, and Veo generation route through OpenRouter today, so keep `OPENROUTER_API_KEY` set even when reasoning is self-hosted. Configurable per-provider media endpoints are tracked in [issue #2](https://github.com/Agent-Field/reels-af/issues/2).
283+
- **Use the `openai/` prefix** for OpenAI-compatible servers so the request is shaped correctly; `REEL_AF_API_BASE` then redirects it to your host.
284+
285+
This intentionally isn't the headline workflow — it adds moving parts. For most users, the one-key OpenRouter default is the simpler path.
286+
287+
---
288+
263289
## Troubleshooting
264290

265291
**"OPENROUTER_API_KEY not set in env."** — paste your key into `.env`. The Docker container reads it via `docker-compose.yml`; the CLI reads it via `python-dotenv`.

docker-compose.yml

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -23,6 +23,10 @@ services:
2323
AGENT_NODE_ID: ${AGENT_NODE_ID:-reel-af}
2424
OPENROUTER_API_KEY: ${OPENROUTER_API_KEY:-}
2525
REEL_AF_MODEL: ${REEL_AF_MODEL:-openrouter/deepseek/deepseek-v4-pro}
26+
# Advanced: point reasoning at any OpenAI-compatible endpoint. Empty
27+
# values are treated as unset and fall back to OpenRouter.
28+
REEL_AF_API_BASE: ${REEL_AF_API_BASE:-}
29+
REEL_AF_API_KEY: ${REEL_AF_API_KEY:-}
2630
REEL_AF_TTS_MODEL: ${REEL_AF_TTS_MODEL:-google/gemini-3.1-flash-tts-preview}
2731
REEL_AF_IMAGE_MODEL: ${REEL_AF_IMAGE_MODEL:-openrouter/google/gemini-2.5-flash-image}
2832
REEL_AF_VIDEO_MODEL: ${REEL_AF_VIDEO_MODEL:-openrouter/google/veo-3.1-lite}

src/reel_af/app.py

Lines changed: 9 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -66,8 +66,15 @@
6666
description="URL or topic → vertical viral reel via a multi-reasoner DAG.",
6767
ai_config=AIConfig(
6868
model=os.getenv("REEL_AF_MODEL", "openrouter/deepseek/deepseek-v4-pro"),
69-
api_key=os.environ.get("OPENROUTER_API_KEY", ""),
70-
api_base="https://openrouter.ai/api/v1",
69+
# Reasoning endpoint defaults to OpenRouter. Advanced users can point
70+
# `.ai()` calls at any OpenAI-compatible endpoint (local vLLM/Ollama,
71+
# a self-hosted gateway, another aggregator) without code changes:
72+
# REEL_AF_API_KEY overrides the key (falls back to OPENROUTER_API_KEY)
73+
# REEL_AF_API_BASE overrides the base URL (empty/unset → OpenRouter)
74+
# Media (TTS/image/video) still routes through OpenRouter, so
75+
# OPENROUTER_API_KEY remains required. See README "Bring your own model".
76+
api_key=os.getenv("REEL_AF_API_KEY") or os.environ.get("OPENROUTER_API_KEY", ""),
77+
api_base=os.getenv("REEL_AF_API_BASE") or "https://openrouter.ai/api/v1",
7178
),
7279
dev_mode=True,
7380
)

tests/conftest.py

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
"""Test bootstrap.
2+
3+
Adds the ``src`` layout and the ``tests`` directory to ``sys.path`` so the
4+
``reel_af`` package and the local ``util`` helper module import cleanly
5+
whether or not the project has been ``pip install``-ed.
6+
"""
7+
8+
from __future__ import annotations
9+
10+
import sys
11+
from pathlib import Path
12+
13+
_ROOT = Path(__file__).resolve().parent.parent
14+
_SRC = _ROOT / "src"
15+
_TESTS = Path(__file__).resolve().parent
16+
17+
for _p in (str(_SRC), str(_TESTS)):
18+
if _p not in sys.path:
19+
sys.path.insert(0, _p)

tests/test_byo_endpoint.py

Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,44 @@
1+
"""Tier A — bring-your-own reasoning endpoint.
2+
3+
Locks the additive override path introduced by ``REEL_AF_API_BASE`` /
4+
``REEL_AF_API_KEY``. These coexist with the OpenRouter defaults pinned in
5+
``test_provider_config.py``: with both vars unset, config resolves to
6+
OpenRouter exactly as before. Each test maps to a validation-contract item.
7+
"""
8+
9+
from __future__ import annotations
10+
11+
from util import run_config_probe
12+
13+
DEFAULT_API_BASE = "https://openrouter.ai/api/v1"
14+
15+
# Clear the BYO vars by default so the host environment can't leak in; each
16+
# test sets only what it exercises.
17+
_CLEAR = {"REEL_AF_API_BASE": None, "REEL_AF_API_KEY": None}
18+
19+
20+
def test_api_base_override():
21+
# CA2 — a custom endpoint is honoured verbatim.
22+
cfg = run_config_probe({**_CLEAR, "REEL_AF_API_BASE": "http://localhost:8000/v1"})
23+
assert cfg["api_base"] == "http://localhost:8000/v1"
24+
25+
26+
def test_api_key_override_takes_precedence_over_openrouter_key():
27+
# CA3 — REEL_AF_API_KEY wins over OPENROUTER_API_KEY when both are set.
28+
cfg = run_config_probe(
29+
{**_CLEAR, "OPENROUTER_API_KEY": "sk-openrouter", "REEL_AF_API_KEY": "sk-byo-endpoint"}
30+
)
31+
assert cfg["api_key"] == "sk-byo-endpoint"
32+
33+
34+
def test_api_key_falls_back_to_openrouter_when_byo_unset():
35+
# CA4 — without REEL_AF_API_KEY, OPENROUTER_API_KEY still supplies the key.
36+
cfg = run_config_probe({**_CLEAR, "OPENROUTER_API_KEY": "sk-openrouter"})
37+
assert cfg["api_key"] == "sk-openrouter"
38+
39+
40+
def test_empty_api_base_falls_back_to_openrouter():
41+
# CA5 — Docker passes `${REEL_AF_API_BASE:-}` as "" when unset; an empty
42+
# string must NOT clobber the default endpoint.
43+
cfg = run_config_probe({**_CLEAR, "REEL_AF_API_BASE": ""})
44+
assert cfg["api_base"] == DEFAULT_API_BASE

tests/test_gates_and_patches.py

Lines changed: 99 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,99 @@
1+
"""API-key gates and the SDK runtime patches.
2+
3+
The provider refactor will relax the hard ``OPENROUTER_API_KEY`` requirement,
4+
so these pin the *current* gate behaviour to make any change deliberate and
5+
visible. The patch tests guard the end-to-end-callability fixes the README
6+
depends on (data-URL images, video-download auth, the /audio/speech route).
7+
"""
8+
9+
from __future__ import annotations
10+
11+
import base64
12+
import io
13+
14+
import pytest
15+
from PIL import Image
16+
17+
# ───── missing-key gates ─────────────────────────────────────────────
18+
19+
20+
async def test_article_to_reel_errors_without_key(monkeypatch):
21+
monkeypatch.delenv("OPENROUTER_API_KEY", raising=False)
22+
import reel_af.app as app
23+
24+
result = await app.article_to_reel(url="https://example.com/x")
25+
assert result == {"error": "OPENROUTER_API_KEY not set in env."}
26+
27+
28+
async def test_topic_to_reel_errors_without_key(monkeypatch):
29+
monkeypatch.delenv("OPENROUTER_API_KEY", raising=False)
30+
import reel_af.app as app
31+
32+
result = await app.topic_to_reel(topic="the placebo effect")
33+
assert result == {"error": "OPENROUTER_API_KEY not set in env."}
34+
35+
36+
def test_cli_require_key_raises_when_unset(monkeypatch):
37+
monkeypatch.delenv("OPENROUTER_API_KEY", raising=False)
38+
import reel_af.cli as cli
39+
40+
with pytest.raises(SystemExit):
41+
cli._require_key()
42+
43+
44+
def test_cli_require_key_passes_when_set(monkeypatch):
45+
monkeypatch.setenv("OPENROUTER_API_KEY", "sk-or-test")
46+
import reel_af.cli as cli
47+
48+
cli._require_key() # must not raise
49+
50+
51+
# ───── sdk patches ───────────────────────────────────────────────────
52+
53+
54+
def test_generate_speech_method_is_added():
55+
from agentfield.media_providers import OpenRouterProvider
56+
57+
import reel_af.sdk_patches # noqa: F401
58+
59+
assert hasattr(OpenRouterProvider, "generate_speech")
60+
61+
62+
def test_generate_video_is_patched():
63+
from agentfield.media_providers import OpenRouterProvider
64+
65+
import reel_af.sdk_patches # noqa: F401
66+
67+
assert getattr(OpenRouterProvider.generate_video, "__reel_af_patched__", False) is True
68+
69+
70+
def test_image_output_save_decodes_data_urls(tmp_path):
71+
from agentfield.multimodal_response import ImageOutput
72+
73+
import reel_af.sdk_patches # noqa: F401
74+
75+
# A 4x4 red PNG encoded as a data: URL — Gemini returns these.
76+
buf = io.BytesIO()
77+
Image.new("RGB", (4, 4), (255, 0, 0)).save(buf, format="PNG")
78+
raw = buf.getvalue()
79+
data_url = "data:image/png;base64," + base64.b64encode(raw).decode()
80+
81+
out = tmp_path / "decoded.png"
82+
ImageOutput(url=data_url).save(out)
83+
84+
assert out.read_bytes() == raw # decoded locally, no network fetch
85+
86+
87+
def test_apply_all_is_idempotent():
88+
from agentfield.media_providers import OpenRouterProvider
89+
90+
import reel_af.sdk_patches as patches
91+
92+
before = OpenRouterProvider.generate_video
93+
patches.apply_all() # second application
94+
patches.apply_all() # third
95+
after = OpenRouterProvider.generate_video
96+
97+
assert after is before # no re-wrapping
98+
assert getattr(after, "__reel_af_patched__", False) is True
99+
assert hasattr(OpenRouterProvider, "generate_speech")

tests/test_provider_config.py

Lines changed: 77 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,77 @@
1+
"""Lock the documented provider/config contract (README "Customize" table).
2+
3+
These characterise the env-var → config resolution that the upcoming
4+
"bring your own provider" change will refactor. They must keep passing
5+
afterwards: the new behaviour is purely additive, so the OpenRouter
6+
defaults and the already-documented overrides cannot regress.
7+
"""
8+
9+
from __future__ import annotations
10+
11+
from util import run_config_probe
12+
13+
# README-documented defaults (Customize table + .env.example).
14+
DEFAULT_MODEL = "openrouter/deepseek/deepseek-v4-pro"
15+
DEFAULT_API_BASE = "https://openrouter.ai/api/v1"
16+
DEFAULT_TTS = "google/gemini-3.1-flash-tts-preview"
17+
DEFAULT_IMAGE = "openrouter/google/gemini-2.5-flash-image"
18+
DEFAULT_VIDEO = "openrouter/google/veo-3.1-lite"
19+
20+
# Every model-selecting env var, cleared so we observe true defaults.
21+
_CLEAR_MODELS = {
22+
"REEL_AF_MODEL": None,
23+
"REEL_AF_TTS_MODEL": None,
24+
"REEL_AF_IMAGE_MODEL": None,
25+
"REEL_AF_VIDEO_MODEL": None,
26+
"REEL_AF_USE_VEO": None,
27+
}
28+
29+
30+
def test_clean_env_resolves_documented_defaults():
31+
cfg = run_config_probe(_CLEAR_MODELS)
32+
assert cfg["model"] == DEFAULT_MODEL
33+
assert cfg["api_base"] == DEFAULT_API_BASE
34+
assert cfg["tts_default"] == DEFAULT_TTS
35+
assert cfg["image_model"] == DEFAULT_IMAGE
36+
assert cfg["video_model"] == DEFAULT_VIDEO
37+
assert cfg["use_veo"] is False
38+
39+
40+
def test_api_key_sourced_from_openrouter_key():
41+
cfg = run_config_probe({**_CLEAR_MODELS, "OPENROUTER_API_KEY": "sk-probe-123"})
42+
assert cfg["api_key"] == "sk-probe-123"
43+
44+
45+
def test_reel_af_model_override():
46+
# Exact example from the README quick-start.
47+
override = "openrouter/anthropic/claude-sonnet-4"
48+
cfg = run_config_probe({**_CLEAR_MODELS, "REEL_AF_MODEL": override})
49+
assert cfg["model"] == override
50+
# Overriding the reasoning model leaves the endpoint untouched.
51+
assert cfg["api_base"] == DEFAULT_API_BASE
52+
53+
54+
def test_media_model_overrides():
55+
cfg = run_config_probe(
56+
{
57+
**_CLEAR_MODELS,
58+
"REEL_AF_TTS_MODEL": "google/some-tts",
59+
"REEL_AF_IMAGE_MODEL": "openrouter/black-forest-labs/flux-1.1-pro",
60+
"REEL_AF_VIDEO_MODEL": "openrouter/google/veo-3.1",
61+
}
62+
)
63+
assert cfg["tts_default"] == DEFAULT_TTS # constant is the *fallback*, not the override
64+
assert cfg["image_model"] == "openrouter/black-forest-labs/flux-1.1-pro"
65+
assert cfg["video_model"] == "openrouter/google/veo-3.1"
66+
67+
68+
def test_use_veo_truthy_values_enable_veo():
69+
for truthy in ("true", "TRUE", "True", "1", "yes"):
70+
cfg = run_config_probe({**_CLEAR_MODELS, "REEL_AF_USE_VEO": truthy})
71+
assert cfg["use_veo"] is True, truthy
72+
73+
74+
def test_use_veo_falsey_values_keep_kenburns():
75+
for falsey in ("false", "0", "no", "", "off"):
76+
cfg = run_config_probe({**_CLEAR_MODELS, "REEL_AF_USE_VEO": falsey})
77+
assert cfg["use_veo"] is False, falsey

0 commit comments

Comments
 (0)