Timbre as texture: multi-timescale modulation, spectral homogeneity, medium residue

Replace per-stream filter/drive/reverb with a TextureProfile that describes each
element faithfully:
- rock (terrain): homogeneous + gentle slow envelope + viscous lava residue tail
- fire (atmosphere): broadband + fast chaotic flicker under a slow drift
- agents (entity): kept clean — conscious beings are described by feeling, not
  texture (their character is the sparse unison motif)
Improves scene-tracking (lava corr 0.43->0.63, high band 0.75) and is the same
parameterization a per-feature scalogram would later extract from video.

Author: hschn58

DESIGN.md

@@ -40,17 +40,34 @@ things downstream:
 - **A recurring motif** — the lead is a short theme that is transposed/retrograded
   per segment, so the piece feels composed rather than random.
 
-## Timbre: a voice per stream
-
-Each stream has its own **TimbreKit** (in the genre preset, swappable), and the
-timbre is also modulated by the scene over time:
-
-- `drive` (saturation) follows **tension**
-- a time-varying low-pass opens with **brightness**
-- per-kit `reverb` (space) and `tremolo` (movement)
-
-Cutoffs are chosen so the streams stay spectrally separated (the mid stream sits
-below the high band), which keeps both the mix and the QC gate legible.
+## Timbre: describing each element's texture
+
+Timbre is not a coating on a note — it is the **sonification of the real
+element**. Each stream has a **TextureProfile** (in the genre preset, swappable)
+that describes its element as faithfully as possible:
+
+- **spectral homogeneity** (`bandwidth`): homogeneous/tonal ↔ broadband/noisy
+- **multi-timescale amplitude modulation**: a `slow` drift + a `fast` flicker,
+  where `chaos` makes the flicker noise-driven rather than a clean LFO
+- **medium / residue**: a viscous, dark decay tail — the substance the element
+  sits in
+- saturation (`drive`, follows **tension**) and a low-pass that opens with
+  **brightness**
+
+The three archetypes fall straight out of this:
+
+| Stream | Element | Profile |
+|---|---|---|
+| terrain | rock | homogeneous, narrow; gentle slow in-and-out; a **viscous lava residue** tail |
+| atmosphere | fire | broadband; a **fast chaotic flicker** under a **slow drift** (the flame's wandering centre of mass) |
+| entity | agents | kept clean — conscious beings are described by *feeling*, carried by the sparse unison motif, not by physical texture |
+
+A conscious agent can't be captured by texture, so the entity stream is
+deliberately untextured; its character comes from the composition (sparse,
+collective-unison motif). Cutoffs keep the streams spectrally separated (mid sits
+below the high band) so the mix and QC gate stay legible. Note these profiles are
+the *same object* a per-feature scalogram would measure — the bridge to driving
+timbre from real video texture later.
 
 ## Architecture
 

src/music_making/genre.py

@@ -9,22 +9,30 @@
 
 from dataclasses import dataclass, field
 
-from .timbre import TimbreKit
+from .timbre import TextureProfile
 
 
-def _smooth_funk_timbres() -> dict[str, TimbreKit]:
+def _smooth_funk_timbres() -> dict[str, TextureProfile]:
     return {
-        # warm, rounded low end
-        "terrain": TimbreKit("warm-low", cutoff_base=2200, brightness_depth=0.4,
-                             drive_base=0.18, drive_depth=0.25, reverb=0.06),
-        # present, slightly gritty mid — the 'voice' of the agents. Cutoff kept
-        # below the high band so the mid stream stays out of atmosphere's territory.
-        "entity_activity": TimbreKit("present-mid", cutoff_base=3500, brightness_depth=0.4,
-                                     drive_base=0.10, drive_depth=0.35, reverb=0.16),
-        # airy, shimmering, spacious top
-        "atmosphere": TimbreKit("airy-high", cutoff_base=11000, brightness_depth=0.9,
-                                drive_base=0.04, drive_depth=0.10, reverb=0.22,
-                                tremolo_rate=5.5, tremolo_depth=0.16),
+        # ROCK: homogeneous at macro + micro scale -> narrow/tonal, a gentle slow
+        # in-and-out, no fast flicker; emerging from lava -> a viscous residue tail.
+        "terrain": TextureProfile(
+            "rock", cutoff_base=2000, brightness_depth=0.3, bandwidth=0.05,
+            drive_base=0.18, drive_depth=0.20,
+            slow_rate=0.3, slow_depth=0.30, fast_depth=0.0,
+            residue=0.35, residue_decay=0.5, reverb=0.05),
+        # AGENTS: conscious -> described by feeling, kept clean (the sparse unison
+        # motif in the composition carries their character, not DSP texture).
+        "entity_activity": TextureProfile(
+            "agents", cutoff_base=3500, brightness_depth=0.4, bandwidth=0.05,
+            drive_base=0.10, drive_depth=0.30, reverb=0.16),
+        # FIRE: broadband + a fast, chaotic flicker (flames) modulated by a slow
+        # drift (the centre of mass wandering over long periods).
+        "atmosphere": TextureProfile(
+            "fire", cutoff_base=11000, brightness_depth=0.9, bandwidth=0.55,
+            drive_base=0.04, drive_depth=0.10,
+            slow_rate=0.2, slow_depth=0.35, fast_rate=11.0, fast_depth=0.6, chaos=0.85,
+            reverb=0.22),
     }
 
 
@@ -39,7 +47,7 @@ class GenrePreset:
     lead_program: int
     swing: float  # 0..1 swing pushed onto off-beats
     falsetto: bool  # shift sung vocals up an octave
-    timbres: dict[str, TimbreKit] = field(default_factory=_smooth_funk_timbres)
+    timbres: dict[str, TextureProfile] = field(default_factory=_smooth_funk_timbres)
 
 
 SMOOTH_FUNK = GenrePreset(

src/music_making/timbre.py

@@ -1,13 +1,18 @@
-"""Per-stream timbre.
+"""Per-stream timbre as a sonification of physical texture.
 
-Each frequency stream (terrain/low, entity/mid, atmosphere/high) has its own
-spectral character, described by a configurable ``TimbreKit`` and modulated by
-the scene over time:
+The principle (see DESIGN.md): a stream's timbre should describe the *real
+element* as faithfully as possible — materials and processes by their **texture**
+(spectral homogeneity + multi-timescale modulation + the medium they sit in),
+conscious agents by **feeling** (kept clean here; their character is carried by
+the sparse unison motif in the composition).
 
-  drive      <- tension     (saturation / grit)
-  low-pass   <- brightness  (the filter opens as the scene brightens)
-  reverb     -- space (per-kit)
-  tremolo    -- movement (per-kit)
+A ``TextureProfile`` captures:
+  * spectral placement / homogeneity   (narrow tonal  <->  broadband noisy)
+  * multi-timescale amplitude modulation (slow drift  +  fast/chaotic flicker)
+  * a medium/residue tail               (the substance the element sits in)
+  * saturation + space
+
+Scene-modulated: drive follows `tension`, the low-pass opens with `brightness`.
 """
 
 from __future__ import annotations
@@ -22,20 +27,34 @@
 
 
 @dataclass(frozen=True)
-class TimbreKit:
+class TextureProfile:
     name: str
-    cutoff_base: float        # Hz, low-pass baseline
-    brightness_depth: float   # how much `brightness` opens the filter
-    drive_base: float         # static saturation
-    drive_depth: float        # `tension` adds saturation
-    reverb: float             # wet mix 0..1
-    tremolo_rate: float = 0.0
-    tremolo_depth: float = 0.0
-    program: int | None = None  # optional GM program override for this stream
+    # spectral placement / homogeneity
+    cutoff_base: float          # Hz low-pass baseline
+    brightness_depth: float     # how much `brightness` opens the filter
+    bandwidth: float = 0.0      # 0 = homogeneous/tonal, 1 = broadband/noisy
+    # saturation
+    drive_base: float = 0.0
+    drive_depth: float = 0.0    # `tension` adds saturation
+    # multi-timescale amplitude modulation
+    slow_rate: float = 0.0      # Hz, slow drift (e.g. fire's centre-of-mass wander)
+    slow_depth: float = 0.0
+    fast_rate: float = 0.0      # Hz, fast flicker (e.g. flames)
+    fast_depth: float = 0.0
+    chaos: float = 0.0          # 0 = periodic LFO, 1 = noise-driven chaotic flicker
+    # medium / residue (the substance the element emerges from)
+    residue: float = 0.0        # amount of viscous decay tail
+    residue_decay: float = 0.5  # seconds
+    # space
+    reverb: float = 0.0
+    program: int | None = None  # optional GM program override
+
+
+# Backwards-compatible alias (the concept used to be called a "kit").
+TimbreKit = TextureProfile
 
 
 def _layer_env(sb: Storyboard, layer: str, n: int, control: int = 1024) -> np.ndarray:
-    """A control-rate scene-layer envelope upsampled to ``n`` samples."""
     cn = max(2, min(control, n))
     cvals = np.array([sb.layer_at(layer, i / (cn - 1)) for i in range(cn)], dtype=np.float32)
     return np.interp(np.linspace(0, cn - 1, n), np.arange(cn), cvals).astype(np.float32)
@@ -46,6 +65,50 @@ def _saturate(x: np.ndarray, drive_env: np.ndarray) -> np.ndarray:
     return (np.tanh(x * k) / np.tanh(np.maximum(k, 1e-6))).astype(np.float32)
 
 
+def _sine_lfo(n: int, rate: float, sr: int) -> np.ndarray:
+    t = np.arange(n) / sr
+    return (0.5 * (1.0 + np.sin(2 * np.pi * rate * t))).astype(np.float32)
+
+
+def _noise_lfo(n: int, rate: float, sr: int, seed: int) -> np.ndarray:
+    """Chaotic flicker: white noise band-limited to ~`rate`, normalized 0..1."""
+    rng = np.random.default_rng(seed)
+    w = rng.standard_normal(n).astype(np.float32)
+    cutoff = min(0.99, max(rate, 0.5) / (sr / 2))
+    sos = signal.butter(2, cutoff, btype="low", output="sos")
+    f = signal.sosfilt(sos, w)
+    f -= f.min()
+    return (f / (f.max() or 1.0)).astype(np.float32)
+
+
+def _modulate(x: np.ndarray, sr: int, p: TextureProfile) -> np.ndarray:
+    env = np.ones(len(x), dtype=np.float32)
+    if p.slow_depth > 0 and p.slow_rate > 0:
+        env *= 1.0 - p.slow_depth * (1.0 - _sine_lfo(len(x), p.slow_rate, sr))
+    if p.fast_depth > 0 and p.fast_rate > 0:
+        per = _sine_lfo(len(x), p.fast_rate, sr)
+        if p.chaos > 0:
+            fast = p.chaos * _noise_lfo(len(x), p.fast_rate, sr, seed=1234) + (1 - p.chaos) * per
+        else:
+            fast = per
+        env *= 1.0 - p.fast_depth * (1.0 - fast)
+    return (x * env).astype(np.float32)
+
+
+def _broadband(x: np.ndarray, sr: int, p: TextureProfile) -> np.ndarray:
+    """Heterogeneity: add high-frequency crackle that follows the signal's
+    amplitude (e.g. the broadband texture of fire)."""
+    if p.bandwidth <= 0:
+        return x
+    amp_sos = signal.butter(2, 20 / (sr / 2), btype="low", output="sos")
+    amp = signal.sosfilt(amp_sos, np.abs(x)).astype(np.float32)
+    rng = np.random.default_rng(777)
+    noise = rng.standard_normal(len(x)).astype(np.float32)
+    hp = signal.butter(2, 2000 / (sr / 2), btype="high", output="sos")
+    noise = signal.sosfilt(hp, noise).astype(np.float32)
+    return (x + p.bandwidth * 0.5 * noise * amp).astype(np.float32)
+
+
 def _lp_timevarying(x: np.ndarray, sb: Storyboard, cutoff_base: float, depth: float,
                     block: int = 8192) -> np.ndarray:
     n = len(x)
@@ -57,8 +120,7 @@ def _lp_timevarying(x: np.ndarray, sb: Storyboard, cutoff_base: float, depth: fl
     pos = 0
     while pos < n:
         end = min(n, pos + block)
-        t_norm = ((pos + end) / 2.0) / n
-        br = sb.layer_at("brightness", t_norm)
+        br = sb.layer_at("brightness", ((pos + end) / 2.0) / n)
         cutoff = cutoff_base * (0.4 + 1.3 * depth * br) + cutoff_base * 0.2
         cutoff = max(200.0, min(nyq * 0.95, cutoff))
         sos = signal.butter(2, cutoff / nyq, btype="low", output="sos")
@@ -69,6 +131,21 @@ def _lp_timevarying(x: np.ndarray, sb: Storyboard, cutoff_base: float, depth: fl
     return out
 
 
+def _residue(x: np.ndarray, sr: int, p: TextureProfile) -> np.ndarray:
+    """Viscous medium tail: a dark, smeared decay after each hit — e.g. the lava
+    the rock emerges from, persisting for ~`residue_decay` seconds."""
+    if p.residue <= 0:
+        return x
+    length = max(1, int(p.residue_decay * sr))
+    t = np.arange(length) / sr
+    ir = np.exp(-t / (p.residue_decay / 3.0)).astype(np.float32)
+    tail = signal.fftconvolve(x, ir)[: len(x)].astype(np.float32)
+    dark = signal.butter(2, 600 / (sr / 2), btype="low", output="sos")
+    tail = signal.sosfilt(dark, tail).astype(np.float32)
+    tail *= (np.max(np.abs(x)) + 1e-9) / (np.max(np.abs(tail)) + 1e-9)  # match level
+    return ((1.0 - p.residue * 0.5) * x + p.residue * tail).astype(np.float32)
+
+
 _IR: np.ndarray | None = None
 
 
@@ -90,32 +167,26 @@ def _reverb(x: np.ndarray, wet: float) -> np.ndarray:
     return ((1.0 - wet) * x + wet * 0.6 * tail).astype(np.float32)
 
 
-def _tremolo(x: np.ndarray, rate: float, depth: float) -> np.ndarray:
-    if rate <= 0 or depth <= 0:
-        return x
-    t = np.arange(len(x)) / audio.SR
-    lfo = 1.0 - depth * 0.5 * (1.0 + np.sin(2 * np.pi * rate * t))
-    return (x * lfo.astype(np.float32)).astype(np.float32)
-
-
-def apply(samples: np.ndarray, sb: Storyboard, kit: TimbreKit) -> np.ndarray:
+def apply(samples: np.ndarray, sb: Storyboard, p: TextureProfile) -> np.ndarray:
     x = samples.astype(np.float32)
     if len(x) == 0:
         return x
-    drive_env = kit.drive_base + kit.drive_depth * _layer_env(sb, "tension", len(x))
+    drive_env = p.drive_base + p.drive_depth * _layer_env(sb, "tension", len(x))
     x = _saturate(x, drive_env)
-    x = _lp_timevarying(x, sb, kit.cutoff_base, kit.brightness_depth)
-    x = _tremolo(x, kit.tremolo_rate, kit.tremolo_depth)
-    x = _reverb(x, kit.reverb)
+    x = _modulate(x, audio.SR, p)        # slow drift + fast/chaotic flicker
+    x = _broadband(x, audio.SR, p)       # heterogeneity / crackle
+    x = _lp_timevarying(x, sb, p.cutoff_base, p.brightness_depth)
+    x = _residue(x, audio.SR, p)         # viscous medium tail
+    x = _reverb(x, p.reverb)
     peak = float(np.max(np.abs(x))) or 1.0
     if peak > 1.0:
         x = x / peak * 0.99
     return x
 
 
-def render_stem(midi_path: str, wav_path: str, sb: Storyboard, kit: TimbreKit,
+def render_stem(midi_path: str, wav_path: str, sb: Storyboard, profile: TextureProfile,
                 soundfont: str | None = None) -> str:
-    """Render a MIDI stem and stamp the stream's timbre onto it."""
+    """Render a MIDI stem and stamp the stream's texture onto it."""
     audio.render_midi(midi_path, wav_path, soundfont=soundfont)
-    audio.save_wav(wav_path, apply(audio.load_wav(wav_path), sb, kit))
+    audio.save_wav(wav_path, apply(audio.load_wav(wav_path), sb, profile))
     return wav_path

tests/test_timbre.py

@@ -1,8 +1,7 @@
 import numpy as np
 
-from music_making import audio, timbre
-from music_making import from_text
-from music_making.timbre import TimbreKit
+from music_making import audio, from_text, timbre
+from music_making.timbre import TextureProfile
 
 
 def _high_energy(x: np.ndarray) -> float:
@@ -11,21 +10,51 @@ def _high_energy(x: np.ndarray) -> float:
     return float((spec[freqs > 4500] ** 2).sum())
 
 
-def test_lowpass_kit_darkens_signal():
+def _shorttime_rms_std(x: np.ndarray, win: int = 2048) -> float:
+    n = (len(x) // win) * win
+    frames = x[:n].reshape(-1, win)
+    return float(np.std(np.sqrt((frames ** 2).mean(axis=1))))
+
+
+def test_lowpass_profile_darkens_signal():
     sb = from_text("a calm night", seed=0, duration_sec=4.0)
-    rng = np.random.default_rng(0)
-    x = (rng.standard_normal(int(4 * audio.SR)) * 0.2).astype(np.float32)
-    dark = TimbreKit("dark", cutoff_base=800, brightness_depth=0.0,
-                     drive_base=0.0, drive_depth=0.0, reverb=0.0)
-    y = timbre.apply(x, sb, dark)
-    assert _high_energy(y) < _high_energy(x) * 0.5
+    x = (np.random.default_rng(0).standard_normal(int(4 * audio.SR)) * 0.2).astype(np.float32)
+    dark = TextureProfile("dark", cutoff_base=800, brightness_depth=0.0)
+    assert _high_energy(timbre.apply(x, sb, dark)) < _high_energy(x) * 0.5
 
 
 def test_apply_preserves_length_and_bounds():
     sb = from_text("bright fire and heat", seed=0, duration_sec=3.0)
     x = (np.random.default_rng(1).standard_normal(int(3 * audio.SR)) * 0.3).astype(np.float32)
-    kit = TimbreKit("k", cutoff_base=6000, brightness_depth=0.5, drive_base=0.2,
-                    drive_depth=0.3, reverb=0.2, tremolo_rate=5.0, tremolo_depth=0.2)
-    y = timbre.apply(x, sb, kit)
+    fire = TextureProfile("fire", cutoff_base=9000, brightness_depth=0.7, bandwidth=0.5,
+                          drive_base=0.1, drive_depth=0.3, slow_rate=0.2, slow_depth=0.3,
+                          fast_rate=11.0, fast_depth=0.6, chaos=0.85, reverb=0.2)
+    y = timbre.apply(x, sb, fire)
     assert len(y) == len(x)
     assert np.max(np.abs(y)) <= 1.0
+
+
+def test_residue_smears_a_burst_forward():
+    sb = from_text("solid rock", seed=0, duration_sec=2.0)
+    x = np.zeros(int(1.0 * audio.SR), dtype=np.float32)
+    burst = int(0.05 * audio.SR)
+    x[:burst] = np.random.default_rng(2).standard_normal(burst).astype(np.float32) * 0.5
+    late = slice(int(0.2 * audio.SR), int(0.7 * audio.SR))
+
+    plain = TextureProfile("plain", cutoff_base=12000, brightness_depth=0.0)
+    viscous = TextureProfile("viscous", cutoff_base=12000, brightness_depth=0.0,
+                             residue=0.6, residue_decay=0.5)
+    e_plain = float(np.sum(timbre.apply(x, sb, plain)[late] ** 2))
+    e_visc = float(np.sum(timbre.apply(x, sb, viscous)[late] ** 2))
+    assert e_visc > e_plain * 2
+
+
+def test_fast_flicker_adds_amplitude_movement():
+    sb = from_text("bright fire", seed=0, duration_sec=2.0)
+    t = np.arange(int(2.0 * audio.SR)) / audio.SR
+    tone = (0.3 * np.sin(2 * np.pi * 220 * t)).astype(np.float32)
+    still = TextureProfile("still", cutoff_base=12000, brightness_depth=0.0)
+    flicker = TextureProfile("flicker", cutoff_base=12000, brightness_depth=0.0,
+                             fast_rate=11.0, fast_depth=0.8, chaos=0.0)
+    assert _shorttime_rms_std(timbre.apply(tone, sb, flicker)) > \
+        _shorttime_rms_std(timbre.apply(tone, sb, still))