Skip to content

Age specialization: per-age regime functions/constraints and age-dependent continuous-state grids#398

Open
hmgaudecker wants to merge 44 commits into
mainfrom
feat/age-specialized
Open

Age specialization: per-age regime functions/constraints and age-dependent continuous-state grids#398
hmgaudecker wants to merge 44 commits into
mainfrom
feat/age-specialized

Conversation

@hmgaudecker

@hmgaudecker hmgaudecker commented Jul 6, 2026

Copy link
Copy Markdown
Member

Summary

Adds age specialization to pylcm: one regime definition whose functions,
constraints, or continuous-state grids
vary with age. Two user-facing markers
share a common base (_AgeSpecialized), each wrapping a build(age) factory
plus a signature(age) function — cells with equal signatures share one built
object, so compiled programs are deduplicated across ages.

AgeSpecializedFunction — per-age functions and constraints

  • Swaps the implementation of a regime function or constraint by age. The
    motivating use case is calendar-policy tracking in cohort models: a GETTSIM
    tax function whose policy environment moves with the agent's age, rebuilt per
    (cohort, age) cell without duplicating regimes.
  • Solution: Q_and_F resolves the specialized function per age during backward
    induction; representative-age resolution in _process_regime_core keeps
    templates and the params template consistent (including
    Phased(AgeSpecializedFunction)).
  • Simulation: next_state is period-indexed so per-age transitions simulate
    correctly; to_dataframe(additional_targets=...) rejects age-specialized
    functions it cannot represent at a single age.
  • Guardrails (each rejected at model construction with a targeted error):
    deterministic state transitions that are age-specialized (bare or via
    MarkovTransition(...), including per-target and Phased forms); regime
    transitions that are age-specialized, wrapped, or whose dependency graph
    reads an age-specialized helper; terminal-regime markers.
  • Docs: user-guide page docs/user_guide/age_specialized.md.

AgeSpecializedGrid — age-dependent continuous-state grids

  • A continuous state's grid bounds may move with age while n_points (and
    the grid class) stay fixed — shape-invariant, so V-array shapes, the rolling
    continuation template, and kernel-dedup topology are untouched. The
    motivating use case is an asset state with an age-dependent borrowing floor
    a̲(age) (Guvenen & Smith 2014, and any life-cycle model with an age-varying
    constraint): on a single fixed grid, cells below a tighter age's floor are
    infeasible, and their -inf continuation poisons the value function through
    interpolation (0 · (−inf) = NaN, propagated backward).
  • Solution: the current-period state axis is runtime data — each period's node
    values are substituted into the shared compiled kernel without retracing;
    the continuation V_{t+1} is interpolated on the target regime's
    period-(t+1) grid
    , with the Q_and_F group signature folding in the
    continuation-grid identity so periods never false-share a program.
  • Simulation: the forward pass interpolates the continuation on the same
    per-period grids as the solve.
  • Validation: build(age) must return the same grid class and n_points at
    every age (RegimeInitializationError otherwise); markers are accepted only
    as state grids.
  • Design notes: DESIGN-age-dependent-grids.md.

Hardening

The function-specialization surface went through several rounds of external
adversarial review (ChatGPT-Pro code audit) driven by the downstream rmsm
project; every hole found (transitive regime-transition reads,
MarkovTransition-wrapped markers, params-template blind spots,
representative-age leaks into additional_targets) is closed with a
regression test. Downstream, an annual lifecycle model with per-(cohort, age)
GETTSIM policy environments solves and simulates on this branch, and its
estimation runs on GPU. The grid surface is exercised end-to-end by the
Guvenen–Smith (2014) replication in lcm-replications, where the age-tracking
asset floor eliminates the -inf/NaN poisoning of the fixed-grid solve.

Test plan

  • Full suite: 1229 passed, 48 skipped (pixi run -e tests-cpu tests);
    ruff check and ty both fully clean.
  • Functions: tests/regime_building/test_age_specialization.py
    (validation surface, sharing by signature),
    tests/solution/test_age_specialized_solve.py (solve/simulate
    equivalence against a hand-built per-age model).
  • Grids: tests/solution/test_age_specialized_grid_solve.py — an
    age-invariant AgeSpecializedGrid reproduces the plain fixed-grid solve
    bit-for-bit; an age-tracking floor solves with no NaN poisoning;
    V monotone in wealth; simulation yields finite, positive consumption;
    non-shape-invariant grids are rejected.
  • Equivalence gates: constant AgeSpecializedFunction signature
    reproduces the unspecialized solution exactly; models without any
    age-varying grid build byte-identically to the pre-feature path.

🤖 Generated with Claude Code

https://claude.ai/code/session_016BLLuvQwD2bdXgaTcJWWUm

hmgaudecker and others added 30 commits June 21, 2026 19:09
…river)

Two tests on the 4-CPU distributed seam:
- test_distributed_solve_matches_single_device_per_type: the F6 correctness
  contract — a sharded solve must equal the single-device solve per type slice.
  Green now; the co-map fix must preserve it (catches a wrong device-local index).
- test_distributed_solve_kernel_does_not_all_gather_continuation_v: the red
  driver — the backward-induction kernel currently all-gathers the continuation
  V across the type shard; it must read only its device-local slice.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
vmap_1d gains co_mapped_in_axes: an optional per-argument in_axes override so
a pytree argument's leading axis can be mapped in lockstep with the mapped
variables. The backward-induction co-map uses it to slice each
next_regime_to_V_arr leaf to the device-local type, so the continuation-V
interpolation reads only its own shard.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Fixed, distributed states (e.g. a permanent type sharded one block per device)
never transition, so a regime's continuation value depends only on its own slice
of the next-period V-array. The grid-search solve kernel now co-maps each such
state with the matching axis of every next_regime_to_V_arr leaf that carries it:
an outer vmap peels the leading axis off both the state grid and the continuation
V, so the interpolation reads only the device-local slice and XLA inserts no
all-gather of the full V-array onto every device.

- max_Q_over_a splits the co-mapped (leading) states from the inner productmap and
  wraps it in per-state co-map vmaps; the V-interpolator drops those coordinates and
  Q_and_F omits the sliced next-states.
- processing detects the co-mappable states (distributed and identity-transition)
  and builds per-state, per-leaf in_axes so a target regime that prunes the state
  keeps its full leaf.
- Simulation is unchanged: it keeps the full continuation V (subjects are not
  type-aligned across devices), so the co-map applies to the solve path only.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…seam (#385)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ectations

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Add 'age' entry to initial_conditions in epstein_zin.md Run section
- Document certainty_equivalent parameter in get_model docstring
- Add docstrings to _power_transform and _power_inverse helpers

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…model

Rename `TransformedExpectation` to `QuasiArithmeticMean` and
`PowerCertaintyEquivalent` to `PowerMean`; the `certainty_equivalent`
field, params pseudo-function, and `risk_aversion` parameter are unchanged.

Move the engine-side implementation (`CE_VALUE_ARG`, the power transform
pair, and `resolve_certainty_equivalent`) into `_lcm/certainty_equivalent.py`
so the public module is a thin, deep-module namespace and the solver seam in
`Q_and_F.py` only imports the resolver.

Collapse the toy `tests/test_models/epstein_zin_health.py` and the example
into a single parametrized `lcm_examples.epstein_zin` (`EZRegimeId`,
`get_model` with a required `certainty_equivalent` and grid-size knobs whose
defaults reproduce the toy numerically). The numpy pinning references pass
unchanged.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Replace `docs/examples/epstein_zin.md` with `docs/examples/epstein_zin.ipynb`:
the same recursion, mapping, and pitfalls as markdown cells, plus code cells
that solve and simulate a 20-period model for two risk-aversion values and
render a plotly figure of mean wealth by age (grey vs accent, direct labels).
Update the toctree in `myst.yml` and the link in `examples/index.md`.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…he example

- lcm.aggregators exposes the two standard Koopmans aggregators; the
  default H is now the public H_linear, and H_epstein_zin is
  parametrized by the intertemporal elasticity of substitution
  (curvature rho = 1 - 1/psi computed inside, psi = 1 as the
  Cobb-Douglas limit).
- PowerMean handles risk_aversion = 1 as the geometric-mean (log)
  limit exp(E[log V']) instead of rejecting it; the numpy reference
  and pinning tests cover it.
- Example notebook: merged the redundant positivity pitfalls, added an
  expected-utility baseline to the wealth figure, log_level='off'.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… trace

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The example model gains three parameters, all defaulting to the previous
behavior: income (above the consumption floor, saving becomes possible),
health_cost (an out-of-pocket expense while in bad health - uninsurable
expense risk in the spirit of the Atal et al. medical spending), and
bequest_scale (prices the bequest in consumption-equivalent units;
H_epstein_zin is a weighted power mean, so the alive value sits at the
scale of per-period consumption and an unscaled sqrt(wealth) bequest
makes death the good branch of the certainty equivalent). next_wealth
clips to the wealth grid so none of the knobs can push states off-grid.

The docs page is rewritten around the fixed model:

- Atal, Fang, Karlsson & Ziebarth (2025) is now characterized correctly:
  their baseline is time-separable CARA expected utility; their
  robustness specification has exactly the CES-aggregator-around-a-
  certainty-equivalent structure used here, with a CARA certainty
  equivalent (expressible as a QuasiArithmeticMean).
- The pylcm-mapping section reflects the shipped H_linear/H_epstein_zin
  and explains that per-period utility must live in consumption units
  because H is a power mean.
- A new pitfall documents the bequest-scaling trap.
- The figure simulates 1,000 subjects with saving, health costs, and a
  gentler mortality hazard, so survivor counts stay meaningful at every
  plotted age, and the closing text explains the economics: both types
  dissave early out of impatience, the risk-averse type holds and
  rebuilds a larger precautionary buffer, and the ranking flips when the
  bad-health spell becomes too expensive to self-insure.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…IES/RA sweep

The intro now describes the original Atal, Fang, Karlsson & Ziebarth (2025)
model accurately: an annual Yaari life-cycle savings problem over ages 25-94,
a seven-category health Markov chain driving expenditure and mortality, and
the guaranteed-renewable GLTHI vs short-term insurance contracts. Their
baseline is time-separable expected utility (CARA gamma=4e-4, CRRA sigma=4
robustness, delta=0.966); the headline is that GLTHI reaches ~96% of
first-best welfare, robust to disentangling risk aversion and the IES
(sec. VI.D.1, within 0.7%). A 'what this keeps and drops' paragraph is
explicit that the equilibrium contract/premium layer is out of scope, so the
paper's headline welfare gap is not something this consumer-block example
reproduces.

A new closing section makes the disentangling concrete: a 2D sweep of the
welfare cost of the uninsurable health-expense risk over a log-2 grid of risk
aversion and the IES, with expected utility marked as the exact anti-diagonal
(IES = 1/gamma). The cost is a risk premium — it roughly triples down the
risk-aversion axis and barely moves along the IES axis — so an EU model,
confined to the anti-diagonal, confounds the two. This is exactly the degree
of freedom the certainty_equivalent seam adds and the lever the paper's
robustness section pulls.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The example model enters at age 25 rather than 60, matching the annual
horizon of Atal et al. instead of a retirement-only slice. Only the entry
age changes and periods stay annual, so the discount factor needs no
recompounding.

The docs page runs the full 25-to-85 lifecycle: 2,000 subjects under a
Gompertz-like mortality hazard (low when young, rising with age), which
keeps a well-populated surviving cohort through midlife so both figures read
cleanly. The risk-averse agent holds a persistently larger precautionary
buffer, and the welfare cost of the health-expense risk roughly doubles down
the risk-aversion axis while barely moving along the IES axis. Prose and
quoted numbers track the new lifecycle. The notebook executes in about ten
seconds on CPU, within the Read-the-Docs build budget.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The intro markdown cell had lost its newlines and collapsed into a single
line, so its headings and paragraphs ran together on the rendered page.
Rebuild it with one array element per line. Also switch the recursion's
display equation from a ```{math}``` directive — which this project's MyST
does not process inside notebook cells, leaking raw LaTeX — to the $$ form
used by every other notebook, so it typesets.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A subject that enters a terminal regime is carried with frozen state
through every later period the regime is active. to_dataframe now emits
only the entry row per subject (terminal_rows="first", the default);
terminal_rows="all" keeps the absorbing representation.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The stored regression frames capture the absorbing representation, so the
comparison must request it explicitly rather than inherit the collapsed
default. Only the mortality model exposed this on CI — its subjects die
before the horizon under float32 draws — but all four shape-pinning tests
get the explicit knob.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
hmgaudecker and others added 4 commits July 3, 2026 11:59
JAX writes an executable to the persistent compilation cache only when its
compile time exceeds jax_persistent_cache_min_compile_time_secs (default:
1 second). A pylcm model compiles as many small per-regime/per-period
programs, most of which fall under that threshold, so the cache stayed
empty and every fresh process recompiled the whole model — including the
AOT-compiled solve and simulate programs, which do consult the cache once
it is populated.

Importing lcm zeroes the threshold via jax.config, so it takes effect
regardless of whether jax was imported first; a user-set
JAX_PERSISTENT_CACHE_MIN_COMPILE_TIME_SECS environment variable wins.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Reject AgeSpecialized as a deterministic state-transition value and
  anywhere in terminal regimes (policy-dependent laws of motion use a
  plain transition reading an AgeSpecialized `functions` entry).
- Reject additional_targets that read AgeSpecialized functions; record
  the specialized names on SimulationPhase.
- Resolve AgeSpecialized inside Phased(solve=..., simulate=...) in the
  params template.
- Resolve markers at a representative age for the used-variable check
  and the published function sets.
- Document the two contracts and the supported scope in
  docs/user_guide/age_specialized.md (added to the toc).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… functions

Regime-transition probabilities are built once, not per period, so a
policy-specialized value flowing into next_regime — directly or through
plain helper functions — would silently reuse one age's policy closure
across all periods. Regime construction now walks the transition's
parameter names transitively through the functions mapping (unwrapping
Phased sides, per-target dicts, and MarkovTransition) and raises with a
routing hint. Docstring and user guide state the full guarded surface.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The regime-transition guard covered bare AgeSpecialized nodes and
dependency-graph reads through plain helper functions, but a
MarkovTransition whose wrapped function is itself AgeSpecialized slipped
through to a later failure. The scope check now rejects it at Regime
construction, mirroring the state-transition check, across coarse,
Phased, and per-target forms; docstring and user guide state the full
guarded surface.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@review-notebook-app

Copy link
Copy Markdown

Check out this pull request on  ReviewNB

See visual diffs & provide feedback on Jupyter Notebooks.


Powered by ReviewNB

@read-the-docs-community

read-the-docs-community Bot commented Jul 6, 2026

Copy link
Copy Markdown

hmgaudecker and others added 10 commits July 7, 2026 09:30
…se + AgeSpecializedGrid marker + resolver/validator

- _AgeSpecialized base class shared by AgeSpecializedFunction (functions) and
  the new AgeSpecializedGrid (age-varying continuous-state grid, shape-invariant).
- age_specialization.py: grid resolution helpers (resolve_state_grids,
  state_grids_signature, has_age_specialized_grid) and the shape-invariance
  validator (same class + n_points at every age).
- Word-boundary rename across engine, processing, validation, tests, docs.
- Foundational only: engine wiring (per-period axis + continuation interp) still TODO.
…polation)

- Accept AgeSpecializedGrid in Regime.states (typing + construction validators +
  phase splitter treat it as a both-phase continuous grid).
- process_regimes: representative-resolve markers for all age-invariant machinery;
  build per-period VInterpolationInfo (continuation on period t+1's grid) and
  per-period state axes (V_t tabulated on period t's grid).
- _build_Q_and_F_per_period: group signature folds in the continuation grid
  identity at t+1; each group's Q_and_F uses period t+1's interpolation info.
- SolutionPhase.period_state_axes + backward_induction override the base axis with
  period-t nodes (same shape => shared kernel, no retrace).
- Age-invariant AgeSpecializedGrid reproduces the plain fixed-grid solve bit-for-bit
  (verified). Simulate path still TODO.
…tion)

Thread period_to_regime_v_interp into _build_simulation_phase so the forward
simulation's Q_and_F interpolates V_{t+1} on period t+1's grid, matching solve.
Simulation states are per-subject particles (no grid axis), so only the
continuation needs per-period wiring. 165 simulate + regime-building tests green;
age-varying model simulates with finite, positive consumption.
tests/solution/test_age_specialized_grid_solve.py:
- age-invariant grid reproduces the plain fixed-grid solve bit-for-bit;
- age-tracking floor solves with no NaN poisoning (the feature's purpose);
- V nondecreasing in wealth (economic sanity);
- forward simulation gives finite, positive consumption;
- non-shape-invariant grid (varying n_points) is rejected at build.
_state_grids_with_carried_domains passes AgeSpecializedGrid through unchanged; it
is a continuous state, so every consumer (all filter for DiscreteGrid) skips it.
Completes the marker's acceptance across the simulate->to_dataframe path.
- Wrap E501 long lines (docstrings/comments lengthened by the AgeSpecialized ->
  AgeSpecializedFunction rename); ruff-format the touched files.
- Fix ty diagnostics introduced by the feature: type _states_for_period's
  state_action_space as StateActionSpace; getattr-based _grid_identity; runtime
  import of AgeGrid for validate_age_specialized_grids (a TYPE_CHECKING-only import
  broke beartype's runtime forward-ref resolution); drop unused ty:ignore comments.
- Extract _states_for_period helper (fixes PLC0206 + PLR0915); PERF401 comprehension.

ruff check clean; full tests-cpu suite 1229 passed / 0 failed; remaining 7 ty
diagnostics are pre-existing on feat/age-specialized (canonicalize.py, mock_regime.py).
- canonicalize: _desugar_identity accepts AgeSpecializedGrid (branches only on
  DiscreteGrid, so a marker correctly yields ContinuousState).
- processing: narrow carried_grids with isinstance(grid, Grid) — a runtime no-op
  (carried states are Phased(simulate=Grid) by the phase grammar) that fixes the
  union flowing into build_regime_transition_probs_functions.
- tests: MockRegime.functions as Mapping[str, object] (documented loose typing,
  covariant); policy_bonus helpers typed UserFunction; drop a stale ty:ignore.

ty: All checks passed (was 20 diagnostics before this series). ruff clean.
Full tests-cpu suite: 1229 passed / 0 failed.
@hmgaudecker hmgaudecker changed the title Add AgeSpecialized: per-age specialization of regime functions and constraints Age specialization: per-age regime functions/constraints and age-dependent continuous-state grids Jul 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant