|
| 1 | +# Callable Function Parameters |
| 2 | + |
| 3 | +**Status**: Implemented |
| 4 | + |
| 5 | +**Issue**: [GH-1424](https://github.com/NVIDIA/warp/issues/1424) |
| 6 | + |
| 7 | +## Motivation |
| 8 | + |
| 9 | +Warp users can write higher-order Python helpers that accept a callable and |
| 10 | +apply it to values, but the same pattern did not work inside user-defined |
| 11 | +`@wp.func` code. A function parameter annotated as `Callable` was not matched |
| 12 | +consistently during Warp overload resolution, especially across the two standard |
| 13 | +library import paths: |
| 14 | + |
| 15 | +```python |
| 16 | +from typing import Callable as TypingCallable |
| 17 | +from collections.abc import Callable as AbcCallable |
| 18 | +``` |
| 19 | + |
| 20 | +The aliases are illustrative; Warp treats both origins as the same type-erased |
| 21 | +`Callable` marker. This design implements the user-defined function portion of |
| 22 | +GH-1424 as a first pass: support direct inline calls such as |
| 23 | +`apply(double_it, 3.0)` from a kernel or another `@wp.func`, without depending |
| 24 | +on local function-variable assignment behavior. |
| 25 | + |
| 26 | +## Requirements |
| 27 | + |
| 28 | +| ID | Requirement | Priority | Notes | |
| 29 | +| --- | ----------- | -------- | ----- | |
| 30 | +| R1 | Recognize `typing.Callable` and `collections.abc.Callable` as callable annotations. | Must | Includes bare and parameterized forms. | |
| 31 | +| R2 | Allow user-defined `@wp.func` objects to match `Callable` parameters. | Must | Callable values are type-erased. | |
| 32 | +| R3 | Preserve the existing `"c"` type code for callable annotations. | Must | Used by module hashing and overload keys. | |
| 33 | +| R4 | Keep annotation recognition compatible with Python 3.10 through 3.14. | Must | Avoid private `typing` internals. | |
| 34 | +| R5 | Include callable argument and default targets in module hashes and dependencies. | Must | Prevent stale compiled modules when callable targets change. | |
| 35 | +| R6 | Reject unsupported callable specializations explicitly. | Must | Built-ins are deferred; custom grad/replay functions are first-pass non-goals. | |
| 36 | + |
| 37 | +**Non-goals**: |
| 38 | + |
| 39 | +- Validate parameterized callable signatures such as |
| 40 | + `Callable[[float], float]`. |
| 41 | +- Implement runtime dispatch for arbitrary Python callables. |
| 42 | +- Support kernel-local function variable assignment. |
| 43 | +- Support built-in Warp functions such as `wp.sin` or `wp.add` as callable |
| 44 | + arguments in this first pass. They are rejected until built-in callable |
| 45 | + identity can participate safely in specialization hashing and dependency |
| 46 | + tracking. |
| 47 | +- Support callable-specialized functions that have custom gradient or replay |
| 48 | + functions. |
| 49 | + |
| 50 | +Parameterized callable annotations are recognized as callable markers, but |
| 51 | +their argument and return types remain unchecked. |
| 52 | + |
| 53 | +## Design |
| 54 | + |
| 55 | +### Approach |
| 56 | + |
| 57 | +The implementation treats `Callable` parameters as compile-time function |
| 58 | +references. A callable parameter never becomes a runtime C++ function pointer or |
| 59 | +kernel parameter. Instead, code generation specializes the user-defined function |
| 60 | +for the concrete user-defined Warp function passed at each call site. |
| 61 | + |
| 62 | +This means: |
| 63 | + |
| 64 | +- `apply(double_it, x)` and `apply(triple_it, x)` produce separate specialized |
| 65 | + native functions. |
| 66 | +- Callable parameters are bound into the specialized function's codegen symbol |
| 67 | + table. |
| 68 | +- Callable parameters are omitted from the emitted forward and reverse C++ |
| 69 | + function signatures. |
| 70 | +- Runtime calls pass only the non-callable arguments. |
| 71 | + |
| 72 | +The callable annotation predicate lives in `warp/_src/types.py` as |
| 73 | +`is_callable_annotation(annotation)`. It returns true for: |
| 74 | + |
| 75 | +- bare `typing.Callable`, |
| 76 | +- bare `collections.abc.Callable`, |
| 77 | +- `typing.Callable[...]`, |
| 78 | +- `collections.abc.Callable[...]`. |
| 79 | + |
| 80 | +The helper uses `typing.get_origin()` plus the canonical |
| 81 | +`collections.abc.Callable` object, which covers the supported Python versions |
| 82 | +without relying on private implementation details. |
| 83 | + |
| 84 | +### Alternatives Considered |
| 85 | + |
| 86 | +One option was to normalize annotations globally during `@wp.func` registration. |
| 87 | +That would make all `Callable` forms identical up front, but it would also |
| 88 | +change the raw annotations stored on every function and could affect unrelated |
| 89 | +signature handling. A small predicate keeps the behavior localized. |
| 90 | + |
| 91 | +Another option was to support built-in Warp functions as callable values. |
| 92 | +GH-1424 includes built-ins, but this first pass narrows support to user-defined |
| 93 | +targets. Built-ins require hashing and dependency behavior for callable |
| 94 | +identity, so they are rejected explicitly to avoid partial support that could |
| 95 | +reuse stale cached modules. |
| 96 | + |
| 97 | +### Key Implementation Details |
| 98 | + |
| 99 | +`get_type_code()` returns `"c"` for callable annotations before the generic |
| 100 | +type branches. This keeps module hash type-code behavior stable for bare and |
| 101 | +parameterized `Callable` forms. |
| 102 | + |
| 103 | +`func_match_args()` treats a `warp._src.context.Function` value as compatible |
| 104 | +with any callable annotation. Non-function values continue to fail normal |
| 105 | +overload resolution. |
| 106 | + |
| 107 | +During `Adjoint.add_call()`, default arguments are applied first. Callable |
| 108 | +arguments and callable defaults are then collected. Built-in function values are |
| 109 | +rejected, and user-defined function values trigger creation of a specialized |
| 110 | +`Function` clone. The clone receives: |
| 111 | + |
| 112 | +- a hash-suffixed native function name, |
| 113 | +- a fresh `Adjoint` with the original annotations, including the return |
| 114 | + annotation, |
| 115 | +- `callable_arg_values` used to bind callable parameter names during codegen. |
| 116 | + |
| 117 | +Module hashing and dependency discovery both inspect callable arguments and |
| 118 | +callable defaults. User-defined callable targets are included in the referenced |
| 119 | +function set for hashes and in module references/dependents for invalidation. |
| 120 | + |
| 121 | +Callable-specialized functions with custom gradients or custom replay functions |
| 122 | +are rejected. Their custom functions are tied to the unspecialized native |
| 123 | +function signature, while callable specialization removes callable runtime |
| 124 | +parameters from the emitted signature. |
| 125 | + |
| 126 | +## Testing Strategy |
| 127 | + |
| 128 | +`warp/tests/test_func.py` covers: |
| 129 | + |
| 130 | +- bare `typing.Callable` and `collections.abc.Callable` runtime calls, |
| 131 | +- parameterized `Callable[[float], float]` runtime calls, |
| 132 | +- generic user functions that combine `Callable` with `Any`, |
| 133 | +- default callable arguments, |
| 134 | +- keyword callable arguments, |
| 135 | +- nested user-defined function calls, |
| 136 | +- callable targets affecting module hashes, |
| 137 | +- callable targets updating cross-module dependents, |
| 138 | +- return annotation preservation on specialized functions, |
| 139 | +- explicit rejection for built-in callable targets, |
| 140 | +- explicit rejection for callable-specialized functions with custom grad or |
| 141 | + replay functions. |
| 142 | + |
| 143 | +Local verification should include the focused `TestFunc` suite and pre-commit |
| 144 | +over the changed files. CI provides the full Python 3.10 through 3.14 matrix. |
0 commit comments