pyo3-stub-gen supports writing type information directly in Python stub syntax (.pyi format) instead of relying solely on automatic Rust-to-Python type translation. This feature is essential for:
- Complex Python types that don't map cleanly from Rust (e.g.,
collections.abc.Callable,typing.Protocol) - Function overloads (
@overloaddecorator support) - Type overrides when automatic translation is insufficient
- Providing more Pythonic type annotations
While automatic type translation works for most cases, some Python type patterns cannot be represented in Rust:
# Python stub with Callable - no direct Rust equivalent
def process(callback: collections.abc.Callable[[str], int]) -> int: ...
# Function overloads - same name, different signatures
@overload
def get_value(key: str) -> str: ...
@overload
def get_value(key: int) -> int: ...Python stub syntax support allows developers to specify these types directly in familiar Python notation.
Use the python parameter in #[gen_stub_pyfunction] to specify the complete function signature in Python stub syntax:
use pyo3::prelude::*;
use pyo3_stub_gen::derive::*;
#[gen_stub_pyfunction(python = r#"
import collections.abc
import typing
def fn_with_callback(
callback: collections.abc.Callable[[str], typing.Any]
) -> collections.abc.Callable[[str], typing.Any]:
"""Example using python parameter."""
"#)]
#[pyfunction]
pub fn fn_with_callback<'a>(callback: Bound<'a, PyAny>) -> PyResult<Bound<'a, PyAny>> {
callback.call1(("Hello!",))?;
Ok(callback)
}Generated stub:
import collections.abc
import typing
def fn_with_callback(
callback: collections.abc.Callable[[str], typing.Any]
) -> collections.abc.Callable[[str], typing.Any]:
"""Example using python parameter."""Features:
- ✅ Complete control over the generated signature
- ✅ Supports complex types like
Callable,Protocol, generics - ✅ Allows custom docstrings in the stub
- ✅ Import statements are automatically extracted and placed at module level
When to use:
- Need to specify complex Python types
- Want complete control over the stub signature
- Single function or method needs override
Use gen_function_from_python! inside submit! block to add additional type signatures for the same function:
use pyo3::prelude::*;
use pyo3_stub_gen::{derive::*, inventory::submit};
// Rust implementation (also generates one stub signature automatically)
#[gen_stub_pyfunction]
#[pyfunction]
pub fn overload_example(x: f64) -> f64 {
x + 1.0
}
// Additional overload signature
submit! {
gen_function_from_python! {
r#"
def overload_example(x: int) -> int:
"""Overload for integer input"""
"#
}
}Generated stub:
from typing import overload
@overload
def overload_example(x: float) -> float: ...
@overload
def overload_example(x: int) -> int:
"""Overload for integer input"""Features:
- ✅ Ideal for function overloads (
@overloaddecorator) - ✅ Keeps type definitions separate from implementation
- ✅ Allows multiple signatures for the same function name
When to use:
- Need function overloads (multiple type signatures)
- Want to supplement auto-generated signatures
- Separate type definitions from implementation
Use gen_methods_from_python! to define additional method signatures for a class:
use pyo3::prelude::*;
use pyo3_stub_gen::{derive::*, inventory::submit};
#[gen_stub_pyclass]
#[pyclass]
pub struct Calculator {}
#[gen_stub_pymethods]
#[pymethods]
impl Calculator {
fn add(&self, x: f64) -> f64 {
x + 1.0
}
}
// Additional overload for integer type
submit! {
gen_methods_from_python! {
r#"
class Calculator:
def add(self, x: int) -> int:
"""Add operation for integers"""
"#
}
}Generated stub:
from typing import overload
class Calculator:
@overload
def add(self, x: float) -> float: ...
@overload
def add(self, x: int) -> int:
"""Add operation for integers"""Features:
- ✅ Supports method overloads within classes
- ✅ Can define multiple methods in a single macro call
- ✅ Integrates with auto-generated class definitions
When to use:
- Need method overloads in classes
- Want to add alternative signatures for existing methods
- Supplement auto-generated class stubs
The #[gen_stub_pyfunction], #[gen_stub_pyclass], and #[gen_stub_pymethods] macros automatically generate submit! blocks internally to register type information with the inventory crate:
// You write:
#[gen_stub_pyfunction]
#[pyfunction]
pub fn my_func(x: i32) -> i32 { x + 1 }
// Macro generates (simplified):
#[pyfunction]
pub fn my_func(x: i32) -> i32 { x + 1 }
inventory::submit! {
PyFunctionInfo {
name: "my_func",
parameters: /* ... */,
return_type: /* ... */,
/* ... */
}
}You can add additional submit! blocks to register alternative type signatures:
// First signature (auto-generated from #[gen_stub_pyfunction])
#[gen_stub_pyfunction]
#[pyfunction]
pub fn process(x: f64) -> f64 { x }
// Second signature (manual submit!)
submit! {
gen_function_from_python! {
r#"
def process(x: int) -> int: ...
"#
}
}Result: Two PyFunctionInfo entries are registered for "process", causing the stub generator to interpret them as overloads and generate @overload decorators.
The stub generator detects overloads by name:
// During stub generation:
if function_signatures.len() > 1 {
// Multiple signatures for same name → generate @overload
for signature in &function_signatures[..function_signatures.len() - 1] {
write!("@overload\n{}", signature);
}
// Last signature without @overload (implementation signature)
write!("{}", function_signatures.last());
}Python stub syntax is parsed at compile-time using the rustpython-parser crate:
use rustpython_parser::{parse_program, ast};
let python_code = r#"
def my_function(x: int, y: str = 'default') -> bool:
"""Function docstring."""
"#;
// Parse Python code to AST
let module = parse_program(python_code, "<embedded>")
.map_err(|e| syn::Error::new(span, format!("Failed to parse Python: {}", e)))?;
// Extract function definition
for stmt in &module.statements {
if let ast::Stmt::FunctionDef(func_def) = stmt {
// Extract name, parameters, return type, docstring
process_function(func_def)?;
}
}The parser extracts key information from Python AST:
Function Name:
let name = func_def.name.to_string();Parameters:
for arg in &func_def.args.posonlyargs {
// Positional-only parameters
}
for arg in &func_def.args.args {
// Regular parameters
}
for arg in &func_def.args.kwonlyargs {
// Keyword-only parameters
}Parameter Types:
fn extract_type_annotation(annotation: &ast::Expr) -> TypeInfo {
match annotation {
ast::Expr::Name(name) => {
// Simple type: int, str, bool
TypeInfo::builtin(&name.id)
}
ast::Expr::Subscript(subscript) => {
// Generic type: list[int], dict[str, int]
TypeInfo::with_generic(/* ... */)
}
ast::Expr::BinOp(binop) if binop.op == BitOr => {
// Union type: int | str
TypeInfo::union(/* ... */)
}
// ... other patterns
}
}Default Values:
if let Some(default_expr) = &arg.default {
let default_str = python_ast_to_python_string(default_expr)?;
// Store as DefaultExpr::Python(string)
}See Default Value for Function Arguments for detailed default value handling.
Return Type:
if let Some(returns) = &func_def.returns {
extract_type_annotation(returns)
} else {
TypeInfo::none() // No annotation → None
}Docstring:
fn extract_docstring(body: &[ast::Stmt]) -> String {
if let Some(ast::Stmt::Expr(expr)) = body.first() {
if let ast::Expr::Constant(constant) = &expr.value {
if let ast::Constant::Str(s) = &constant.value {
return s.clone();
}
}
}
String::new()
}Import statements in Python stub syntax are automatically extracted and placed at module level:
#[gen_stub_pyfunction(python = r#"
import collections.abc
import typing
from datetime import datetime
def my_func(x: collections.abc.Callable[[int], str]) -> datetime:
"""Example with imports."""
"#)]Generated stub:
import collections.abc
import typing
from datetime import datetime
def my_func(x: collections.abc.Callable[[int], str]) -> datetime:
"""Example with imports."""The stub generator deduplicates imports across all functions/classes:
// Multiple functions use typing.Callable
// → Only one "import typing" in generated stubCommon imports are added automatically when needed:
import typing- When usingtyping.Any,typing.Sequence, etc.import collections.abc- When usingcollections.abc.Callable, etc.import numpy- When using NumPy types (withnumpyfeature)
The pyo3_stub_gen.RustType["TypeName"] marker allows referencing Rust types within Python stub syntax:
use pyo3::prelude::*;
use pyo3_stub_gen::{derive::*, inventory::submit};
submit! {
gen_function_from_python! {
r#"
def sum_list(values: pyo3_stub_gen.RustType["Vec<i32>"]) -> pyo3_stub_gen.RustType["i32"]:
"""Sum a list of integers"""
"#
}
}During Python stub parsing, RustType["TypeName"] is detected and replaced with the Rust type's PyStubType implementation:
fn extract_type_annotation(expr: &ast::Expr) -> TypeInfo {
match expr {
ast::Expr::Subscript(subscript) => {
if is_rust_type_marker(subscript) {
// Extract "Vec<i32>" from RustType["Vec<i32>"]
let rust_type_name = extract_rust_type_name(subscript)?;
// Parse as Rust type
let rust_type: syn::Type = syn::parse_str(&rust_type_name)?;
// Use PyStubType trait implementation
return TypeInfo::from_rust_type(&rust_type);
}
// ... other subscript handling
}
// ...
}
}The marker is expanded according to the context:
For function arguments (input types):
RustType["Vec<i32>"]
→ Vec::<i32>::type_input()
→ TypeInfo::with_generic("typing.Sequence", vec![TypeInfo::builtin("int")])
→ typing.Sequence[int]For return types (output types):
RustType["Vec<i32>"]
→ Vec::<i32>::type_output()
→ TypeInfo::with_generic("list", vec![TypeInfo::builtin("int")])
→ list[int]Generic Types:
submit! {
gen_function_from_python! {
r#"
def process(
data: pyo3_stub_gen.RustType["HashMap<String, Vec<i32>>"]
) -> pyo3_stub_gen.RustType["Vec<String>"]:
"""Process data."""
"#
}
}Custom Types:
#[gen_stub_pyclass]
#[pyclass]
struct MyClass;
submit! {
gen_function_from_python! {
r#"
def create() -> pyo3_stub_gen.RustType["MyClass"]:
"""Create MyClass instance."""
"#
}
}Ensuring Type Consistency:
// Ensures Python stub matches Rust type exactly
submit! {
gen_function_from_python! {
r#"
def get_config() -> pyo3_stub_gen.RustType["Arc<Config>"]:
"""Get shared config."""
"#
}
}When you want complete control over method signatures without using proc-macros, you can submit all method information manually using submit! and gen_methods_from_python!:
use pyo3::prelude::*;
use pyo3_stub_gen::{derive::*, inventory::submit};
/// Demonstrates manual submission of class methods using the `submit!` macro
#[gen_stub_pyclass] // Use proc-macro for submitting class info
#[pyclass]
pub struct ManualSubmit {}
// No #[gen_stub_pymethods]
// i.e., the following methods will not appear in the stub unless we manually submit them
#[pymethods]
impl ManualSubmit {
#[new]
fn new() -> Self {
ManualSubmit {}
}
fn increment(&self, x: f64) -> f64 {
x + 1.0
}
// Returns the input object as is
fn echo<'arg>(&self, obj: Bound<'arg, PyAny>) -> Bound<'arg, PyAny> {
obj
}
}
// Manually submit method info for the `ManualSubmit` class.
submit! {
// Generator macro to create `pyo3_stub_gen::PyMethodsInfo` from a Python code snippet
gen_methods_from_python! {
r#"
# The class name must match the Rust struct name.
class ManualSubmit:
def __new__(cls) -> ManualSubmit:
"""Constructor for ManualSubmit class"""
...
def increment(self, x: float) -> float:
"""Add 1.0 to the input float"""
...
# Using manual submission, we can write @overload decorators for the `echo` method.
# Since Python's overload resolution depends on the order of definitions,
# we write the more specific type (int) first.
@overload
def echo(self, obj: int) -> int:
"""If the input is an int, returns int"""
@overload
def echo(self, obj: float) -> float:
"""If the input is a float, returns float"""
"#
}
}When to use:
- Need complete control over method signatures
- Want to define overloads that don't match Rust implementation exactly
- Building code generation tools or macros
For classes where most methods can use automatic generation but some need manual signatures (e.g., complex type annotations or overloads), you can mix #[gen_stub_pymethods] with manual submit! blocks:
use pyo3::prelude::*;
use pyo3_stub_gen::{derive::*, inventory::submit};
/// Example demonstrating manual submission mixed with proc-macro generated method info
#[gen_stub_pyclass] // Use proc-macro for submitting class info
#[pyclass]
pub struct PartialManualSubmit {}
// Manually submit method info for the `PartialManualSubmit` class.
//
// Since we also use `#[gen_stub_pymethods]` for this class, what we should submit here are only:
// - `@overload` entries
// - Complex type annotations that cannot be expressed in the Rust type system for `#[gen_stub(skip)]`-ed methods
//
// Note
// ----
// The `submit!` invocation must appear before the `#[gen_stub_pymethods]` impl block when including `@overload` entries,
// because Python overload resolution depends on definition order and pyo3-stub-gen orders them by source position.
submit! {
gen_methods_from_python! {
r#"
import typing
import collections.abc
class PartialManualSubmit:
@overload
def echo_overloaded(self, obj: int) -> int:
"""Overloaded version for int input"""
def fn_override_type(self, cb: collections.abc.Callable[[str], typing.Any]) -> collections.abc.Callable[[str], typing.Any]:
"""Example method with complex type annotation, skipped from #[gen_stub_pymethods]"""
"#
}
}
/// Generates method info for the `PartialManualSubmit` class using proc-macro
#[gen_stub_pymethods]
#[pymethods]
impl PartialManualSubmit {
/// The constructor for PartialManualSubmit
#[new]
fn new() -> Self {
PartialManualSubmit {}
}
// Returns the input object as is
fn echo<'arg>(&self, obj: Bound<'arg, PyAny>) -> Bound<'arg, PyAny> {
obj
}
// Returns the input object as is (overloaded)
fn echo_overloaded<'arg>(&self, obj: Bound<'arg, PyAny>) -> Bound<'arg, PyAny> {
obj
}
/// Method with complex type annotation, skipped from #[gen_stub_pymethods]
#[gen_stub(skip)]
pub fn fn_override_type<'a>(&self, cb: Bound<'a, PyAny>) -> PyResult<Bound<'a, PyAny>> {
cb.call1(("Hello!",))?;
Ok(cb)
}
}Key points:
- ✅ Use
#[gen_stub(skip)]to skip methods that need manual type annotations - ✅ Place
submit!blocks before the#[gen_stub_pymethods]impl block for proper overload ordering - ✅ Combine automatic generation for simple methods with manual submission for complex cases
When to use:
- Most methods work with automatic generation
- A few methods need complex type annotations (e.g.,
Callable) - Need to add overloads to auto-generated methods
#[gen_stub_pyclass]
#[pyclass]
pub struct DataProcessor;
#[gen_stub_pymethods]
#[pymethods]
impl DataProcessor {
// Auto-generated signature
fn process_float(&self, x: f64) -> f64 {
x * 2.0
}
// Manual signature with complex types
#[gen_stub(python = r#"
import collections.abc
def process_callback(
self,
callback: collections.abc.Callable[[float], float]
) -> float:
"""Process with callback"""
"#)]
fn process_callback(&self, callback: Bound<'_, PyAny>) -> f64 {
// Implementation
0.0
}
}#[gen_stub_pyfunction]
#[pyfunction]
pub fn convert(x: f64) -> f64 {
x
}
submit! {
gen_function_from_python! {
r#"
def convert(x: pyo3_stub_gen.RustType["Vec<f64>"]) -> pyo3_stub_gen.RustType["Vec<f64>"]:
"""Convert list of floats"""
"#
}
}Generated stub:
from typing import overload
@overload
def convert(x: float) -> float: ...
@overload
def convert(x: typing.Sequence[float]) -> list[float]:
"""Convert list of floats"""submit! {
gen_function_from_python! {
r#"
import typing
class Comparable(typing.Protocol):
def __lt__(self, other: typing.Any) -> bool: ...
def sort_items(items: list[Comparable]) -> list[Comparable]:
"""Sort comparable items"""
"#
}
}pyo3-stub-gen-derive/src/gen_stub/parse_python/
├── mod.rs # Main parsing utilities
├── pyfunction.rs # Function definition parsing
└── pyclass.rs # Class definition parsing (future)
parse_python_function_stub (parse_python/pyfunction.rs):
pub fn parse_python_function_stub(
stub_str: &str,
original_item: &ItemFn,
) -> Result<PyFunctionInfo> {
// 1. Parse Python code to AST
let module = parse_program(stub_str, "<embedded>")?;
// 2. Find function definition
let func_def = find_function_def(&module)?;
// 3. Extract name
let name = func_def.name.to_string();
// 4. Build parameters
let parameters = build_parameters_from_ast(&func_def.args)?;
// 5. Extract return type
let return_type = extract_return_type_from_ast(&func_def.returns)?;
// 6. Extract docstring
let doc = extract_docstring(&func_def.body);
// 7. Extract imports
let imports = extract_imports(&module)?;
Ok(PyFunctionInfo {
name,
parameters,
return_type,
doc,
imports,
original_item: original_item.clone(),
})
}build_parameters_from_ast (parse_python/mod.rs):
fn build_parameters_from_ast(args: &ast::Arguments) -> Result<Parameters> {
let mut parameters = Vec::new();
// Positional-only parameters (before /)
for arg in &args.posonlyargs {
parameters.push(ParameterWithKind {
arg_info: extract_arg_info(arg)?,
kind: ParameterKind::PositionalOnly,
default_expr: extract_default(arg)?,
});
}
// Regular parameters
for arg in &args.args {
parameters.push(ParameterWithKind {
arg_info: extract_arg_info(arg)?,
kind: ParameterKind::PositionalOrKeyword,
default_expr: extract_default(arg)?,
});
}
// *args
if let Some(vararg) = &args.vararg {
parameters.push(ParameterWithKind {
arg_info: extract_arg_info(vararg)?,
kind: ParameterKind::VarPositional,
default_expr: None,
});
}
// Keyword-only parameters (after *)
for arg in &args.kwonlyargs {
parameters.push(ParameterWithKind {
arg_info: extract_arg_info(arg)?,
kind: ParameterKind::KeywordOnly,
default_expr: extract_default(arg)?,
});
}
// **kwargs
if let Some(kwarg) = &args.kwarg {
parameters.push(ParameterWithKind {
arg_info: extract_arg_info(kwarg)?,
kind: ParameterKind::VarKeyword,
default_expr: None,
});
}
Ok(Parameters::new_from_vec(parameters))
}Python syntax errors are reported at compile-time with helpful messages:
#[gen_stub_pyfunction(python = r#"
def my_func(x: int # Missing closing paren
"#)]Error:
error: Failed to parse Python stub: Expected ')', found newline at line 1
--> src/lib.rs:10:5
|
10 | #[gen_stub_pyfunction(python = r#"
| ^^^^^^^^^^^^^^^^^^^
Missing or invalid type annotations:
#[gen_stub_pyfunction(python = r#"
def my_func(x): # Missing type annotation
"""Example"""
"#)]Error:
error: Parameter 'x' missing type annotation
--> src/lib.rs:10:5
Invalid Rust type in RustType marker:
submit! {
gen_function_from_python! {
r#"
def my_func(x: pyo3_stub_gen.RustType["Vec<>"]): ... # Invalid Rust syntax
"#
}
}Error:
error: Failed to parse Rust type "Vec<>": expected type, found `>`
Test Python stub parsing:
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn test_parse_simple_function() {
let stub = r#"
def my_func(x: int) -> str:
"""Example function"""
"#;
let result = parse_python_function_stub(stub, &dummy_item_fn());
assert!(result.is_ok());
let info = result.unwrap();
assert_eq!(info.name, "my_func");
assert_eq!(info.parameters.len(), 1);
assert_eq!(info.doc, "Example function");
}
#[test]
fn test_parse_with_imports() {
let stub = r#"
import typing
import collections.abc
def my_func(x: collections.abc.Callable[[int], str]) -> typing.Any:
"""With imports"""
"#;
let result = parse_python_function_stub(stub, &dummy_item_fn());
assert!(result.is_ok());
let info = result.unwrap();
assert!(info.imports.contains(&"typing"));
assert!(info.imports.contains(&"collections.abc"));
}
}Use insta for snapshot testing:
#[test]
fn test_gen_function_from_python_output() {
let input = quote! {
gen_function_from_python! {
r#"
def example(x: int, y: str = 'default') -> bool:
"""Example function"""
"#
}
};
let output = gen_function_from_python_impl(input).unwrap();
insta::assert_snapshot!(output.to_string());
}// Good: Use Python syntax for Callable
#[gen_stub_pyfunction(python = r#"
import collections.abc
def my_func(cb: collections.abc.Callable[[int], str]) -> None: ...
"#)]
// Avoid: Complex override attributes
#[gen_stub_pyfunction]
#[gen_stub(override_return_type(type_repr="...", imports=(...)))]// Good: Let automatic translation handle simple types
#[gen_stub_pyfunction]
#[pyfunction]
fn simple_func(x: i32) -> String { /* ... */ }
// Avoid: Manual specification for simple types
#[gen_stub_pyfunction(python = r#"
def simple_func(x: int) -> str: ...
"#)]// Good: Use RustType to ensure consistency
submit! {
gen_function_from_python! {
r#"
def process(data: pyo3_stub_gen.RustType["MyConfig"]) -> None: ...
"#
}
}
// Avoid: Hardcoding type that might change
submit! {
gen_function_from_python! {
r#"
def process(data: MyConfig) -> None: ...
"#
}
}// Good: Document why overload is needed
// Overload for integer input (more efficient path)
submit! {
gen_function_from_python! {
r#"
def process(x: int) -> int:
"""Process integer (optimized)"""
"#
}
}Currently, gen_stub_pyclass(python = "...") is not supported. Use gen_methods_from_python! instead for class methods.
Python type annotations in stub syntax are not validated against Rust implementation at compile-time. Type checking happens only when users run mypy/pyright on the generated stubs.
Some complex Python type patterns may not be fully supported. Fallback to typing.Any or manual override if needed.
By default, type aliases are generated using pre-Python 3.12 syntax:
from typing import TypeAlias
MyAlias: TypeAlias = int | strFor Python 3.12+, you can enable the type statement syntax in pyproject.toml:
[tool.pyo3-stub-gen]
use-type-statement = trueThis generates:
type MyAlias = int | strNote
This configuration applies to all type aliases defined with the type_alias! macro and gen_type_alias_from_python!.
- Architecture - Overall system architecture
- Type System - Rust to Python type mappings
- Default Value for Function Arguments - Parameter default values
- Procedural Macro Design - How proc-macros work internally