pyo3 kyle Q&A

[jessekrubin]

Howdy @kylebarron,

I was just curious if you have experimented with single vs multi threaded tokio runtimes and if so, did you find anything interesting?

Maybe a dumb question, but does the python gil negate the benefits of the tokio runtime?

Best,
jesse

[kylebarron]

The Python GIL does not negate the benefits of the tokio runtime as long as the Rust tasks are longer than the Python tasks. The Python tasks are extremely lightweight futures that just represent when the heavier Rust tasks finish. But I would like to make it user configurable how many cores tokio should use (or just better document the environment variable tokio uses to choose how many cores to use in the threadpool)

[jessekrubin]

Misc question:

In the validate_buffer call when extracting py-bytes, is the strides doing anything? Are there any cases where the buf.is_c_contiguous() check doesn't cover strides other than 1?

fn validate_buffer(buf: &PyBuffer<u8>) -> PyResult<()> {
    if !buf.is_c_contiguous() {
        return Err(PyValueError::new_err("Buffer is not C contiguous"));
    }

    if buf.strides().iter().any(|s| *s != 1) {
        return Err(PyValueError::new_err(format!(
            "strides other than 1 not supported, got: {:?} ",
            buf.strides()
        )));
    }

    Ok(())
}

[kylebarron]

I think in this case the stride doesn't matter because we're forcing it to be c contiguous.

[jessekrubin]

💯 that is what I was thinking but thought you might know better!

[jessekrubin]

@kylebarron Do you think it would be worth adding some sort of fast path for the pyo3_bytes::PyBytes struct to bypass the buffer protocol just for the python native bytes?

[kylebarron]

It should be microseconds of overhead

[jessekrubin]

Hi @kylebarron

Have you played with different global allocators for obstore? if so what were you findings?

PS: I changed this discussions name

[kylebarron]

No I haven't. You're welcome to test different allocators and make an issue with what you find.

[jessekrubin]

Will do if I find the time. Was just curious if you had.