Stack overflow on decoding Vec<Blobs>

[daltoncoder]

Was working on something that involved ssz encoding ethereum consensus payloads that contained blobs and I get a stack overflow when i trying to decode them. Happens right at blobs > 90_795 bytes with an empty function

simple reproduce:

use ssz::Encode as _;
#[test]
fn test_ssz_blob() {
    let blob = [0u8; 90_796];
    let vec_of_blobs = vec![blob];
    let decoded_bytes: Vec<[u8; 90_796]> =
        ssz::Decode::from_ssz_bytes(&vec_of_blobs.as_ssz_bytes()).unwrap();
}

output:

running 1 test

thread 'block::test_ssz_blob' has overflowed its stack
fatal runtime error: stack overflow

I think i can fix this on my end by implementing encode and decode myself, since i think ethereum blobs are already ssz encoded. But im curious if there is a way to fix this limit or is the idea of ssz encoding a list of blobs that size just not a good idea

[michaelsproul]

I'm curious, does the test pass if you run it in --release?

Will try to repro on our end and see what is causing the stack overflow. Our decoding of fixed size arrays might be a little inefficient

[daltoncoder]

I'm curious, does the test pass if you run it in --release?

Will try to repro on our end and see what is causing the stack overflow. Our decoding of fixed size arrays might be a little inefficient

Just tried it in --release and it did pass the previous test. In release mode it appears the new max blob size in empty function is [u8; 522_494]; anything higher than that it starts to fail with stack overflow again. Which i think would be fine for dealing with ethereum blobs since they are 128kbs. I even tried pushing alot of blobs onto the vec at that size and there was no stack overflow in release

[michaelsproul]

I suspect the issue is that we're storing the full blob on the stack during decoding, perhaps multiple times. I'm afk today but will take a proper look next week. Using Box<[u8; N> (or Arc) might be another way around it.

[daltoncoder]

@michaelsproul in case this is helpful when you take a look:

I poked around a bit to see if i could help and i dont think there is too much you can do since T::from_ssz_bytes returns self as direct value which in the case of T = [u8; N] will go to the stack
https://github.com/sigp/ethereum_ssz/blob/main/ssz/src/decode/impls.rs#L433

So maybe its just on the caller to be smarter and to use something like Vec instead of a fixed slice when dealing with large blobs since i think most stack limits are around 1-2mb

The use of collect() in that impl was moving alot onto the stack though and i was able to handle Vec of blobs up to size [u8, 348_187] (from 90_796) in debug mode if i changed the impl of Decode for Vec to this

    fn from_ssz_bytes(bytes: &[u8]) -> Result<Self, DecodeError> {
        if bytes.is_empty() {
            Ok(vec![])
        } else if T::is_ssz_fixed_len() {
            let fixed_len = T::ssz_fixed_len();

            let num_items = bytes.len() / fixed_len;
            let mut result = Vec::with_capacity(num_items);

            for chunk in bytes.chunks(fixed_len) {
                result.push(T::from_ssz_bytes(chunk)?);
            }

            Ok(result)
        } else {
            decode_list_of_variable_length_items(bytes, None)
        }
    }