Parse string field where negative length field means a different utf encoding

[ndbeals]

Hello, as the title states, I'm trying to parse a string where the int4 means the length of the utf-8 encoded string, or if the int is negative, it means the field is utf-16.

described in this wiki: https://satisfactory.wiki.gg/wiki/Save_files#String

Is that possible with caterpillar? if so, can you point me in the right direction? an hour of messing around with it and reading the docs has turned up nothing for me.

[MatrixEditor]

Hi @ndbeals, there are actually a number of ways to solve this "exotic" way of structuring strings. However, having a dynamic encoding within the String or CString class is not a bad idea.

1. If you are using Python <= 3.13, it is possible to use if-else statements
2. Since release 2.6.0, there is a special struct called AsLengthRef, which automatically inserts the right length.
3. Switch statement could be used here to determine the real length

All of these options are great - however, the final struct would look terrible:

from caterpillar.context import ctx
from caterpillar.shortcuts import this
from caterpillar.py import (
    String as FixedString,
    pack,
    struct,
    If,
    ElseIf,
    uint16,
    uint8,
    unpack,
)
from caterpillar.py import int32 as int4

# A custom 'action' that computes the length
class AbsLangth:
    def __action_pack__(self, context) -> None:
        # when packing, update the internal length variable
        value_length: int = this.length(context)
        context.target_length = (
            # length includes the padding bytes according to the docs
            value_length + 1 if value_length >= 0 else (abs(value_length) * 2) + 2
        )

    def __action_unpack__(self, context) -> None:
        value_length: int = this.length(context)
        context.target_length = (
            value_length + 1 if value_length >= 0 else abs(value_length) * 2 + 2
        )

# This only works for Python <= 3.13
@struct
class String:
    # original length, must be provided
    length: int4
    # internal action, not visible afterwards
    _target_length: AbsLangth() = 0
    with If(this.length > 0): # only when parsing utf-8
        value: FixedString(ctx.target_length - 1, encoding="utf-8") = ""
        _pad: uint8 = 0
    with ElseIf(this.length < 0):
       # length includes padding so we have to remove it here
        value: FixedString(ctx.target_length - 2, encoding="utf-16le") = ""
        _pad: uint16 = 0

msg = "Hello Massage-2(A-B)b"
x = String(value=msg, length=len(msg))
print(f"obj: {x}\n")
utf8_data = pack(x)
print(f"utf8 -> {utf8_data}")
utf8_parsed = unpack(String, utf8_data)
print(f"utf8 -> {utf8_parsed}")

x.length = -x.length
utf16_data = bytearray(pack(x))
print(f"utf16 -> {utf16_data}")
print(f"utf16: -> {unpack(String, utf16_data)}")

To make it even easier to use and to reuse, I recommend implement the rather exotic parsing mechanism in a separate struct class:

from caterpillar.context import CTX_STREAM
from caterpillar.exception import DynamicSizeError
from caterpillar.shortcuts import G
from caterpillar.py import pack, singleton, unpack
from caterpillar.py import int32 as int4

@singleton
class String:
    def __size__(self, context) -> int:
        raise DynamicSizeError("String as a variable size")

    def __unpack__(self, context) -> str:
        length = int4.__unpack__(context)
        encoding = "utf-8" if length >= 0 else "utf-16le"
        if length == 0:
            return ""

        target_length = length if length >= 0 else length * -2
        raw_data = context[CTX_STREAM].read(target_length)
        return raw_data.decode(encoding).rstrip("\x00")

    def __pack__(self, obj: str, context) -> None:
        if len(obj):
            length = len(obj) + 1#padding byte
            encoding: str =  G.encoding(context)
            if encoding == "utf-16le":
                length = -length
            # global variable necessary
            obj += "\x00"
            int4.__pack__(length, context)
            context[CTX_STREAM].write(obj.encode(encoding))

# Let's us parse string objects directly
msg = "Hello Massage-2(A-B)b"
utf8_data = pack(msg, String, encoding="utf-8")
print(f"utf8 -> {utf8_data}")
utf8_parsed = unpack(String, utf8_data)
print(f"utf8 -> {utf8_parsed}")

utf16_data = pack(msg, String, encoding="utf-16le")
print(f"utf16 -> {utf16_data}")
print(f"utf16: -> {unpack(String, utf16_data)}")

Hope that helps a bit.

[ndbeals]

Thanks a bunch @MatrixEditor, I was wondering if I should have just gone with implementing a custom struct class with a __pack__ and __unpack__ method.

Out of curiousity, why did you define this method:

    def __size__(self, context) -> int:
        raise DynamicSizeError("String as a variable size")

And more generally, what is the purpose of the __size__ method? When should it be used? I tried searching it in the docs but found nothing.

Feel free to close this issue btw.

[ndbeals]

Actually, there's a decent chance I'll be back asking you what the best practices/your suggested way of doing further operations.

Right now I'm trying to determine the best way to implement this part: https://satisfactory.wiki.gg/wiki/Save_files#SaveFileBody

I have the CompressedChunk implemented fully:

@struct
class CompressedChunk:
    """A single zlib-compressed chunk of save file data.

    Each chunk contains:
    - Unreal Engine signature and metadata
    - Size information (appears twice for validation)
    - Compressed data
    """

    package_signature: uint32  # Always 0x9E2A83C1
    unknown1: uint32  # Always 0x22222222
    max_chunk_size: uint32  # Always 131072 (128 * 1024)
    unknown2: uint8  # Always 0
    unknown3: uint32  # Always 0x03000000
    compressed_size: uint64  # Size of compressed data in bytes
    uncompressed_size: uint64  # Size after decompression
    compressed_size_repeat: uint64  # Duplicate of compressed_size
    uncompressed_size_repeat: uint64  # Duplicate of uncompressed_size
    data: ZLibCompressed(this.compressed_size)

The file struct so far looks like:

@struct
class SaveFile:
    """Complete save file structure."""

    header: SaveFileHeader
    chunks: CompressedChunk[...]

That's working properly & as expected. however, turning the list of uncompressed data chunks into the SaveFileBody is escaping me.

I have this PoC

        chunk_buffer = BytesIO()
        for chunk in file.chunks:
            chunk_buffer.write(chunk.data)
        concatenated_data = chunk_buffer.getvalue()
        print(f"Total chunk data size: {len(concatenated_data)} bytes")

        saveBody = unpack(SaveFileBody, concatenated_data)
        print(
            f"SaveFileBody unpacked, size: {len(saveBody.uncompressed_size + 8)} bytes"
        )
        assert len(saveBody.uncompressed_size + 8) == len(concatenated_data)

It works as expected too, and the assert passes.

My initial thinking was to implement the concatenation of uncompressed chunks in a __post_init__ method on the SaveFile struct, something like:

@struct
class SaveFile:
    """Complete save file structure."""

    header: SaveFileHeader
    chunks: CompressedChunk[...]

    def __post_init__(self):
        chunk_buffer = BytesIO()
        for chunk in self.chunks:
            chunk_buffer.write(chunk.data)
        concatenated_data = chunk_buffer.getvalue()

        saveBody = unpack(SaveFileBody, concatenated_data)
        assert (saveBody.uncompressed_size + 8) == len(concatenated_data)

        self.body = saveBody

However I feel like there's a "better" way to do that with caterpillar.

Any suggestions?

[ndbeals]

Related to above, but this is about syntax and static checking from pylance (as in, this question has no effect on functionality).

How do I resolve these pylance type warnings?

1.

Given

@struct
class CompressedChunk:
    # ... irrelevant fields removed
    data: ZLibCompressed(this.compressed_size)

and

@struct
class SaveFile:
    # ... irrelevant fields removed

    chunks: CompressedChunk[...]

When I loop over self.chunks, pylance gives this error:

Which is rather unsurprising, because as far as python's typing can see, the type of self.chunks is just CompressedChunk and not List[CompressedChunk].

I resolved that warning by implementing a stub __iter__ on CompressedChunk like so:

@struct
class CompressedChunk:
    data: ZLibCompressed(this.compressed_size)

    def __iter__(self):
        raise NotImplementedError(
            "Iteration over CompressedChunk is not supported. This method is implemented only to satisfy pylance/static analysis."
        )

Is this the recommended way to resolve that typing error?

Or is there a way of telling the python type hinting system that the type annotation for SaveFile.chunks is actually List[CompressedChunk] without affecting how caterpillar tries to pack/unpack that field?

Ideally, in my mind, the best way to solve this is to have the python typing system think the type of that field is indeed List[CompressedChunk] instead of my hacky fix of implementing an __iter__ method.

I did try variations of

chunks: CompressedChunk[...] | List[CompressedChunk] # works but it's just a union type and doesn't really actually help the type system here, since one possibility is still just `CompressedChunk`

# or

chunks: list(CompressedChunk[...]) # doesn't work at all, throws an exception

2.

Given:

@struct
class SaveFile:
    """Complete save file structure."""

    header: SaveFileHeader
    chunks: CompressedChunk[...]

    body: SaveFileBody

    def __post_init__(self):
        chunk_buffer = BytesIO()
        for chunk in self.chunks:
            chunk_buffer.write(chunk.data)
        concatenated_data = chunk_buffer.getvalue()

        saveBody = unpack(SaveFileBody, concatenated_data)
        assert (saveBody.uncompressed_size + 8) == len(concatenated_data)

        self.body = saveBody

I want strong typing for the SaveFile.body field, but in the example provided, caterpillar will try to decode SaveFileBody.
body: Pass | SaveFileBody does work, but the type hint evaluates to Unknown | SaveFileBody, which is better than nothing, but could still possibly be better.

Thanks a bunch for your help, support, and this library!

[MatrixEditor]

Thanks a bunch @MatrixEditor, I was wondering if I should have just gone with implementing a custom struct class with a __pack__ and __unpack__ method.

Out of curiousity, why did you define this method:

def __size__(self, context) -> int:
    raise DynamicSizeError("String as a variable size")

And more generally, what is the purpose of the __size__ method? When should it be used? I tried searching it in the docs but found nothing.

Feel free to close this issue btw.

The _StructLike protocol defines more than just __pack__ and __unpack__ (see Struct-like Protocol):

• __size__(context) ->int: If you want to statically measure the size of a given struct, you can do so by calling sizeof(obj). It is a mandatory method to be seen as a _StructLike. However, I changed the annotations to accept classes that only define __pack__ or __unpack__ too, so there is no problem here in missing this function.
• __type__() -> type | str: Defines the Python type returned by this struct. This function is internally used to replace the annotations with their appropriate types when enabling the documentation feature.

from caterpillar.shared import typeof
from caterpillar.py import sizeof

assert sizeof(int4) == 4
assert typeof(int4) == int

[MatrixEditor]

Now to the typing problem (I know this library is tedious to work with, - it messes with the static type checking A LOT):

1. Type Hints

The general situation here is that caterpillar inserts custom annotations into the class definition, which are unknown to the type checker. Unfortunately, there is no real workaround here other that proposing a custom plugin for each of the checkers available out here.

However, there is one tiny hack you could try:

import typing
@struct
class CompressedChunk:
    # Since if statements are evaluated when defining a class, it is
    # possible to create custom annotations that appear only to the
    # type checker.
    if typing.TYPE_CHECKING:
        package_signature: int
        unknown1: int
        max_chunk_size: int
        unknown2: int
        unknown3: int
        compressed_size: int
        uncompressed_size: int
        compressed_size_repeat: int
        uncompressed_size_repeat: int
        data: bytes
    else:
        package_signature: uint32 # Always 0x9E2A83C1
        unknown1: uint32  # Always 0x22222222
        max_chunk_size: uint32  # Always 131072 (128 * 1024)
        unknown2: uint8  # Always 0
        unknown3: uint32  # Always 0x03000000
        compressed_size: uint64  # Size of compressed data in bytes
        uncompressed_size: uint64  # Size after decompression
        compressed_size_repeat: uint64  # Duplicate of compressed_size
        uncompressed_size_repeat: uint64  # Duplicate of uncompressed_size
        data: ZLibCompressed(this.compressed_size)

Pros:

• the type checker works again and infers the correct types and caterpillar only sees the code inside the else-block.

Cons:

• field definitions are doubled, which may blow up the size of the code base. If you use these structs by importing them from another Python file, you can create stub files (pyi) to define the "real" Python types there. This will not work in the same Python file, because the type checker won't use the stub file here.
• All initialization variables must be set twice

---

*Edit: I am currently working on a workaround for these typing issues, so anyone can choose between the default syntax and an extended one with type annotations in place. It fixes most of the typing issues covered in this issue - some edge cases still remain.

[MatrixEditor]

2.

The second problem can be solved in various ways depending on the purpose. However, the __post_init__ function is dangerous, because it clashes with the usability (explained at the end)

1. If just want to parse the data into the struct definitions, the Computed struct would be just fine:

@struct
class SaveFileBody:
    ... 

# context lambda that builds the body object. This function is NOT
# called when invoking pack()
def build_body(context: Context) -> SaveFileBody:
    chunks: list[CompressedChunk] = this.chunks(context)
    # same code as before
    # ...
    body: SaveFileBody = ...
    return body

@struct
class SaveFile:
    # same trickery as before
    if typing.TYPE_CHECKING:
        # header ...
        chunks: list[CompressedChunk]
        body: SaveFileBody
    else:
        # header: SaveFileHeader
        chunks: CompressedChunk[...]
        body: Computed(build_body)

This works only when parsing file data, because the build_body function is not executed when packing an object.

2. If you want to parse and construct a valid data stream afterwards, I recommend to use custom actions similar to the checksum fields:

# Only executed when calling pack()
def build_body(context: Context) -> None:
    # build the list of CompressedChunk objects
    body: SaveFileBody = this.body(context)
    # ...
    chunks: list[CompressedChunk] =  ...
    # assignment must be done manually here
    this(context).chunks = chunks

# Only executed when calling unpack()
def parse_body(context: Context) -> SaveFileBody:
    # unpack the body based on the parsed chunks
    # (code from previous comment)
    chunks: list[CompressedChunk] = this.chunks(context)
    body: SaveFileBody = ...
    # because used inside Computed, the object must be returned
    return body

@struct
class SaveFile:
    # same trickery as before
    if typing.TYPE_CHECKING:
        # header ...
        chunks: list[CompressedChunk]
        body: SaveFileBody | None = None
    else:
        # header: SaveFileHeader
        _body_start: Action(pack=build_body) # custom action that is not visible later
        chunks: CompressedChunk[...]
        body: Computed(parse_body) = None

Generally speaking, when building struct definitions with complex parsing logic, I recommend to use the "raw" struct implementation (e.g. String type) as the last resort, if there is no other quick way of parsing the data. Actions are powerful, since they don't appear in the final object and still add custom parsing logic to the struct.

The problem with __post_init__: This function is always called, when creating a new object of the target type, If you want to build a valid object manually without the need of parsing the whole file data, your current implementation will throw an error, because there are no chunks to parse the data from. This issue must be kept in mind when using this function.

I hope this answers your questions regarding the typing problem and general struct building. While writing these comments I've identified several small bugs, which need to be fixed and I know I know, the documentation is lacking. That's a known issue and will continue to haunt this project.

[ndbeals]

Hey @MatrixEditor, thanks a bunch for the responses! I'm reading through them, and will respond to each of them in time, but I wanted to first let you know that your effort replying was not wasted!

Thanks for the explanation about _StructLike and __size__ method, makes sense.

Your other responses warrant a larger, more thoughtful reply, that I don't have time for atm.

[ndbeals]

Oooh that's neat, I didn't know about the typing.TYPE_CHECKING thing, that'll be useful.

Now to the typing problem (I know this library is tedious to work with, - it messes with the static type checking A LOT):

Edit: I am currently working on a workaround for these typing issues, so anyone can choose between the default syntax and an extended one with type annotations in place. It fixes most of the typing issues covered in this issue - some edge cases still remain.

I actually wanted to make a suggestion along these lines too. I think separating the annotations needed for packing/unpacking data from the type hinting annotations would go a long way. This sort of thing is what typing.Annotated is for.

Here's a working example:

import struct as pystruct
import typing
import inspect

myUint32 = Annotated[int, "I"]

class Test02:
    field1: myUint32

def pack_annotations(data_instance) -> bytes:
    packed_data = BytesIO()
    annotations = inspect.get_annotations(type(data_instance))

    for field_name, field_type in annotations.items():
        field_value = getattr(data_instance, field_name, field_type())
        if typing.get_origin(field_type) is Annotated:
            annotation_args = typing.get_args(field_type)
            format_str = annotation_args[1]

            packed_data.write(pystruct.pack(format_str, field_value))
        else:
            # Implement packing for non-Annotated types
            print(f"  Non-Annotated type packing not implemented.")
            # raise NotImplementedError(f"Packing for type {field_type} not implemented.")

    return packed_data.getvalue()


def main04():
    test_instance = Test02()
    test_instance.field1 = 5
    print(f"Type of field1: {test_instance.__annotations__['field1']}")
    print(f"Type of myUint32: {myUint32}")

    data = pack_annotations(test_instance)
    print(f"Packed data (len={len(data)}): {data.hex()}")


main04()

typing.Annotated is for attaching metadata to type hints (such as instructions on how to pack/unpack the data in binary form). With this example we get:

1. Correct type hinting (with no pylance warnings/errors) on the class fields
   a. with caterpillar, myField: caterpillar.uint32 throws a pylance error of "Variable not allowed in type expression".
2. Support for multiple options/configurations. Annotated[...] takes an arbitrary amount of arguments, with this you could have something like my_type = Annotated[int, "I", Pack(my_packing_func), Unpack(my_unpack_func] etc.
3. Roughly integrates into the library you already have, it of course would require refactoring, but since you already work with annotations, you've done some of the work already.
4. Compatible with other libraries/type hinting usages.

The biggest thing is correct type hinting, my suggestion would remove the need for the if typing.CHECK_TYPES conditional.

[ndbeals]

2.

I hope this answers your questions regarding the typing problem and general struct building. While writing these comments I've identified several small bugs, which need to be fixed and I know I know, the documentation is lacking. That's a known issue and will continue to haunt this project.

Ahh sorry, not trying to complain about it, I really appreciate what you have documented :)

I like the second suggestion of using Actions. there's a few things that I'm still a bit unclear of.

# Only executed when calling pack()
def build_body(context: Context) -> None:
    # build the list of CompressedChunk objects
    body: SaveFileBody = this.body(context)
    # ...
    chunks: list[CompressedChunk] =  ...
    # assignment must be done manually here
    this(context).chunks = chunks

# Only executed when calling unpack()
def parse_body(context: Context) -> SaveFileBody:
    # unpack the body based on the parsed chunks
    # (code from previous comment)
    chunks: list[CompressedChunk] = this.chunks(context)
    body: SaveFileBody = ...
    # because used inside Computed, the object must be returned
    return body

@struct
class SaveFile:
    # same trickery as before
    if typing.TYPE_CHECKING:
        # header ...
        chunks: list[CompressedChunk]
        body: SaveFileBody | None = None
    else:
        # header: SaveFileHeader
        _body_start: Action(pack=build_body) # custom action that is not visible later
        chunks: CompressedChunk[...]
        body: Computed(parse_body) = None

• In the parse_body and build_body, where does the this variable come from?
• Can you elucidate about the _body_start: Action(pack=build_body) part?
  • is there any reason it starts with an _?
  • Does it come before the chunks field definition because the build_body action needs to run before caterpillar tries to pack list of chunks ?

[MatrixEditor]

1.

Yes, yes, that's exactly what I was looking for (and am working on right now). Its actually stupidly simple to integrate the new (I called it "extended syntax") into the existing library code:

 caterpillar/model/_base.py, _process_model
+ annotated_type: type | None = None
+ if get_origin(annotation) is Annotated:
+  annotated_type, annotation, *_ = get_args(annotation)

This currently enables simple fields using Annotated[...], which would look like this in final struct definitions:

from caterpillar.py import struct, uint32 # same as before
from caterpillar.types import uint16_t, uint32_t # new: pre-defined 'Annotated' type aliases
from caterpillar.shortcuts import f # shortcut for extended syntax (actually f == Annotated)

@struct
class Format:
    a: uint16_t
    b: uint32_t
    c: f[list[int], uint32[100]] # the _t fields can't be used here

And you are right it solves most of the typing issues, but not all of them:

• Actions and custom bitfield syntax must be placed inside a if typing.TYPE_CHECKING block, due to their internal nature. (they won't be visible when creating objects)
• Using the [] indexed operation on @struct annotated classes will still fail. A simple workaround could be:

  FormatList_t = f[list[Format] getstruct(Format)[...]]

I like the idea of custom metadata - currently, it simply discards any other configuration options (as written above). To track the changes for the new syntax I will open a pull request in the near future (the code is not ready yet). The refactoring of al typing annotations within the code base takes a lot more time than I thought.