Make serialized programs runnable in a fresh engine (constant map merge + faithful round trip)#677
Open
koalazub wants to merge 2 commits into
Open
Conversation
ConstantMap::to_bytes wrote constants as display strings and from_bytes re-parsed and re-lowered them, so quoted forms collapsed ((quote x) came back as x), let-form list constants were rewritten as lambdas, and whitespace characters could not be re-read at all. Serialize the constants through a structural serde mirror of the constant value domain instead, so deserialization rebuilds the exact values that were written.
…ant map A source-compiled RawProgramWithSymbols shares the engine compiler's ConstantMap, so constant indices in its bytecode and everything the VM runs afterwards agree on one table. A deserialized program carried its own detached map: run_executable installed that map as the thread's live table while the compiler kept its own, so an eval performed while the program ran compiled indices against the compiler's map but read them out of the program's map, returning the wrong constant or a free identifier. raw_program_to_executable now rebases detached programs onto the compiler's constant map, rewriting PUSHCONST, EQUALCONST and register-immediate payloads, which also removes the constant-map swapping in execute_non_interactive_program_image. Also replace the unwrap-heavy write_to_file/read_from_file pair, which silently appended .txt to the given filename, with serialize_to_path and deserialize_from_path taking the caller's path and propagating errors; the old names remain as thin wrappers.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Found these while building startup caching for the nothelix plugin on the helix fork that I use. We compile the whole plugin as one RawProgramWithSymbols, serialise it to disk, and run it from the artifact on later launches so the editor doesn't recompile ~9k lines of scheme every start at this pint of writing.
Two things came up on that path.
Running a deserialised program installs its constant map as the VM's live table while Compiler::constant_map stays separate. The program itself runs fine, but any eval afterwards compiles constant indices against the compiler's map and resolves them in the program's map, so you get FreeIdentifier or the wrong literal back. The fix rebases a detached program onto the compiler's live map in raw_program_to_executable (add_or_get and rewriting the constant referencing payloads), and drops the map swap in execute_non_interactive_program_image since it's covered by the same path now.
ConstantMap's to_bytes/from_bytes went through display strings and a reparse, which re-lowers on the way back in. (quote x) came back as x and let forms came back as lambdas, so a round trip changed the program.
note: merge_constants_into has to know which opcodes reference constants (PUSHCONST, EQUALCONST and the register PASS variants). I checked the list against the current vm.rs but it's the kind of thing that could rot quietly if new opcodes land. Happy to restructure if you'd rather that knowledge live somewhere else.