How do I ensure side effects of a parser is executed?

[lynzrand]

I am writing a parser that builds a CST using cstree. Since it uses a builder to build the tree, all tree-building operations are side effects, with the actual parser returning ().

A demo project can be seen in this gist: https://gist.github.com/lynzrand/713ba99f3a6987b38ceb569758cb8bdf

Although the .ignored() combinator only states to ignore its outputs and keep ZST containers non-allocating, it seems like it also suppresses side effects of other parsers, like the fold callback in .pratt(). (In the demo, you can see that there's no printing to stdout)

To ensure the CST gets built correctly, I need to replace .ignored() with .map(drop) (or equivalent; you can try this out in the gist's code and see the difference), and ensure there's no .ignored() containing any parsing subtree that needs to be built.

So, my question is: is there a way to ensure side effects are executed even if it's in .ignored(), or is there a better way to work with CSTs?

---

Update: #928 referenced that closures should be semantically "pure". That means that if I want to construct a CST sideband, I need to be extra careful to not let chumsky optimize away any side effects?

[zesterer]

Hmm, so if I'm understanding you correctly, the issue is that your map/fold functions generate outputs, but also produce a side-effect that is observed by your CST state?

So, as stated, the mapping/folding functions are intended to only be used for transforming parser outputs. In the short turn you can 'hack' a fix together by using .try_map(|x| Ok(x)): because try_map can change the outcome of a parse, and because it depends on the output, it forces previous outputs to be generated.

Out of interest, why does node creation have to be impure? Is it not sufficient to simply use Inspector::on_token to record the input as it is read?

[lynzrand]

Yes, the CST state needs to be created as a side effect of parsing. I'll try try_map for this usage, thanks!

---

The side effect comes from cstree. In cstree (if I'm using it correctly), the green tree must be built using GreenNodeBuilder. The builder builds the tree using an internal stack of seen tokens and subtrees, and the user calls method to manipulate the internal state.

From its docs:

// Build a tree
let mut builder: GreenNodeBuilder<MySyntax> = GreenNodeBuilder::new();
builder.start_node(Root);
builder.token(Int, "42");
builder.finish_node();
let (tree, cache) = builder.finish();

If I'm using it correctly, impl Inspector's on_token() only solves the problem of capturing the concrete token list when building the tree, but not building the tree nodes that contains the tokens. As you can see, the methods for building tree nodes do not return anything and is pure side effect.

I ended up with these implementations:

Implementation of Inspector that adds the source string span as a token when each token is eaten into chumsky.

impl<'src> Inspector<'src, ParseInput<'src>> for NodeBuilder<'src> {
    type Checkpoint = cstree::build::Checkpoint;

    fn on_token(&mut self, token: &lex::Token) {
        let slice = &self.src[token.span.into_range()];
        self.inner.token(token.kind, slice);
    }

    fn on_save<'parse>(
        &self,
        _cursor: &chumsky::input::Cursor<'src, 'parse, ParseInput<'src>>,
    ) -> Self::Checkpoint {
        self.inner.checkpoint()
    }

    fn on_rewind<'parse>(
        &mut self,
        marker: &chumsky::input::Checkpoint<'src, 'parse, ParseInput<'src>, Self::Checkpoint>,
    ) {
        self.inner.revert_to(*marker.inspector());
    }
}

Checkpointing and node creation utilities. (P!['a, O] == impl Parser<'a, Input<'a>, O, Extra<'a>>)

/// Consumes no input but creates and return a checkpoint on the node builder
pub fn ckpt<'a>() -> P!['a, cstree::build::Checkpoint] {
    custom(|iref| {
        let builder: &mut NodeBuilder = iref.state();
        let ckpt = builder.inner.checkpoint();
        Ok(ckpt)
    })
}

/// When the given parser succeeds, commit a CST with the given kind, using all
/// subtrees and tokens parsed within parser.
///
/// The output type of this parser is the same as the input parser.
pub fn for_node<'a, O>(kind: Kind, parser: P!['a, O]) -> P!['a, O] {
    custom(move |iref| {
        let builder: &mut NodeBuilder = iref.state();
        let ckpt = builder.inner.checkpoint();

        let res = iref.parse(&parser)?;

        let builder = &mut iref.state().inner;
        builder.start_node_at(ckpt, kind);
        builder.finish_node();

        Ok(res)
    })
}

for_node itself seems to emit the correct tree structure regardless of whether ignored() is used, since it's a custom parser and is probably not optimized. However, there's no equivalent parser when using Pratt parsing, which led to the issue met in the main thread.