Custom Tree-sitter Grammars

[ahenager]

It would be useful if there was a way to supply custom tree-sitter grammars to Semble. This way, if there is a grammar you don't support, users can implement it themselves without forking your repository. This would also be useful for DSLs that would probably never be needed by other users.

There is prior art in Coco-Search, as it provides a similar functionality.

[stephantul]

Hello! Thanks for your interest.

This is interesting, but not something we'll likely implement. tree-sitter-language-pack has very good support already. In addition, for languages for which we don't have tree-sitter support, we fall back to line chunking, which actually works really well. We're also likely removing tree-sitter-language-pack in the near future, in favor of simpler chunking.

[ahenager]

Thanks for the reply. I must be missing something in how to configure this, as files in our custom templating language are just not being indexed, based on their extension (.kjs).

[stephantul]

Hey @ahenager,

Ah that's interesting and a different issue IMO. If custom code isn't picked up, we can add an option to include these files based on extension. That's separate from how they are chunked. We'll discuss if that's feasible.

[ahenager]

Well, .kjs files are pretty much just useful to those in our company, so the only semantic way I can think to add this is just add a configuration option to include file types for line chunking indexing, perhaps?

[stephantul]

Yeah exactly, we actually already expose the opion to do this via the python API.

from semble.index import SembleIndex

s = SembleIndex.from_path("my/path", extensions=[".kjs"])

The caveat is, however, that this will only index .kjs files, and nothing else. If you want to recover the original behavior but extend it with .kjs files, you could do this:

from semble.index import SembleIndex
from semble.index.file_walker import FILE_TYPES

s = SembleIndex.from_path("my/path", extensions=[".kjs", *FILE_TYPES])

But we don't expose this option yet through the mcp server or the cli.

[ahenager]

Thanks, I can probably find a way to use this. I think it would be ideal to be able to use this with the CLI/MCP, but I can work around that or create a PR if necessary.