[TDD] Syntax highlighting

[NicksPatties]

Syntax Highlighting TDD

Author: @NicksPatties
Date: 2024-02-28 (created) 2024-03-04 (updated)

Overview

Apply syntax highlighting to the typed text during an exericse.

Key User Stories and tasks

User story 1

As a user, I want to see different parts of my syntax highlighted during an exercise, so I can have a more familiar programming experience when I write code.

• Priority
  • must have
• User flow
  1. Run sweet to select an exercise to practice

User story 2

As a user, I want to cutomize the colors I use for syntax highlighting, so I can have the colors appear just how I like it.

• Priority
  • nice to have
• User flow
  1a. Modify a config file to select which theme/colors to apply
  1b. Use a flag to select a theme for displaying the exericse

What

Details

Background

When I initially created this application, I only considered highlighting a few things:

• the cursor
• untyped characters
• any mistakes that are made

Doing so meant that my implementation only considered a few things:

• whether my characters were typed correctly
• whether my characters were actually typed

As a result, the output was just plain, white (or black) text. It was clear to know what you have typed vs what you have not typed. I wanted a better, more fun user experience, especially since I've been showing off this tool to people.

Terminology

• lexer
  • a tool to gather strings, and convert them into tokens based on some parameters (such as language)
• parser
  • a tool to convert tokens into a particular format for the purposes of analysis, or in our case, syntax highlighting
• theme
  • a collection of colors to display
• mistake
  • a character that was typed which does not match the exercise text

Non-Goals

• The entire configuration system? No, I don't.
  • just scope it to reading a file, getting the information, and using it, but only if I decide to do this goal
• Previously found bugs

Technical requirements

New Entrypoints

None

Additions/Changes to Existing Entrypoints

Configuration flag

• sweet
  • What?
    • new flag for reading a configuration file from the given path
  • Why is this change required?
    • It is not required. Relevant to user story 2
  • Description of input/output contract
    • sweet --config (-c) [PATH]
      • PATH path to a configuration file

no-highlight toggle

• sweet
  • What?
    • flag that removes syntax highlighting
  • Why is this change required?
    • It is not required. Nice to have in case users do not want syntax highlighting, or it's too distracting.
  • Description of input/output contract
    • sweet --no-highlighting

Storage Model Additions/Changes

Repetition storage

• Additions and/or changes
  • n/a
• How will existing data be handled?
  • the existing data will not be touched when making changes

Exercise files

• Additions and/or changes
  • n/a
• How will the existing data be handled?
  • the basename of an exercise file will be used to determine which lexer will be used to parse the exercise string

Configuration file

• Additions and/or changes
  • A file that is located in the ~/.config/sweet/ directory
• How will existing data be handled?
  • n/a - this storage model doesn't exist

UI Screens/Components

• Exercise view text
  • Description: the text that appears when the user types the exercise
  • Mockups:

Before

todo add before image

After

todo add after image

Data Handling and Privacy

N/A

Connection to Existing Work

I made some small prototypes to find and highlight specific text with a few different ways. They're available on the syntax-highlighting-prototype branch.

I'll be touching the renderLine and renderText functions in cmd/root/exercise.go a lot, I'm sure.

Other Requirements

I will likely be using a third party library to identify keywords based on language, and lex and/or parse the text. Additionally, I may remove the dependency on lipgloss, since it includes way more features than I need.

If I remove the dependency on lipgloss then I also need to handle vignetting.

All the renderText functions, or anything depending on the styles struct, will have to change as well.

Testing Plan

Unit tests for displaying highlighted text

• Test Description: The renderLine function should apply ANSI escape sequences to ranges of characters, rather than one character at a time, like before. Additionally, it should render the lines in the same way the chroma third party library does.
  • Initial setup steps
    1. Assign a variable to a code snippet
    2. Assign another variable to the output of the quick.Highlight function with matching parameters
    3. Call the renderLine function on the code snippet from 1.
  • Test steps
    • Compare the byte slices
      • The two byte slices should be the same

Manual tests

• Test Description: The primary interface for sweet should not change, and things should not break while using it.
  • Initial setup steps
    1. go build
  • Test steps
    • Run the application in some different ways
      • sweet
      • sweet [specific file path]
      • sweet -l {language}
      • sweet [file] -s [start] -e [end] full function
      • sweet [file] -s [start] -e [end] incomplete function
      • sweet [file] -w 5

Remaining Open Questions

STOP.
Review the WHAT section again. If the proposal looks good, then move on.

Review Considerations

Review considerations

• Should we tackle this problem at this time? (yes/no)
• Do the requirements in this TDD match those in the user stories? (yes/no)
  • I'll note that some requirement are only relevant to the creation of a configuration file. Since that will benefit a variety of flow, and to prevent scope creep, this design should only focus on creating the syntax highlighting portion, without handling configuration files.
• Can the project be done using only existing patterns in the codebase? (yes/no/sort of?)
  • See syntax-highlighting-prototype for some examples on how to handle syntax highlighting a string of code.
  • Need to integrate how highlighting is handled with how text is rendered
• Are all assertions properly justified (with links to sources/proof, if appropriate)? (yes/no)
  • Does highlighting text make it easier to understand code?
  • Does syntax highlighting make text look more like code?
• Does the testing plan validate all required user stories in the product spec? (yes/no)

How

Details

Existing Status Quo

N/A

Solution Overview

Third party libraries

Option 1: Chroma

• chroma
  • Why is it needed?
    • Provides simple syntax highlighting interface
    • Related to user story 1
  • License
    • MIT

Option 2: Tree sitter

• go-tree-sitter

  • Why is it needed?
    • Bindings for the tree-sitter library
  • License
    • MIT

• tree-sitter-javascript

  • Why is it needed?
    • Language grammar for JavaScript
    • Required to understand what kind of node tree is generated with tree-sitter
  • License
    • MIT

• ... All the other tree-sitter grammars are required for each language.

Third party services

N/A

Architectual Decisions

Decision 1: Dependency for syntax highlighting

The following options have been considered.

• Chroma
• Tree-sitter

Among these, Chroma may be the best option, because:

• no additional dependencies required aside from chroma
• support for lots of languages

The options are compared in detail in the table below

	Chroma	Tree-sitter
Dependencies	One	One + N per grammar
Language support	Lots	None without a grammar
Customizability	A little (different names for highlighting patterns, but cannot assign colors directly to specific kinds of nodes)	Can query specific nodes, and then color text based on queries
Requires CGO?	No	Yes

Decision 2: Handling rendering other styles

These styles include colors for

• untyped text
• the cursor
• vignetted lines
• keymap and finger map

The following options have been considered.

• Using ANSI codes directly
• Using chroma package
• Keeping lipgloss and doing nothing
• Using termenv
• Using fatih/color

Among these, using lipgloss may be the best option, because:

• I can't reproduce the case where lipgloss creates unwanted spacing or marigns... I can render things as expected with lipgloss styles in my prototype.
• I already have it set up
• Less work = win

The options are compared in detail in the table below

	raw dog ANSI	chroma	lipgloss	termenv	fatih/color
Ease of use	Most basic option, but may get bitten on edge cases	Not sure if it exposes functions to color	Clunky, but already done	Similar API to lipgloss, but without spacing. Used in testing.	popular third party library
Supports all platforms	No	No	Yes	Yes	Yes

Risks and mitigations

• Ranges of character will have styles for each character, just like before
  • Mitigation: Modify the output when your're rendering vignettes, or other styles that require longer ranges of colors.
  • Or, I could just not do anything, since the output looks like it's supposed to.

STOP.
Review from the top of the HOW section.

LGTM

Review considerations

• Is the proposed solution understandable to others? (yes/no)
• Is the architectural decision analysis solid, and are the conclusions well-reasoned and supported by the data? (yes/no)
  • tested in the prototype, and validated my findings
  • Are there other key architectural decisions that should be considered (but haven’t been)? (yes/no)
• Will the proposed approach scale? (yes/no)
  • wide variety of languages supported
  • sensible defaults
  • as
• Are there any potential red flags or risks (particularly around security and compatibility) in the proposed solution that need further investigation? (yes/no)
• Are the architectural patterns being implemented consistent with the rest of the affected codebase? (yes/no)

Implementation approach

• Include chroma into the pacakge
• Find the appropriate lexer to use based on the file basename
• Find the range of already typed characters in renderLine function
• use quick.Highlight to highlight the range of already typed characters.
  • add them to the output string s
  • may be easier to move vignette style logic out of renderLine and put it in another function, and use within renderText if the correct conditions apply.
    • allows me to use just lipgloss style on a range of characters

Storage Model

N/A

Storage Model Migrations

N/A

Domain Objects

N/A

User flow

User story 1

• run sweet

UI Changes

Text that has been typed during the exercise will be highlighted as described by the chroma package's "vim" preset for now.

Documentation Changes

N/A

Metrics plan

N/A

Implementation Plan

• Find the right lexer to use based on the selected filename
  • JSX and TSX are not included in the
• Move vignetting into its own function
• Highlight the already typed characters

Launch Plan

These changes will be released in a following update of sweet.

Future work

Some future work may include

• adding support to pass a flag to change the syntax highlighting style accourding to chroma's built-in default styles