Encoding formatted text

[adrianholovaty]

Here's a proposal for how to encode "formatted text" — that is, text that contains optional inline formatting instructions. @mscuthbert, @dspreadbury and I have come up with it and refined it, off and on, over the last few months.

The idea is that formatted text would be encoded as an array of "chunks," where each chunk has its own optional formatting instructions. Let's jump in with some example encodings.

Examples

[
  {
    "text": "Test ",
    "style": {"font": "Times New Roman", "size": 12}
  },
  {
    "text": "example ",
    "style": {"font": "Times New Roman", "size": 12, "weight": "bold"}
  },
  {
    "text": "here",
    "style": {"font": "Times New Roman", "size": 12, "weight": "bold", "font-style": "italic"}
  }
]

[
  {
    "text": "Text in which the formatting is not provided"
  }
]

[
  {
    "text": "Text with\na newline"
  }
]

[
  {
    "text": "Tempo di Marcia ",
    "style": {"font": "Freight Text Pro", "size": 16}
  },
  {
    "smuflGlyphs": ["noteQuarterUp"],
    "smuflStyle": {"font": "Bravura", "size": 20}
  },
  {
    "text": " = 90",
    "style": {"font": "Proxima Nova", "size": 14}
  }
]

Text content

As I hope the examples make clear, each chunk must define either "text" or "smuflGlyphs". This allows for interspersed text with musical glyphs — obviously very common in music notation and a key requirement for this.

Chunks would be read sequentially, and a graphics engine would be responsible for assembling them together with the proper formatting. (For right-to-left writing systems, we'd need to provide a way to specify RTL.)

All space (including newlines and spaces between words) must be explicitly encoded within the "text" strings. JSON supports newline characters encoded directly in strings as "\n".

A few notes on "smuflGlyphs":

• This uses the canonical glyph names, rather than the underlying Unicode codepoints, for readability.
• It's an array, to support multiple glyphs displayed directly next to each other.
• Some smufl glyphs likely won't make sense in this context, or might break graphical rendering. I'm not sure whether there's anything we can or should do about this.

Styling

Each chunk in the array can have its own "style", which specifies which typeface to use, along with size and other formatting. Please don't read too much into the values I made up here ("font-style", "weight") — that's all up for discussion and refinement. The main idea is that "style" would be a reasonably simple and easy-to-parse dictionary with the styling instructions.

Note this is deliberately not CSS, because I want to avoid requiring people to parse an entirely different language when they're parsing MNX. We could perhaps be inspired by the CSS property names, but I'd like to keep this as a dictionary of key-values instead of a string of CSS.

Global styles

These examples don't include any concept of global styles. It would be nice if an MNX document could define global styles and individual formatted text instances could reference them. For example:

[
  {
    "text": "Tempo di Marcia ",
    "styleID": "tempoMarkingName"
  },
  {
    "smuflGlyphs": ["noteQuarterUp"],
    "styleID": "tempoMarkingNote"
  },
  {
    "text": " = 90",
    "styleID": "tempoMarkingNumber"
  }
]

And somewhere else in the document, the styles could be defined, along with some document-level defaults:

{
  "styles": {
    "tempoMarkingName": {"font": "Freight Text Pro", "size": 16},
    "tempoMarkingNote": {"font": "Bravura", "size": 20},
    "tempoMarkingNumber": {"font": "Proxima Nova", "size": 14}
  },
  "defaults": {
    "style": {"font": "Times New Roman", "size": 12},
    "smuflStyle": {"font": "Bravura", "size": 18}
  }
}

Note this deliberately uses two separate keys "style" and "smuflStyle". One is the default text style and the other is the default smufl glyph style.

Flat vs. nested

This array would be flat, not nested. Clearly another approach would be to use nesting (hierarchy). After some spirited discussion, the three of us designing this concluded the flat approach is easier for developers to work with.

The downside of flatness is: there's redundancy. But the upside is a significantly simpler mental model.

Prior art

@mscuthbert looked into prior art for JSON formats for encoding formatted text but didn't find anything that seemed to meet our needs. If anybody is aware of existing JSON formats that could work for MNX, please chime in with the info.

Which MNX objects use formatted text?

For now, let's deliberately sidestep the question of which MNX objects actually use formatted text. It's tempting to just add a way to encode generic "text boxes" within a score, but that raises several questions such as:

• How are the box's coordinates and dimensions encoded?
• Which units do these measurements use? (e.g., staff spaces vs. paper-based units)
• Is a box placed absolutely (at an X/Y coordinate within a page) or relatively (at some offset from a musical element)?
• Is a box encoded within the sequence data or within an instantiation (a score)? If the former, is it possible to change the box's coordinates and dimensions on a per-score basis?

All interesting stuff to think about, but our immediate goal is to just design the encoding of formatted text itself. Later we can address the encoding of graphic layout information.

Comments

That's basically it! Thoughts and feedback welcome.

[mscuthbert]

I think we had thought of having font be an array "font": ["Adobe Garamond", "Times New Roman", "serif"] like css; but I think with global styles it's a lot of redundancy rather than having a single point of truth (it's not like the backup for TNR will be serif here, but sans-serif there...) so I'm happy to remove that suggestion. Maybe the styles can have have fallbackID: "otherStyleID" to solve that problem.

Note this deliberately uses two separate keys "style" and "smuflStyle". One is the default text style and the other is the default smufl glyph style.

Can we call the two defaults "textStyle" and "musicStyle"?

• textStyle because these two might not end up being the only two defaults that we'll end up defining similarly here (lineStyle? etc.).
• musicStyle rather than smuflStyle because we may end up supporting mappings of codePoints to semantic meaning besides SMuFL (like for older, pre-SMuFL fonts), and I still want to be optimistic and think that someday in the lifetime of MNX, Unicode will adopt the entire SMuFL range somewhere and that could be a future as well. (the default should of course be SMuFL mappings)

btw, I was the main proponent of hierarchical/cascading styles rather than flat encoding but as I did more research it appeared that hierarchies are primarily endorsed for exchanging/storing a document and flat for actively working with/manipulating a document, and since one of the key features of MNX over MusicXML is that it can be a format for active work/internal representation, it seems that flat is better.

I do think that one type of cascading might be very very useful and not hard to implement, and that is: a run of text should be able to have both a styleID reference and a local style object. So that you could do something like this:

[
  {
    "text": "Fast, but not ",
    "styleID": "expressiveText"
  },
  {
    "text": "too",
    "styleID": "expressiveText",
    "style": {"font-style": "italic"}
  },
  {
    "text": " fast",
    "styleID": "expressiveText"
  }
]

where a change in definition of expressiveText (font-family, font-weight, font-size) could cascade to all three text runs but still preserve the italic. This "base style + override" is supported by a lot of modern music software.

[rpatters1]

What is the list of values for "weight" and "font-style", at least on day one? Also, some fonts build their weight into the font name "Arial Bold", "Arial Bold Italic". For my purposes it will be difficult to tease out family name, weight, and style from these (working only from enigma data), so the spec needs to be lenient about this.

Enigma strings allow the following.

• effects italic, bold, underline, and strikeout. It would be nice to include at least underline.
• baseline adjustment and superscript adjustment. (I haven't figured out what the difference is: they both seem to accomplish the same thing of moving the baseline up or down.)
• tracking. This adjusts inter-character spacing and it expressed in EMs. (Which are 1/1000 of an em-dash width in the font.)

The last two bullets are occasionally quite useful, but might be considered Cadillac features in MNX.

Finally, Enigma provides a mechanism for text to be "hidden" and to be "absolute". "Hidden" means it displays lightly on screen but is not printed. Visible and hidden text can be mixed in a single text block. "Absolute" means it always displays actual size, regardless of scaling context. Both are very useful.

Finally, I am not too concerned about size, but I wonder if the spec would allow individual font elements to be changed without having to restate the entire font information. That means the first example could alternatively be encoded as

[
  {
    "text": "Test ",
    "style": {"font": "Times New Roman", "size": 12}
  },
  {
    "text": "example ",
    "style": {"weight": "bold"}
  },
  {
    "text": "here",
    "style": {"font-style": "italic"}
  }
]

It's a lot less letters. The downside is that "bold" and "italic" would need to have a way to be turned off. Whatever the committee decides, this approach in general with work very well with Enigma text.

[mscuthbert]

I don't think that on and off switches are something we're considering because they're not how any software we examined encodes text since Wordperfect 5.1 (which was great in its own way). The two choices are basically state the style fully for each run of text, or have a hierarchy of inheritance. The latter has its positives too, but not enough to definitely make it more important than the former.

I think the idea is to support modern font families where there can be more than 4 styles. A lot of the older families that include "Bold" as part of their name was because they used "Font" + Bold to mean "Semibold" and "Font Bold" = real bold (and some allowed "Font Bold" + Bold = "Black" or faux Black). If "Bold" is part of the font's name then it would need to be included in the font name here.

[rpatters1]

About the SMuFL glyph names, will it be possible to use optional glyph names if the font has those glyph names?

Also, I hope it will still be possible to embed code points in the "text" field in any case.

[markkaylor]

This is my first time contributing here, so apologies if I’m missing context, but I wanted to share a bit of prior art that might be relevant. Frameworks for building rich text editors like TipTap, Lexical, and Slate represent their data as JSON and they also deliberately avoid CSS.

Those editors go a little further by avoiding all presentational styling in the data model. They rely on semantic nodes and marks . Nodes describe the structure, and marks describe the formatting, while the renderer decides how everything is actually displayed. This removes any notion of CSS from the content model and also avoids encoding things like \n directly in strings.

I realize this might push against the goal of keeping everything fully flat, but I thought it might still be helpful to share in case it sparks any ideas or aligns with some of the design goals.

Please feel free to disregard if this doesn’t fit where MNX is headed, just hoping it might be useful as another data-model reference point.

TipTap inspired example:

[
  {
    "type": "heading",
    "content": [
      { "type": "text", "content": "Tempo di Marcia" },
      { "type": "smuflGlyph", "content": ["noteQuarterUp"] },
      { "type": "text", "content": " = 90" }
    ]
  },
  {
    "type": "paragraph",
    "content": [
      { "type": "text", "content": "Test" },
      { "type": "text", "content": "example", "marks": [{ "type": "bold" }] },
      {
        "type": "text",
        "content": "here",
        "marks": [
          { "type": "bold" },
          { "type": "italic" }
        ]
      }
    ]
  },
  {
    "type": "paragraph",
    "content": [
      { "type": "text", "content": "paragraph node type creates new line" }
    ]
  }
]

[rpatters1]

One of the issues that arises in music is that the size of staves (or staff spaces) relative to text varies from program to program. Specifically

• Finale uses 24 point music fonts for the notehead to exactly fill a space.
• MuseScore (and possibly Dorico) use 20 point music fonts for the notehead to exactly fill a space.

This value has a knock-on effect for every font size in the document, including those that do not scale with staffsize. (Even if a font does not scale with staffsize, its appearance relative to any given staff is affected by the music fontsize.)

We need to be able to specify the music fontsize somewhere. This is the size that an unscaled staff uses. And we need to be able to specify whether a fontsize scales with staffsize in any specific text. The latter probably needs to be either inline in each text or part of a style specification.

[rpatters1]

It turns out (after wrestling with this in MuseScore), that what we need is a default staff size (or staff-space size). This is a size in actual units (perhaps mm) of a staff with no scaling applied. Without this, it is impossible to scale font sizes (that scale with staff size) to match the source. This could either be a document-level setting or a convention by fiat.

[dougheyer]

In the engraving business there are several standard staff or "punch" sizes: Giant, English, Peter, Cadenza. Etc. Each size is given in inches and mm. Usually the staff size is equivalent to the size of a C clef, note size the height of a staff space. Pick a staff size as it would appear on the printed page then do what you want with the display. It is often convenient for the user to be able to specify sizes and distances relative to a space size. I have found it useful to give a space the value of 1, staff size is 4, zoom around and let matrix transformation handle the details.
Because text and symbol sizes can vary depending on context, text size must always either be implicit based on predefined text purpose (title, tempo, lyric, and so on) or specific. Whatever the case we need the flexibility to specify the measurement type (inches, mm, points, staff spaces) as well as the size. And the user should be able to enter sizes with the same flexibility.

[rpatters1]

In my experience (MuseScore and Finale), the standard engraving "punch" sizes are expressed as scaling percentages of the unscaled size, which is not a user facing value. I think I should probably explain why unscaled staff size has to be part of the coordinate system for font sizes.

In music systems, much (if not most) text scales with the staff size. To interchange between them, the font sizes have to be scaled based on the difference in size of the unscaled staff. Otherwise, the text appears too small or too large relative to staves in the new system. Let me illustrate with an example.

Finale's unscaled staff size is exactly 24 points (72 DPI points). So if I create a tempo mark with Times 14, it looks like this relative to the staff:

MuseScore's unscaled staff size is an oddly specific fractional number of millimeters, but it is roughly 83% the size of Finale's. So Times 14 looks like this in MuseScore.

But if we scale that 14 point size by the difference in unscaled staff sizes, the equivalent font size in MuseScore becomes 14 * 0.8333 = 11.667. With Times 11.667 in MuseScore, the text looks like this.

Now, as you can see, the Tempo marking is almost exactly the same size relative to the staff as in Finale. If I had used the exact MuseScore default size it would be exactly the same. But I would have to dig into a debugger to find it, and this is close enough for the illustration.

The point is, MNX must specify the size (in points or mm) of an unscaled staff. It can be a doc setting, or the committee can simply pick a number. Without one, interchange of text sizes between music programs becomes impossible.

[rpatters1]

I should also reiterate that there must be a font setting that specifies whether text scales with the staff or is a fixed size irrespective of staff size. This is a requirement for musical text that may not exist in other kinds of text handling contexts. I would defer to anyone else's superior knowledge on that point, though.