Arabic mark reordering description could mislead

[bobh0303]

I appreciate the inclusion of UTR#53 discusion in opentype-shaping-arabic.md, and I appreciate the desire to simplify the wording to make it easier to understand.

However the wording as currently in the document could mislead readers.

First, readers will need to understand that the first step of UTR#53 AMTRA algorithm is to perform NFD normalization. As far as I can tell, this step is completely omitted from opentype-shaping-arabic.md.

I am aware that the authors of Harfbuzz disagree with NFD, preferring to use NFC. But in either case, the mark order is normalized, which is essential to the remainder of the algorithm.

Secondly, opentype-shaping-arabic.md then gives the following statement:

Second, move any subsequence of combining-class-230 characters that begins with a 230_MCM character to the beginning of the sequence, before all "Shadda" characters. The subsequence must be moved as a group.

This is misleading because it says to move the entire sequence of combining-class-230 marks, when in fact UTR53 says to just move the initial MCMs from that sequence:

If a sequence of ccc=230 characters begins with any MCM characters, move the sequence of such MCM characters to the beginning of S (before any characters with ccc=33). (emphasis added)

There is a similarly-worded statement for combining-class-220 marks which similarly diverges from UTR#53.

Finally, it is important to realize that the algorithm -- just like Unicode normalization itself -- isn't applied to the complete sequence of marks that might be adjacent to an Arabic letter. Rather, it is applied to each maximal-length substring, S, of non-starter (D107) characters. The nuance is that a U+034F: COMBINING GRAPHEME JOINER is a mark but is a Starter in Unicode terms (since it has ccc=0). In essence, CGJ characters divide a mark sequence up into pieces and then mark reordering (whether part of normalization or AMTRA) happens within those pieces, but not across their boundaries.

(I'll admit that this last point may be more nuanced than you want to cover within this document. But the concept applies to Unicode normalization in general, so might be appropriate somewhere else in your documents.)

Regards,
Bob

[n8willis]

Hi Bob; thanks for writing these notes up — you may be one of the only people who has taken a look at this portion of the docs, so your input is a great help.

Secondly, opentype-shaping-arabic.md then gives the following statement:

Second, move any subsequence of combining-class-230 characters that begins with a 230_MCM character to the beginning of the sequence, before all "Shadda" characters. The subsequence must be moved as a group.

This is misleading because it says to move the entire sequence of combining-class-230 marks, when in fact UTR53 says to just move the initial MCMs from that sequence:

Well, the intent of the docs sentence was to say "just move the MCM ones", but if that's not clear then rewording is in order for sure. (The phrasing with "subsequence" is the crux of it, if I understand you correctly.)

My first take on a rephrasing might be something like this:

Second, if any sequence of combining-class-230 characters begins with a subsequence of 230_MCM characters, then move that 230_MCM subsequence to the beginning of the combining-class-230 sequence, before all "Shadda" characters. The 230_MCM subsequence must be moved as a group.

How does that read, from your POV? It is a lot of repetition of sequence/subsequence and the 230 terms in a short space, so there is high potential for confusion or highway-blindness, I'm sure. The last sentence may be superfluous, even.

The 220 case would be handled in the same fashion, naturally.

First, readers will need to understand that the first step of UTR#53 AMTRA algorithm is to perform NFD normalization. As far as I can tell, this step is completely omitted from opentype-shaping-arabic.md.

I am aware that the authors of Harfbuzz disagree with NFD, preferring to use NFC.

Yeah ... this one is a fair bit trickier, as you have observed. HarfBuzz's normalization model is what is recorded in the normalization doc in this repo, and the original intent of the project was to codify "what HarfBuzz does" into a clean-roomable reference, so we do probably need to not diverge from that.

Perhaps the root issue is that the entire normalization discussion was added quite a ways after the initial batch of shaper-behavior documentation (Indic2 and Arabic, at least). So it, too, has probably received a lot less scrutiny from readers.

I think it's somewhat up for debate how much normalization discussion can get worked into these docs in a prescriptive form, just because implementers in the real-world may be doing any number of other operations on the text input before handing it. HarfBuzz's "defer to the font" normalization approach makes sense for displaying output, but other components in the application stack would presumably have other requirements.

As a result, currently the individual script documents only refer to normalization in a couple of spots, as a note that it happens and as an option when discussing decomposition.

In both cases, that's basically just a "you should expect normalization to be handled before you get here" that passes the buck. We'd probably have to elevate it to a more direct discussion to get into specifics. I would be interested to hear what some implementers have to say about it, though.

Finally, it is important to realize that the algorithm -- just like Unicode normalization itself -- isn't applied to the complete sequence of marks that might be adjacent to an Arabic letter. Rather, it is applied to each maximal-length substring, S, of non-starter (D107) characters.

I should probably think about this one a little more, especially since this reply is already long. But updating the discussion of CGJ seems useful and non-destructive, since it's not really covered at present.

[n8willis]

On a tangentially related matter, @bobh0303 would you have the bandwidth to suggest some valid sequences that could be used to illustrate AMTRA? I think visuals would certainly help at least some readers.