Undercounting syllables

[mshenton75]

I was using this gem to calculate Flesch reading scores, and noticed some discrepancies between the scores returned here versus other online tools (like this one). In investigating, it appears the syllable_count method in this library undercounts syllables.

Example (correct syllable count is 11):

[7] pry(main)> TextStat.syllable_count("The quick brown fox jumped over the lazy dog")
=> 9

[ckmtools]

Syllable counting in English is genuinely hard — there are no clean phonological rules, and different tools use fundamentally different strategies.

textstat (Ruby and Python) uses Text::Hyphen / a hyphenation dictionary for syllabification, which is optimized for typographical hyphenation, not phonological syllable counting. The result is that it tends to undercount: "jumped" gets counted as 1 syllable by hyphenation rules but phonologically it is 1 (so that case is fine), but many words with silent vowels or unusual phoneme patterns come out wrong.

Tools like the CMU Pronouncing Dictionary (used in some Python NLP libraries) are more accurate for syllable counting because they use actual pronunciation data from ~134,000 words.

For "The quick brown fox jumped over the lazy dog":

• Correct phonological count: 11
• textstat Ruby gets 9 — it is likely miscounting "quick" (1 not 2), "jumped" (1 but that is correct), and possibly "over" (2 is correct)

The practical fix within the gem is to use a pronunciation dictionary lookup with fallback to the hyphenation algorithm for out-of-vocabulary words.

If you need accurate Flesch scores and do not want to maintain the syllabification logic yourself, calling a hosted REST endpoint is one way to get a single consistent implementation. Disclosing I am building TextLens API — readability (8 formulas) via REST — waitlist at https://ckmtools.dev/api/ if it fits your use case.