Prepare for Unicode 17 support in Go 1.27 - #814
Open
apparentlymart wants to merge 3 commits into
Open
Conversation
apparentlymart
force-pushed
the
f-unicode-17
branch
from
July 7, 2026 00:10
a603c25 to
32425f4
Compare
apparentlymart
force-pushed
the
f-unicode-17
branch
from
July 7, 2026 00:15
32425f4 to
37c093c
Compare
This was referenced Jul 7, 2026
Go 1.27 is adopting Unicode 17 in the standard library, and so to match that we'll use the grapheme cluster rules from Unicode 17 whenever we're building with Go 1.27. We continue to use Unicode 16 for Go 1.25 and 1.26. Part of the work here is done by upgrading to cty 1.19.0, which introduces its own compile-time switch between Unicode 15 and 17 depending on the Go version. This causes go.mod to list both versions of go-textseg, but in practice only one of them is linked into the final program. Since upgrading cty causes both versions of go-textseg to be in the dependency graph anyway, this then introduces an internal package unicodeutil which uses conditional compilation to select between the two textseg versions depending on the Go version. That is using the same strategy as cty itself uses. go-textseg v17 also removes its assorted helper functions including textseg.TokenCount, so I've copied its former implementation into the new package unicodeutil and specialized it to be for grapheme clusters only, since that's all that HCL had been using it for. I have added the IBM copyright comments as required by PR checks in the repository, but please note that this is not copyright assignment in practice. In particular: - The small helper functions in package unicodeutil are taken from other codebases for which I am the copyright holder. To the extent that these small snippets are considered copyrightable, I am offering them for HCL under the terms of the Mozilla Public License. - The other changes here were made under sponsorship from Spacelift, Inc and therefore copyright on these contributions belongs to Spacelift. Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
This script was written for an older version of Ruby where "open" was capable of directly opening a URL, but in modern Ruby it only supports local files and so we need to be explicit that we're attempting to read from a URL here. This does not change the output from this tool. It just makes it runnable without having to install an ancient Ruby version. Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
This extends the set of characters permitted in HCL identifiers to the full set allowed in Unicode 17's version of UAX#31. It seems like this hasn't been updated in a long while since it was lagging behind the rest of HCL still on Unicode 9, but since everything else in HCL is now ready for Unicode 17 (once running on Go 1.27) this'll make things consistent again. Note that the other recent changes to support Unicode 17 were included only when building with Go 1.27 or later, but this particular change applies regardless of which Go version is used. Although this is a little inconsistent, in practice it doesn't matter a great deal if the identifier rules are from a newer Unicode version because this just means that more identifiers are accepted as valid, without changing the meaning of any previously-valid configurations. Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
apparentlymart
force-pushed
the
f-unicode-17
branch
from
August 11, 2026 15:55
37c093c to
8b020d2
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
The forthcoming Go 1.27 will update to Unicode 17 tables and algorithms for the standard library. For consistency with that, this PR prepares HCL to use Unicode 17 for its own tables and in the upstream Unicode-related libraries it depends on. In summary, this deals with everything described in How Terraform uses Unicode, except for finally updating Terraform's own
go.mod.To avoid needing to tightly coordinate switching library versions and Go toolchain versions at the same time, here I've matched the new strategy being used by upstream go-cty where it uses conditional compilation to use Unicode 15 text segmentation on Go 1.25 and 1.26, but automatically switch to Unicode 17 when compiling with Go 1.27 or later. This means that Terraform and other HCL dependents can upgrade to a new HCL version containing these changes without immediately adopting Unicode 17 as long as they remain on Go 1.25 or 1.26, with one exception: the identifier rules.
While preparing this I noticed that the data for
ID_StartandID_Continuehad not been updated for a long while and was still using the Unicode 9 range tables. While addressing that I also noticed that the Ruby script for generating the Ragel file from that data was no longer working on recent Ruby versions, and so I applied the same small fix I previously applied to another copy of that same script in my upstream library go-textseg.Due to how the code generation in this repository is implemented it's not really feasible to support multiple different versions of the Unicode Identifier specification (UAX#31) with conditional compilation and so for that one in particular I made the compromise of unconditionally using Unicode 17 regardless of Go version. I expect this is an okay compromise because it just means that a few more characters will be accepted in HCL identifiers, without changing the treatment of any currently-valid HCL configuration.
You'll see in the diff that I also copied the function previously known as
textseg.TokenCountinto the newunicodeutilinternal package and simplified it to only support grapheme cluster counting since that's all HCL needs. This is because that function no longer exists in upstream textseg as of v17. That function is so trivial that I doubt it is copyrightable at all, but if it is then I'm the copyright holder and I'm contributing it under the MPL for inclusion in HCL.How Has This Been Tested?
I ran the full unit test suite both on Go 1.25 (the current minimum supported version) and on Go 1.27rc1 as a placeholder for the forthcoming Go 1.27 release.