ICU is not supported in this build

[mindplay-dk]

Are you building pglite without ICU support?

When running this query in the REPL:

CREATE DATABASE mydb
  ENCODING = 'UTF8'
  LOCALE_PROVIDER = 'icu'
  ICU_LOCALE = 'und'
  TEMPLATE = template0;

I get:

ICU is not supported in this build

Would it be possible to provide it as an extension? I'm afraid maybe it's not.

I realize ICU may add substantially to the binary footprint, and I'm guessing you omitted this for size?

But without it, PGLite works basically for English and not much else - any multi lingual application is going to require a Unicode collation, so that users can sort and search as expected. Even in English-only software, user names often have special characters, so this matters for sorting and searching even then.

I would guess this adds at least 5 MB of footprint, so this would likely triple your footprint. Not that 10 MB is a lot in this day and age. And PGLite has uses outside of the browser, on the server. (Not that sorting and searching is any less important on the client side, but...)

Would you consider shipping two builds?

[tdrz]

@mindplay-dk Thank you for the very good report!

Valid points, I believe ideally we would support it as an extension or some sort of module that is bundled in a different package, such that (hopefully) it doesn't influence the core size. I don't think shipping two builds is something that we'd like to do atm.

I hope I'll have some time next week to give it a first try. No promises though!

[tdrz]

Are you building pglite without ICU support?

in postgres-pglite/build-pglite.sh: --without-icu

[tdrz]

@mindplay-dk I don't have a lot of experience with locales and I'm trying to understand our options here. I see the following related options in initdb:

      --locale=LOCALE       set default locale for new databases
      --lc-collate=, --lc-ctype=, --lc-messages=LOCALE
      --lc-monetary=, --lc-numeric=, --lc-time=LOCALE
                            set default locale in the respective category for
                            new databases (default taken from environment)
      --no-locale           equivalent to --locale=C
      --builtin-locale=LOCALE
                            set builtin locale name for new databases
      --locale-provider={builtin|libc|icu}
                            set default locale provider for new databases

Currently we're setting locale-provider=libc. We're also looking into making it possible to allow changing these params to initdb (although locale-provider=icu will obviously not work if we're not building PGlite with it) as if using the native initdb.

My question is: does setting locale-provider=libc help in any way in building apps for other languages?

[mindplay-dk]

My question is: does setting locale-provider=libc help in any way in building apps for other languages?

I don't think so? libc provides access to locale installed on your system - in WASM, there is no OS that libc can query for locales, so this must be faked or hard coded in the PGLite build somehow, I assume?

There is Intl.Collator in the browser, which provides basically all locales - but the overhead of bridging WASM to JS would probably destroy performance.

You might be able to cut back the size of ICU with a custom build that only supports und - that would probably be enough for most multi-lingual applications, but it's likely still a 3-5 MB footprint.

There's the newer icu4x (an official Unicode project) which tries to be smaller and more modular - but I don't believe this is a "drop-in replacement" by any means, and I don't know if it's substantially smaller.

You could maybe slim down an ICU build by removing some of the larger, more complex languages like Chinese/Japanese, but that doesn't exactly sound like a fair proposition. 😌

I'm afraid the bottom line is that anyone who builds multi-lingual applications is going to be using ICU on their servers, because that's what's available, and that's what it's for. Syncing those databases to PGLite is simply not going to work, or not going to work as expected.

I spent another hour brainstorming and researching, and I am not aware of any smaller alternative.

ICU is large because the locale collation data is large. 😌

[mindplay-dk]

Sadly, from my perspective, this means I can't really consider PGLite, as we have customers all over the world - in my experience, when developers build products without proper unicode sorting/searching, this is always inevitably the first bug report that rolls in, as soon as your product gets any real traction and adoption. The answer is and always has been "switch to a Unicode collation".

Sorting and searching is a pervasive feature in just about any product that stores text, even something as basic as user names - in my honest opinion, this is not something you can skip: it is a hard requirement in just about any product.

And if it isn't (e.g. all your customers are in the US) it eventually will be - if your product succeeds, you will need to support other markets. It's only a matter of when.

Discovering that your entire approach to data management on the frontend is a dead end then would probably come as a shock to most, or wouldn't you agree? 😅

It's crazy impressive how much functionality you crammed into PGLite with that tiny footprint.

But some things just can't be made smaller than they can. 😌

[tdrz]

@mindplay-dk Thank you for the details and the time you put into this.

Unfortunately I didn't have time to look into it this week. Will make it in the end...

[tdrz]

I don't think so? libc provides access to locale installed on your system - in WASM, there is no OS that libc can query for locales, so this must be faked or hard coded in the PGLite build somehow, I assume?

The default libc comes with emscripten.

[tdrz]

@mindplay-dk I had a look at building with icu in this draft PR. See the first comment for details.

[mindplay-dk]

I don't understand all the details, but this looks promising! I did some quick testing and it seems to work. 😄

I guess it's not surprising that this would work - but what are your conclusions about the file size concerns so far?

Is it possible to ship only the ~1 MB runtime and then load the actual locales dynamically?

Is 1 MB acceptable for you in terms of footprint?

It seems like there was some added load time when I started the sandbox (at https://69ae897d36f88f534a351aa9--pglite.netlify.app/repl/) but is it currently still loading the full set of locales? There was no noticeable delay on subsequent page reloads, it seems snappy enough once it's cached...

[tdrz]

I guess it's not surprising that this would work - but what are your conclusions about the file size concerns so far?

It is not surprising but it's a good confirmation 😄

We definitely need a separate package for the locales, potentially with separate files for each locale, or bundled in reasonable sized files. Probably we also need a minimal locale file to allow Postgres to start.

Is it possible to ship only the ~1 MB runtime and then load the actual locales dynamically?

Like I said in that draft PR, it should be possible as long as we populate the filesystem before Postgres starts.

Is 1 MB acceptable for you in terms of footprint?

Unsure atm. I definitely don't like it, I was hoping Postgres would only use those libs "on request" - we could have then bundled them separately with the data file in a separate package. Also, plenty of people are using PGlite "as is", haven't heard many complaints about locales until now.

It seems like there was some added load time when I started the sandbox

The whole thing is ~5x larger with icu so that's expected!

[mindplay-dk]

The whole thing is ~5x larger with icu so that's expected!

yeah, ouch. 😅

well, I don't really have the knowledge to work on something this technical myself.

but I hope you'll keep the issue open, maybe someone will come along who can figure it out.

Also, plenty of people are using PGlite "as is", haven't heard many complaints about locales until now.

well, that's normal, no one ever notices until they start onboarding some italian/spanish/french/german users and they can't find anything with your search or sortable UI 😅