add: load schema cache dump from URI at startup

[mkleczek]

Resolves #4999
by making it unnecessary.

[steve-chavez]

Helping to clarify, the motivation is to solve the same problems on #4999. This by loading the schema cache from a file.

[steve-chavez]

Q: If the schema cache file is stale, we still pick it up and serve requests as "best effort"?

[mkleczek]

Helping to clarify, the motivation is to solve the same problems on #4999. This by loading the schema cache from a file.

@steve-chavez I am still working on the right design and will provide more details soon.

I believe now that we can solve in this PR all pain points we want to solve with materialized views.

[mkleczek]

Q: If the schema cache file is stale, we still pick it up and serve requests as "best effort"?

Yes the idea is to treat schema cache preloaded from the dump as stale and trigger asynchronous load from the db.

[steve-chavez]

So if the URI is slow then we will prefer querying the schema cache.

Won't this will always be the case when doing HTTP requests? It seems like this feature will only make sense when it's obtained from a file since that'd be faster.

[mkleczek]

So if the URI is slow then we will prefer querying the schema cache.

Won't this will always be the case when doing HTTP requests? It seems like this feature will only make sense when it's obtained from a file since that'd be faster.

Well... given performance issues with schema cache queries we've seen reported, I wouldn't be so sure.

HTTP is needed because:

1. In k8s environment it is rare to have a shared filesystem to read the file from.
2. Since we provide schema cache dump from admin server it is very convenient for blue/green deployments - just configure each to point to the other one's admin endpoint and you have schema cache pre-loading for free.

Besides: why would HTTP be slow? The most important thing here is not the access protocol but the fact that we serve the dump from memory. I am sure establishing database session and then querying the catalog is going to be much slower than HTTP GET from localhost.
I am pretty sure HTTP is going to be faster than reading from a materialized view as well.

But really we don't want to decide up front which method is preferred. We need to query the database anyway because pre-loaded dump might be stale. So why wait if we are going to do it anyway? It is better to race - that way we are going to have startup as fast as possible - we really don't care which method wins. We only need to make sure the database result is not overwritten.

There is also another angle: we want to deal with thundering herd issue when querying the db. In clustered environments HTTP gives us a natural way to load balance initial schema cache load among existing PostgREST instances. Once we have #4643 or similar it will become even more important as it might affect startup time.

[steve-chavez]

Good points. So we're always going to race here to get the fastest possible startup, got it. We should make this clear on the docs.

[steve-chavez]

There is also another angle: we want to deal with thundering herd issue when querying the db. In clustered environments HTTP gives us a natural way to load balance initial schema cache load among existing PostgREST instances.

@mkleczek I'm still not seeing why is racing preferable to opting for only a schema-cache-uri load.

The Listener should still be there to tell the instance when the cache is stale, so it requests it again from schema-cache-uri.

If we have a mode for only requiring schema-cache-uri to start, then I believe that should solve #4643 too. The replica instances would make a request to a master one.

---

Considering #5031 (comment), the schema cache is not all that we need for a postgREST to start right now; but maybe we can make it that way.

[mkleczek]

There is also another angle: we want to deal with thundering herd issue when querying the db. In clustered environments HTTP gives us a natural way to load balance initial schema cache load among existing PostgREST instances.

@mkleczek I'm still not seeing why is racing preferable to opting for only a schema-cache-uri load.

The Listener should still be there to tell the instance when the cache is stale, so it requests it again from schema-cache-uri.

If we have a mode for only requiring schema-cache-uri to start, then I believe that should solve #4643 too. The replica instances would make a request to a master one.

That is, I'm afraid, racy:

1. Replicas can issue HTTP GET to master PostgREST instance before it actually re-loaded the schema cache.
2. There might not be a "master" PostgREST instance - it might be dynamic and routing might be based on HTTP method.

Considering #5031 (comment), the schema cache is not all that we need for a postgREST to start right now; but maybe we can make it that way.

I think we should think about making it more organized into "database dependent" and "database independent" configuration - right now some aspects are mixed (ie. config/role settings/schema cache).

[steve-chavez]

Replicas can issue HTTP GET to master PostgREST instance before it actually re-loaded the schema cache.

@mkleczek But we could put a "version" on the schema cache so if the replica requests a new cache and it's the same version then it would retry. (we discussed a "version" previously in #4270 (comment)).

[mkleczek]

Replicas can issue HTTP GET to master PostgREST instance before it actually re-loaded the schema cache.

@mkleczek But we could put a "version" on the schema cache so if the replica requests a new cache and it's the same version then it would retry. (we discussed a "version" previously in #4270 (comment)).

Adding schema versioning that requires parsing a function body and then CREATE OR REPLACE FUNCTION with a new function body in an event trigger is waaaay to brittle and complex for my taste, to be honest.
And the gain is not substantial while has quite a big downside: it slows down start-up in case of master instance being unresponsive (eg. due to temporary network conditions).

Besides - there is a chicken and egg problem here: you want to load schema cache without accessing the database. How would you know the schema version number in this case?

[steve-chavez]

Since this is about fast startups, I was wondering if you have considered the in-db config + role settings queries, which if used are also required for startup (done prior to the schema cache):

postgrest/src/PostgREST/Config/Database.hs

Lines 96 to 134 in 6191243

	queryDbSettings :: Maybe Text -> Session [(Text, Text)]
	queryDbSettings preConfFunc =
	SQL.transactionNoRetry SQL.ReadCommitted SQL.Read $ SQL.statement dbSettingsNames $ SQL.Statement sql (arrayParam HE.text) decodeSettings True
	where
	sql = encodeUtf8 [trimming|
	WITH
	role_setting AS (
	SELECT setdatabase as database,
	unnest(setconfig) as setting
	FROM pg_catalog.pg_db_role_setting
	WHERE setrole = quote_ident(CURRENT_USER)::regrole::oid
	AND setdatabase IN (0, (SELECT oid FROM pg_catalog.pg_database WHERE datname = CURRENT_CATALOG))
	),
	kv_settings AS (
	SELECT database,
	substr(setting, 1, strpos(setting, '=') - 1) as k,
	substr(setting, strpos(setting, '=') + 1) as v
	FROM role_setting
	${preConfigF}
	)
	SELECT DISTINCT ON (key)
	replace(k, '${prefix}', '') AS key,
	v AS value
	FROM kv_settings
	WHERE k = ANY($$1) AND v IS NOT NULL
	ORDER BY key, database DESC NULLS LAST;
	|]
	preConfigF = case preConfFunc of
	Nothing -> mempty
	Just func -> [trimming|
	UNION
	SELECT
	null as database,
	x as k,
	current_setting(x, true) as v
	FROM unnest($$1) x
	JOIN ${func}() _ ON TRUE
	|]::Text
	decodeSettings = HD.rowList $ (,) <$> column HD.text <*> column HD.text

postgrest/src/PostgREST/Config/Database.hs

Lines 136 to 185 in 6191243

	queryRoleSettings :: PgVersion -> Session (RoleSettings, RoleIsolationLvl)
	queryRoleSettings pgVer =
	SQL.transactionNoRetry SQL.ReadCommitted SQL.Read $ SQL.statement mempty $ SQL.Statement sql HE.noParams (processRows <$> rows) True
	where
	sql = encodeUtf8 [trimming|
	with
	role_setting as (
	select r.rolname, unnest(r.rolconfig) as setting
	from pg_auth_members m
	join pg_roles r on r.oid = m.roleid
	where member = quote_ident(current_user)::regrole::oid
	),
	kv_settings AS (
	SELECT
	rolname,
	substr(setting, 1, strpos(setting, '=') - 1) as key,
	substr(setting, strpos(setting, '=') + 1) as value
	FROM role_setting
	),
	iso_setting AS (
	SELECT rolname, value
	FROM kv_settings
	WHERE key = 'default_transaction_isolation'
	)
	select
	kv.rolname,
	i.value as iso_lvl,
	coalesce(array_agg(row(kv.key, kv.value)) filter (where key <> 'default_transaction_isolation'), '{}') as role_settings
	from kv_settings kv
	join pg_settings ps on ps.name = kv.key and (ps.context = 'user' ${hasParameterPrivilege})
	left join iso_setting i on i.rolname = kv.rolname
	group by kv.rolname, i.value;
	|]
	hasParameterPrivilege
	| pgVer >= pgVersion150 = "or has_parameter_privilege(quote_ident(current_user)::regrole::oid, ps.name, 'set')"
	| otherwise = ""
	processRows :: [(Text, Maybe Text, [(Text, Text)])] -> (RoleSettings, RoleIsolationLvl)
	processRows rs =
	let
	rowsWRoleSettings = [ (x, z) | (x, _, z) <- rs ]
	rowsWIsolation = [ (x, y) | (x, Just y, _) <- rs ]
	in
	( HM.fromList $ bimap encodeUtf8 (HM.fromList . ((encodeUtf8 *** encodeUtf8) <$>)) <$> rowsWRoleSettings
	, HM.fromList $ (encodeUtf8 *** toIsolationLevel) <$> rowsWIsolation
	)
	rows :: HD.Result [(Text, Maybe Text, [(Text, Text)])]
	rows = HD.rowList $ (,,) <$> column HD.text <*> nullableColumn HD.text <*> compositeArrayColumn ((,) <$> compositeField HD.text <*> compositeField HD.text)

The in-db configs are not cached but the role settings are cached here:

postgrest/src/PostgREST/Config.hs

Lines 128 to 129 in 6191243

	, configRoleSettings :: RoleSettings
	, configRoleIsoLvl :: RoleIsolationLvl

When adding those I wondered if they should go into schema cache or not, I decided to make them separate to save some work on filtering them for the openAPI output. But with this feature, it looks we should make them part of the schema cache to really get the faster startup?

[steve-chavez]

in-db config + role settings queries

Actually I don't think the "in-db config" can be put inside the schema cache because it depends on it (db-schemas, db-extra-search-path) but the role settings could be put inside the schema cache to save some time.

[wolfgangwalther]

Gave this a high-level readthrough. I wonder what happens when you try to load a schema cache dump from a different PostgREST version, which has made changes to the schema cache structure?

Some of these might just fail the JSON parsing. But others might be more subtle and not cause that failure.

I think we should put the current PostgREST version into the schema cache dump / response by the admin server and check it when loading the dump. If it doesn't match, reject to load the dump.

[mkleczek]

Gave this a high-level readthrough. I wonder what happens when you try to load a schema cache dump from a different PostgREST version, which has made changes to the schema cache structure?

Some of these might just fail the JSON parsing. But others might be more subtle and not cause that failure.

I think we should put the current PostgREST version into the schema cache dump / response by the admin server and check it when loading the dump. If it doesn't match, reject to load the dump.

Note that cross-version schema cache dump read is one of the main use cases: zero-downtime upgrades.

So while understanding the concern, I think we should leave it as is: the risk is minuscule IMO:

• while we do not ensure schema cache dump compatibility, in practice breaking changes are rare. Changes that don't break parsing but affect semantics: even more so
• the time window of having wrong results is short: only during startup until schema cache is loaded from the database

I guess what we can do is making sure parsing of schema cache dump is as strict as possible to minimize the problem you describe.

[wolfgangwalther]

Note that cross-version schema cache dump read is one of the main use cases: zero-downtime upgrades.

I don't understand how that relates. For zero-downtime upgrades, aka #5112, I can just read the schema cache from the database, I don't need to read a dump, right?

[wolfgangwalther]

This is a little bit of a problem. Today loading the schema cache re-loads config and role settings. This PR decouples them so during startup we might start handling requests with pre-loaded schema cache but stale role settings and config.

I think we should:

• move the role settings to the schema cache
• create a "header" in the json dump of the schema cache that contains:
  • PostgREST version as mentioned earlier
  • the current configuration, or maybe a hash of all the values (considering secrets etc... - not sure)
• check that the header matches the current version + configuration when loading the dump, otherwise rejecting it

It only makes sense to load the schema cache dump when it was created with exactly the same configuration. If not, we need a new schema cache.

[mkleczek]

Note that cross-version schema cache dump read is one of the main use cases: zero-downtime upgrades.

I don't understand how that relates. For zero-downtime upgrades, aka #5112, I can just read the schema cache from the database, I don't need to read a dump, right?

Ideally, you run PostgREST like postgrest --schema-cache-uri=http://localhost:3001/schema-cache - ie. you point the new instance to load schema cache from old instance admin server for fast startup. #5112 makes it even more tempting because it allows running admin server with SO_REUSEPORT ensuring new instance admin server is not started before schema cache is loaded.
Granted: with two instances running simultaneously it is less pressing to have fast startup (because old instance waits for the new one to start) but it makes the restart process fast and hence less prone to errors.

I'd even consider having a switch that disables standard schema cache load on startup when schema-cache-uri is provided.

[mkleczek]

• check that the header matches the current version + configuration when loading the dump, otherwise rejecting it

This won't work because of in-db configuration. The whole point of loading from URI is to be able to start serving requests without connecting to the database to read any metadata. With in-db configuration you are always going to have different configuration before reading it from the database.

[wolfgangwalther]

Granted: with two instances running simultaneously it is less pressing to have fast startup (because old instance waits for the new one to start) but it makes the restart process fast and hence less prone to errors.

But by using a schema cache from a different version, we are going to be more prone to errors.

I think it's fine to either provide fast startup or zero-downtime - but we don't need to provide incredibly fast zero-downtime handovers.

Especially in the zero-downtime scenario, I'd value a higher chance of success much more than being a few seconds faster.

[mkleczek]

Granted: with two instances running simultaneously it is less pressing to have fast startup (because old instance waits for the new one to start) but it makes the restart process fast and hence less prone to errors.

But by using a schema cache from a different version, we are going to be more prone to errors.

I am not convinced: when was the last time you had a change in schema cache dump that would cause the errors you described (ie. parsing fine, results wrong)?

[wolfgangwalther]

This won't work because of in-db configuration. The whole point of loading from URI is to be able to start serving requests without connecting to the database to read any metadata. With in-db configuration you are always going to have different configuration before reading it from the database.

I think this should support both db-config enabled and disabled. When db-config is enabled, then serving requests without connection to the database to read any metadata is not the goal, no. PostgREST will have to read its configuration first, but can then proceed to quickly load the schema cache. Still a performance improvement.

If you want to serve requests without hitting the database before the actual request, you will just need to set db-config = false, that's all, no?

[wolfgangwalther]

I am not convinced: when was the last time you had a change in schema cache dump that would cause the errors you described (ie. parsing fine, results wrong)?

It doesn't matter when the last time was that this happend - I have no idea, I didn't have to pay attention to that. What matters is, that I don't want to pay attention to this in the future, when making changes to PostgREST. If we have to keep schema cache compatibility in mind, that's going to be really annoying for development.

[mkleczek]

I am not convinced: when was the last time you had a change in schema cache dump that would cause the errors you described (ie. parsing fine, results wrong)?

It doesn't matter when the last time was that this happend - I have no idea, I didn't have to pay attention to that. What matters is, that I don't want to pay attention to this in the future, when making changes to PostgREST. If we have to keep schema cache compatibility in mind, that's going to be really annoying for development.

Only in theory, but not in practice. Schema cache changes are so rare that it is practically non issue IMO. Schema cache changes not breaking strict parsing (eg. missing fields cause errors) but only semantics - not a chance :)

[mkleczek]

If you want to serve requests without hitting the database before the actual request, you will just need to set db-config = false, that's all, no?

Or load both schema cache and config from URI.

[wolfgangwalther]

Or load both schema cache and config from URI.

I think what you could do would be:

• Put the "pre-in-database" config into the schema cache header.
• Verify against that.
• Put the "post-in-database" config into the schema cache main part.
• Load this config as well.

This would combine both the verification phase and the ability to get a consistent config/schema snapshot from the database.

[mkleczek]

This would combine both the verification phase and the ability to get a consistent config/schema snapshot from the database.

If you have consistent config and schema cache snapshot, why would you need to verify anything?

[wolfgangwalther]

If you have consistent config and schema cache snapshot, why would you need to verify anything?

Because changes to the not-in-db-config should be reflected. For example when I manage exposed schemas via regular configuration and I start a new instance, explicitly removing one of the schemas - I sure don't want this instance to respond to requests for that old schema.

We're only optimizing the cache here, but we should make sure to respect explicit configuration changes the user makes.

[wolfgangwalther]

Only in theory, but not in practice. Schema cache changes are so rare that it is practically non issue IMO. Schema cache changes not breaking strict parsing (eg. missing fields cause errors) but only semantics - not a chance :)

One example would be the schema cache just having the same format, but including/excluding more list items somewhere. For example the schema cache queries were wrong and returned too many rows for, let's say, overloaded RPCs. The new version fixes this. We release a new version, claiming a fix. But PostgREST loads the old schema cache and will serve wrong responses from that. I'd rather have the new version start up properly and serve the correct responses from the start.

Edit: This is even more important when you disable regular schema cache load if an URI is provided.

[mkleczek]

If you have consistent config and schema cache snapshot, why would you need to verify anything?

Because changes to the not-in-db-config should be reflected. For example when I manage exposed schemas via regular configuration and I start a new instance, explicitly removing one of the schemas - I sure don't want this instance to respond to requests for that old schema.

We're only optimizing the cache here, but we should make sure to respect explicit configuration changes the user makes.

Hmm... if you want to change static configuration and restart - why add --schema-cache-uri to the new instance? You either:

• want to change configuration and restart - that would always mean loading schema cache from database (because config changed)
• want to just change configuration - don't restart, just send SIGUSR
• want to change the binary and restart - then versioning would prevent loading schema cache from URI

[wolfgangwalther]

Hmm... if you want to change static configuration and restart - why add --schema-cache-uri to the new instance? You either:

• want to change configuration and restart - that would always mean loading schema cache from database (because config changed)

• want to just change configuration - don't restart, just send SIGUSR

• want to change the binary and restart - then versioning would prevent loading schema cache from URI

I'm thinking about that AWS lambda example. I assume you'd just want to always start with --schema-cache-uri for that. As long as you didn't change the configuration in the meantime, you'll get really fast startup time to serve the request. Once you make a change to the config, the one request will be a bit slower, but the dump file will be written automatically with the other PR and on the next request you're fast again.

There is no decision-making involved in how to actually start and/or reload the process. This is hardcoded.

What am I missing?

[mkleczek]

I'm thinking about that AWS lambda example. I assume you'd just want to always start with --schema-cache-uri for that. As long as you didn't change the configuration in the meantime, you'll get really fast startup time to serve the request. Once you make a change to the config, the one request will be a bit slower, but the dump file will be written automatically with the other PR and on the next request you're fast again.

In this case it makes sense to dump both config and schema cache, load both of them from --schema-cache-uri at the very beginning, start serving requests and reload both config and schema cache later. That way you have both fast startup and you can change configuration.

[wolfgangwalther]

In this case it makes sense to dump both config and schema cache, load both of them from --schema-cache-uri at the very beginning

See #5031 (comment) for why I think this is wrong.

Also, there is another problem with both config validation and dumping/loading, that we didn't consider: LIBPQ environment variables are effectively a part of configuration, but we don't store their values anywhere. So changing any of these (for example by connecting to a different pg server!) would not fail verification and would also create a problem when loading old configuration from file and combining it with new LIBPQ environment variables...

[mkleczek]

In this case it makes sense to dump both config and schema cache, load both of them from --schema-cache-uri at the very beginning

See #5031 (comment) for why I think this is wrong.

I don’t see how - the new config will be used, the exact timing is not that important because either way there always going to be a time window between you changing configuration and PostgREST reflecting it in its responses.

Also, there is another problem with both config validation and dumping/loading, that we didn't consider: LIBPQ environment variables are effectively a part of configuration, but we don't store their values anywhere. So changing any of these (for example by connecting to a different pg server!) would not fail verification and would also create a problem when loading old configuration from file and combining it with new LIBPQ environment variables...

And that’s even more the reason why we should not do any validation and treat this feature as an advanced one that requires the user to understand the risks.

[mkleczek]

I'm thinking about that AWS lambda example. I assume you'd just want to always start with --schema-cache-uri for that. As long as you didn't change the configuration in the meantime, you'll get really fast startup time to serve the request. Once you make a change to the config, the one request will be a bit slower, but the dump file will be written automatically with the other PR and on the next request you're fast again.

There is no decision-making involved in how to actually start and/or reload the process. This is hardcoded.

What am I missing?

I thought about it a little more and I think this is a malformed scenario:

You have to think about lambda function shutdown/startup as "hibernate/wakeup". This PR idea is the same: schema cache dump is a "frozen state" that is used for wake-up - after restart the program is conceptually in the same state as before. Additionally it hooks into after-wakeup to refresh its state with any changes that might have happened in the environment while it was asleep - but that is conceptually "a bonus" that makes user's life easier (ie. no need for SIGUSR after restart).

You cannot rely on lambda shutdown/startup to reload configuration anyway - because shutdown might never happen. So you must take explicit action to reload configuration (and it is a good question what would be the right trigger to reload config in lambda environment as neither SIGUSR nor NOTIFY pgrst are reliable, but this question is for another PR).

Ideally the "hibernate/wakeup" idea should cover both config and schema cache. Unfortunately, to make that happen, PostgREST changes would be much more substantial: configuration right now is read very early in the startup sequence - even before deciding if it is a cli invocation (ie. dump schema / dump config) or actual application startup. That would need to be split: for cli invocations config would need to be read as today but for application startup - it would need to be handled dynamically in AppState.init like this PR schema cache load.

I would leave this PR as is and not go into the consistency check direction at the moment:

1. IMO the better direction is to implement "consistent snapshot" taking/restoring functionality to avoid mismatch in config and schema cache instead.
2. With all the discussed caveats, this PR solves important problems (ie. scale-to-zero and k8s horizontal scaling) that won't really gain anything from the consistency checks.
3. Consistency checks is a can of worms in itself - requires answering not trivial questions about the algorithm and the scope of the data to check.

I've created #5129 to discuss the direction.