pg_receivewal on encrypted wal

[JamesInform]

Description

pg_receivewal is not working as per your documentation, which by the way is a pleasure to read.

Are there any plans to introduce something like pg_tde_receivewal? It would be great to use replication slots for wal archiving instead of archive commands.

Suggested solution

No response

Additional context

No response

[AndersAstrand]

Hello!

We weren't sure what the usecases for a pg_tde enabled pg_receivewal would be, so we didn't prioritize it for the first releases.

But here you are asking for it! What is your usecase? We can probably bump it up the priority list if there is need for it.

[JamesInform]

We are doing incremental backups using pg_basebackup.
In parallel we archive wal using pg_receivewal on a replication slot.
This enables easy point in time recovery capabilities.

Unencrypted wal files introduce the risk of sensitive data being leaked, even if it is only for a limited time.
Encrypted wal fixes this and also secures wals recevied via pg_receivewal.

Replication slots ensure retaining WAL segments exactly until all they have been received on the backup, avoiding premature removal that occurs in archive_command setups where WAL can be archived and deleted independently of replica status. Unlike archive_command, which relies on external scripting and/or shared storage that introduces failure points and polling delays, slots provide direct, connection-aware management without needing WAL archives for real-time wal shipping.

When used in replication, slots also protect against recovery conflicts by coordinating vacuuming across replicas, a benefit not inherent in basic WAL shipping.

So, well ... it would be nice to have a piece like pg_tde_receivewal at hand.

Thanks for your great effort in bringing tde to "community" PostgreSQL.

Cheers,
James

[AndersAstrand]

Do I understand correctly that what you would expect for your usecase is for pg_tde_receivewal to get the encrypted wal, and then re-encrypt it with keys available on the receiving host (similarly to how we handle pg_tde_basebackup streaming mode) before storing it?

Or do you expect it to write the decrypted WAL?

Thank you for your interest in pg_tde!

EDIT: Sorry. I was mistaken, obviously it will receive the unencrypted WAL as that's what's sent for replication.

We don't have a way to coordinate encryption keys between hosts, so the receiver would have to generate it's own keys (that can be based on the same principal key as the sender of course). Similar to how we handle streaming mode for pg_tde_basebackup.

With pg_tde_basebackup it's clear how to distribute these newly generated keys as they will be part of the backup, but with pg_tde_receivewal it's not as clear how someone wanting to decrypt the stored WAL would get hold of them.

[JamesInform]

I haven't tested the behaviour of pg_receivewal if wal is encrypted.
Just read your documentation where you stated that pg_receivewal is currently an unsupported wal tools for encrypted wal.
I concluded that pg_receivewal is no longer usable if wal is encrypted.

Am I wrong with it? Is pg_receivewal still working on encrypted wal but only storing the received wal unencrypted? The latter will be fine by us. An addtional option to encrypt the received wal directly with pg_tde_receivewal would make things smoother.

[AndersAstrand]

I believe pg_receivewal should work for storing the unencrypted WAL in principle, but I would need to test to make sure it doesn't try to actually understand the WAL records its writing. There might be some problem with the custom WAL records pg_tde uses if it does.

[JamesInform]

Would be fine if you can test.

[AndersAstrand]

I did some simple testing and everything seems to work as expected. I also talked to the team and no-one seemed to see any reason why it wouldn't work. Let us know if you run into any issues!

[JamesInform]

Thanks.
I will test on my side and give you feedback.