libpqxx 8.0.1: The Walk of Shame

This always happens. You put out a shiny new release. A major one. It's got new features, overhauled design, cleanups... it's perfect.

If you're lucky you get to enjoy a few days of pride in your spotless work. And then, as more people get the new version, they start finding things. And often there's something that makes you slap your forehead: why didn't I see that?

In this case, 3 things happened:

1. @tiagomalheiro-dtx found a buffer overrun. In practice it's usually just an assertion failure, but on NDEBUG builds it could end up writing a zero byte past the end of a buffer. Good thing I wrote that assertion.
2. @cacharle ran into a build problem with Clang on Windows, and filed a very helpful ticket. That's fixed now.
3. An update to clang-tidy produced loads and loads of new nitpicks about the code. Why can't this happen just before the release?

That third change may actually be pretty intrusive. Lots more constructors and type conversions are marked explicit now. So it's possible that you'll see compilation errors along the lines of "no known conversion from <type A> to <type B>." Most likely when you're calling a function and passing an argument that's of a different type than the actual parameter that the function expects. Previously an implicit conversion "promoted" the argument type to the parameter type, but now you have to command it explicitly.

The safety rules were pretty insistent, so I applied this change fairly broadly. Better now than after everybody's already used to the changes in 8.0, I figured. I left the implicit conversion in some cases where I felt it's completely reasonable.

I'm really sorry for the source compatibility break. It's hardly proper semantic versioning. But it was either that or leave these issues lie until 9.0, when they'd hurt more!

libpqxx 8.0.0

libpqxx 8 is ready for you!

Libpqxx 8 is the library's biggest release ever.

It's been a long trek. On the one hand it was a parallel journey to the
adoption of C++20. We're not relying on all the new C++20 features, but
concepts, std::source_location, spans, and ranges are integral parts of the
design. Also I was hit by a health problem that reduced the amount of work I
could do, and often made it hard for me to trust my own judgment. Luckily that
happened after most of the important design work was done, or you would not see
this release today.

On the other hand, I've also had help: community members reported bugs and
submitted patches, some of very high quality. And people helped each other
out, which is awesome to see. A Copilot subscription on GitHub helped with an
extra layer of code reviews. It found a lot of typos in comments, but also
some genuine and sometimes subtle mistakes. (I didn't let it write actual code
though; it gets too many things wrong, some of which I would probably not have
caught in review. Kernighan's Law applies.) Other large language models, in
particular Claude, helped a lot in solving configuration problems.

Perhaps most of all, my wonderful former employer Eonics brought together a
Mastermind group where a bunch of us discuss our hobby projects, learn, and
share questions and ideas. The group often came up with unexpected and helpful
approaches for my problems. If you have a hobby project, I wholeheartedly
recommend looking for a mastermind group near you! And if you speak Dutch,
have a look at the Eonics website. You'll find podcasts,
hack nights, interviews, and much else to make IT work fun.

Thank you

Several people helped. I'm probably missing somebody, but I wanted to thank...

• @tt4g for helping lots of people with their
  questions, often before I even see them.
• Klaus Silveira @klaussilveira), our
  first donor-level sponsor.
• Hendrik M. @Tosenaeus, who also sponsors us.
• Eonics, CoffeeSprout,
  ComPosiX, and the whole Eonics Mastermind team.
• Kirit Sælensminde, for discussing features, ideas, and
  C++ language changes.
• J.H.M. "Ray" Dassen, much missed, who did a lot to help get libpqxx off the
  ground, all the way back.
• All those who submitted patches, reported problems, or asked for
  improvements.

What's changed?

Just about everything has changed at least a little bit.

349 files changed, 28826 insertions(+), 24686 deletions(-)

Most "standard" code using libpqxx will still work without changes, but often
you'll find that there are now better ways to do things. Functions and classes
that were deprecated three years ago are now gone. Some other ones are now
deprecated, and will be gone a few years from now.

The API has become better at helping you keep your code memory-safe and
verifiable by static code analysis. There are fewer raw, C-style pointers.
At the same time some things have become more efficient through the use of
views, especially string_view, and internal simplifications. Ranges and
spans have made things clearer and simpler. Concepts have made some things
more flexible.

More details follow. See the NEWS file for a more complete list with less
detail, or UPGRADING.md for advice on how to deal with breaking changes.

Things that can break with old code

If you're just using 8.0 as a drop-in replacement for an older version in an
existing application, this may be the only useful section for you.

C++20 is now the minimum. As per the above it's fine if some C++20
features don't work for you yet, but concepts definitely need to work, and
you'll need a bunch of the new C++20 standard library features.

PostgreSQL 11 is now the absolute minimum. You won't be able to connect to
older servers. The exact version is a fairly arbitrary choice. The oldest
version supported upstream is 14, so hopefully 11 is liberal enough.

Exceptions have changed. The whole class hierarchy is different, so if
you do any detailed catching you'll have to review that. On the bright side,
libpqxx exceptions are now all derived from pqxx::failure so you may not need
as many catch blocks as before. Your handler can figure out much of what it
needs by calling the exception's member functions, rather than by knowing its
type.

Rows and fields have changed. The old row and field classes used to
keep a result object alive in memory, even if you destroyed the result
object. Now, most of the time you'll get a row_ref or field_ref which does
not do that. Be careful with lifetimes, but it rarely becomes an issue, and
the new classes are simpler and more efficient.

Result & row iterators have changed. Indexing an iterator now finally does
what the C++ standard says, even though it's probably less intuitive or useful.
If you previously did i[n], you'd now do (*i)[n].

Comparing results, rows, or fields now compares identity, not contents. I
doubt anyone ever used what these == operators did.

pqxx::bytes is now an alias for std::vector<std::byte>, not a
std::basic_string<std::byte>. It's not clear that the fine print in the
standard ever quite allowed the string definition. Some code will require
small changes at compile time. If you ever used the c_str() member function,
now you'll use data(). More appropriate for binary data.

The string conversion API has changed. This is the extensible API that
converts between C++-side objects and their SQL text formats. If you wrote
your own to support additional types, those should still work; but the newer
API is generally easier, safer, and more efficient if you want to optimise.

A lot of strings no longer have or need a terminating zero. This includes
the string conversion API.

In some situations, a params will require text encoding information. In
those situations, pass your connection, transaction, or encoding group as the
first argument to the params constructor.

No longer supports std::filesystem. Nor does the build try to figure out
whether it needs to link any additional libraries to make it work.

A zview can no longer contain a null pointer. This used to be valid for
empty strings, but it no longer is.

More classes are now final.

The scripts in tools/ have been revised. Shell scripts now have a ".sh"
suffix to their name. Some old tools are gone: rmlo,pqxxthreadsafety,
test_all.py.

The JOHAB encoding is not supported. As far as we can tell, there is no
single standard for how this encoding was supposed to work, and support on the
server side never fully worked anyway. It will be removed from PostgreSQL.

Some deprecated items are now gone:

• binarystring (use blob instead).
• connection_base (use just connection, it's the same thing).
• encrypt_password() (use the equivalents in connection).
• dynamic_params (use params instead).
• transaction_base::unesc_raw() (use unesc_bin() instead).
• transaction_base::quote_raw() (use quote(bytes_view) instead).
• Some field constructors.
• Row slicing. Has anyone ever used this?

Querying continues to evolve

Querying data has been growing simpler and more regular for a while now. But
I'll summarise how the 8.0 way of querying differs from the old ways.

Passing statement parameters: You no longer call separate exec(),
exec_params(), or exec_prepared(). Instead, you just call exec(), and
in addition to the query, pass a params object if the statement needs
parameters. If the query is a prepared statement rather than SQL text, you
use its name, wrapped in a pqxx::prepped object:

tx.exec(
    pqxx::prepped{"my_query"},
    pqxx::params{1, 23, "param");

Executing, querying, streaming: These are the three models for executing a
query, and where possible they look similar. "Executing" a query is the
classic exec() function, and it returns a pqxx::result. "Querying" works
very similar, but you pass template arguments to state which types of result
columns you expect, and then you iterate over the return value. "Streaming"
looks a lot like querying but is optimised for large data sets: it's slower for
small numbers of rows but faster for large numbers. (Sadly a streaming query
can not take parameters.)

Expected result sizes: If you want to make sure that a query returns a
specific number of rows, you no longer use exec0(), exec1(), or exec_n();
you just use exec() and then call a member function on the result such as
no_rows(), one_row(), or expected_rows(). There are similar functions
for the number of columns.

Various new features

This is not a complete list! There's more than I can fit in here. You'll
have to explore the documentation or the code to find them all.

Here are some major ones:

Example code. There is now an examples/ directory containing source for
various toy applications. There will be more in the future. Unlike the tests,
these examples are entirely meant for you, to illustrate what you can do with
libpqxx. But unlike the documentation, they get compiled and run along with
the test, so they won't fall out of date.

Source locations & stack traces: A libpqxx exception can now tell you the
associated std::source_location and, if your compiler supports it, a full
std::stacktrace.

Easier SQL arrays: Single-dimensional SQL arrays are now super easy, barely
an inconvenience. You can now...

libpqxx 8.0.0 release candidate 5

Yes, it's another release candidate! I promise, it's not that there were bugs or anything, just that things kept getting better. The problem with maintaining backwards compatibility across minor releases is that a major release is the one opportunity for all sorts of structural changes. So any big improvements have to go in now or wait, possibly for years.

Unless any major problems show up, this should be the last release candidate before the big 8.0 release. Exciting times! I can't do a lot in a day at the moment, so any release is a big deal for me.

Here are the main changes:

You can now combine connection string & option pairs on connections! (#1133)

This feature was contributed by @serpent7776. It lets you pass both a connection string and a series of individual parameters. Once we had this, it became possible to simplify the web of constructors.

Various zview changes:

• You can no longer construct a zview from a null pointer.
• The zview class is now final.
• There is now an implicit conversion to a C-style string (char const *).

The libpqxx exception classes now have a C++26 std::stacktrace, if available.

This means your application can log a full traceback when libpqxx throws an exception, just like in other languages.

Be aware though that this exists only for exceptions that libpqxx throws. If some standard library function throws an exception of its own from within libpqxx code, that is still the standard exception and there's no traceback.

The "config-*" headers in the build are now simpler. (#1152)

You may never even notice this but oh, it got so much better!

• There is now only one of those headers: config-compiler.h.
• The CMake build now filters out unneeded macros from that header, like the autotools build does.
• There is no more config-autotools.h.

It looks like we worked around a regression in Microsoft Visual Studio 2026. (#1160)

It's strange: MSVC 2026 failed to accept a run-of-the-mill concept-based partial template specialisation that MSVC 2022, gcc, and clang all accept. And it's not a new language rule or anything: the MSVC error actually looks to have existed for much longer.

Deprecated extract_version.sh.

Use extract_version.py instead. It has a few extra features, such as the ability to extract a patch-level version number, and a numeric patch-level version number which evaluates to -1 for release candidates and such.

Rewrote header/binary version compatibility check.

There is a bit of code in libpqxx that checks that you're not linking your application binary with a libpqxx version for which it was not built. In most cases when you mismatch versions, the library will probably fail to link at all and this check won't even run. But there are some scenarios where this may detect problems early, for example if they're the same major and minor version but your libpqxx binary has a lower patch level than the one for which your application was built.

For stream(), a type must be convertible and move-constructible. (#1168)

Actually this requirement already existed, at least in C++20 builds, but now it is an explicit requirement in the code, enforced through a concept. Yay C++20!

The build defaults to C++20 minimum.

In recent versions, most of you needed to tell your compilers which C++ version to use when building libpqxx. The tooling wasn't quite there to automate this the way I wanted it, but now it is. You can specify a specific C++ version, but if you don't, libpqxx will automatically compile with the minimum C++ version required. For libpqxx 8.0, that means C++20.

Please try out this new release candidate, and let me know if you run into any trouble!

Actually, also let me know if you don't run into any trouble. 😆 It'll help me release more confidently, and perhaps sooner.

libpqxx 8.0.0 release candidate 4

The useful, now-or-virtually-never changes for 8.0 just keep coming. Here's another release candidate that you may want to try.

Changes from release candidate 3:

• No longer trying to support std::filesystem.
• composite_into_buf() now returns a zview.
• Optionally pass encoding_group into params (or connection or transaction).
• Export helper to render a std::source_location as text.
• More IDE-friendly text format for source locations.
• Enable tests to run even on ASCII databases (just skip some encoding tests).
• Document: result::affected_rows() also works for SELECT. (#1139)
• Use C++26 reflection, if available, to figure out types' names.
• Rename pqxx::out_of_memory to pqxx::server_out_of_memory for clarity.
• pqxx::protocol_violation is now an sql_error, not a broken_connection.
• All libpqxx exceptions now derive from pqxx::failure.
• pqxx::failure now derives straight from std::exception, not std::runtime_error.
• Exceptions now know their name, whether they break the connection, etc.
• All additional properties on libpqxx exceptions are available in pqxx::failure.
• Require PostgreSQL 11 server or better.

libpqxx 7.10.5: minor polish before switching mainline

This is a tiny update. There is almost certainly no urgency for you to upgrade.

I'm just cleaning out some pending changes before I merge the 8.0 release branch. Once I do that, mainline development will officially be on the coming 8.0 release.

Changes in 7.10.5:

• Downgraded Doxygen; new version is leaving some important types out of our documentation. 😦
• Update a deprecated call in README.md. (#1111)
• Fix YAML lint checks, but disable them for now. The lint is all fixed up in the 8.0 branch.

(To those who may have noticed: I withdrew this release and then re-created it. Health problems had me a little confused.)

libpqxx 8.0.0 release candidate 3

Here's another release candidate for libpqxx 8.0. With your feedback, contributions, and sometimes in-depth research, we have a lot of further improvements compared to the previous release candidate!

One big change is that you can now convert directly between simple one-dimensional SQL arrays and C++ containers such as std::vector<int> or std::list<std::string> etc. You can stream these values,read them from a pqxx::result using a field's as<...>() member function, and pass them as parameters.

(The conversion to string was already there but it wasn't working because of a small omission.)

Other important changes:

• You can now use std::string_view in more places that previously required pqxx::zview.
• There's an examples/ directory with working, compiling example code.
• We no longer support the JOHAB encoding. It's a deprecated encoding, and it doesn't look like the backend support was ever complete. Also... which variant of JOHAB? There's several.
• In general, the text handling got smaller, simpler, and more efficient.
• Compiler errors should be slightly clearer when you're writing your own string conversions and struggling to meet the API.
• Several pqxx::result member functions now take pqxx::sl (i.e. std::source_location) rather than pqxx::conversion_context.
• MSVC 2022 didn't actually support multi-dimensional operator[] in C++23 mode. The build now detects that problem.

libpqxx 8.0.0 release candidate 2

Here's a new release candidate for a possible 8.0 release for you to try out. If you run into any problems with it, please bring it up on Github so we can improve things.

Changes from RC1:

• Add begin() const, end() const, etc. to pqxx::array.
• Cursor close() is no longer noexcept. Destructor still is though.
• Fix to make pqxx::array streamable.
• pqxx::zview now asserts the terminating zero.

There are also a few tweaks to the internals, particularly in encoding support and in streaming. Things got a little bit simpler, and hopefully a bit faster as well. I've added a few assertions though, so please let me know if any of those fail.

You can follow the work and see the diffs in PR #914.

libpqxx 7.10.4

A small bug-fix release. My focus is on the upcoming 8.0 release; also expect a fresh release candidate for that.

Changes:

• Logic for controlling non-blocking mode was inverted (non-Windows). (#1057)
• Pass SQL error code when throwing pqxx::insufficient_privilege. (#1077)
• Deprecated the unnamed prepared statement.

Release candidate: 8.0.0-rc1

About this pre-release

I've been suffering from a health problem that makes it hard for me to do any serious brain work. The 8.0 code has been close to a state of completion for months now, but I didn't trust myself to attempt a release.

Friends suggested a Release Candidate, and... here it is!

This is a big change. Some of your code may need changing:

• Items that have been deprecated for years are now gone.
• Other items have become deprecated.
• There are now nicer ways of doing some very basic things.

Before I go into the details though, let me thank our sponsors: @Laro88, CoffeeSprout, @klaussilveira, and @Tosenaeus. It is wonderful to see all the work truly appreciated.

You too can sponsor the work on libpqxx, by the way: monthly or one-off contributions on Github, or through Buy Me A Coffee.

That said, let's go over what's changed in this release.

Changes in 8.0

Many things have changed. Most code written against 7.x will still work, but there may now be nicer ways of doing things. Some of your code may need updating, and I'm sorry about that! But I hope you'll be happy overall.

Let's go through the main things.

C++20

You will need at least C++20 to compile libpqxx. Support does not need to be absolutely complete, but the basics must be there: concepts, various utility functions in the standard library, the language's new object lifetime rules, and so on.

Thanks to std::span, there are now very few raw pointers in the API (or even internally). This makes libpqxx not only safer, but also easier to prove safe. It is important to be able to run static analysis tools that may detect bugs and vulnerabilities. But, in the C++ tradition, it shouldn't cost us anything in terms of speed.

Retiring deprecated items

Some types and functions were already deprecated and are now gone:

• binarystring. Use blob instead.
• connection_base was an alias. Use just regular connection.
• encrypt_password(). Use the equivalent member functions in connection.
• dynamic_params. Use params.
• stream_from. Use transaction_base::stream() instead.
• Some stream_to constructors. Use static "factory" member functions.
• transaction_base::unesc_raw(). Use unesc_bin().
• transaction_base::quote_raw(). Use quote(), passing a bytes_view.
• Some field constructors.
• Result row splicing. I don't think anyone ever used it.

Result iterators

Lifetimes: result and row iterators no longer reference-count their result object. In libpqxx 7, a result object's data would stay alive in memory for as long as you had an iterator referring to it. It seemed like a good idea once, many years ago, but it made iteration staggeringly inefficient.

So, that's changed now. Just like any other iterator in C++, result and row iterators no longer keep your container alive. It's your own problem to ensure that. However result objects are still reference-counted smart pointers to the underlying data, so copying them is relatively cheap.

Semantics: Iterators for the result and row classes have never fully complied with the standard requirements for iterators.

There was a good reason for this: I wanted these iterators to be convenient and work a lot like C pointers. I wanted you to be able to dereference them as if a result object were just an array of pointers to arrays.

But it really started getting in the way. Much as I hate it, the standard requirements say that for any iterator i, i[n] should mean *(i + n). In libpqxx, if you had a result iterator i, i[n] meant "field n in the row to which i points." Really convenient, but not compliant with the standard.

So, that is no longer the case. If you want "field n in the row to which i points," say (*i)[n] or i->at(n).

By the way, these changes mean that result and row iterators are now proper, standard random-access iterators. Some algorithms from the standard library may magically become more efficient when they detect this.

Comparing results, rows, and fields

When you compare two result, row, or field objects with == or !=, it now only checks whether they refer to (the same row/field in) the same underlying data object. It does not look at the data inside.

These comparisons were meant to be helpful but they were never very well defined. And if you don't know exactly what you're getting, why would you want to invest the compute time?

Row and field references

The row and field classes were cumbersome, inefficient, and hopelessly intertwined with iterators.

To avoid all that, use row_ref instead of row and field_ref instead of field. These assume that you keep the original result around, and in a stable location in memory. Unlike row and field, they do not keep the
underlying data object alive through reference-counting. But neither do any of
the standard C++ containers, so I hope you'll find this intuitive. It's certainly more efficient.

The indexing operations now return row_ref and field_ref instead of row and field respectively. It probably won't affect your code, but you're running static analysis and instrumented builds to check for these things, right?

Binary data

As the documentation predicted, the alias pqxx::bytes has changed to stand for std::vector<std::byte>. It used to be std::basic_string<std::byte>. And pqxx::bytes_view is now an alias for std::span<std::byte>.

This may require changes to your code. The APIs for std::basic_string and std::vector differ, perhaps in more places than they should. Do not read the data using the c_str() member function; use data() instead.

Hate to do this to you. However there were real problems with using std::basic_string the way we did. The basic_string template wasn't built for binary data, and there was no guarantee that it would work with any given compiler. Even where it did, we had to work around differences between compilers and compiler versions. That's not healthy.

But there's also good news! Thanks to C++20's Concepts, most functions that previously only accepted a pqxx::bytes argument will now accept just about any block of binary data, such as std::array<std::byte> or a C-style array.

String conversion API

This is a whole chapter in itself. If you were specialising pqxx::type_name or pqxx::string_traits, things just got simpler, safer, and more efficient. But there's also a change in lifetime rules. This can be important for writing safe and correct code, so read on!

Type names

Let's get the easiest part out of the way first: type_name is now deprecated. Instead of specialising type_name, you now specialise a function called name_type(). It returns a std::string_view, so the underlying data can be a string literal instead of a std::string. Some static analysis tools would report false positives about static deallocation of the type_name strings.

Defining a string conversion

Then there's pqxx::string_traits. Several changes here. If you were defining your own conversions to/from SQL strings, the 8.x API is...

1. leaner, losing several members but mainly into_string().
2. safer, using views and spans rather than raw C-style pointers.
3. simpler, dropping the terminating zero at the end of various strings.
4. friendlier, accepting std::source_location for better error reporting.
5. richer, capable of dealing with different text encodings.
6. faster, because of the lifetime rule changes (described below).

Your existing conversions may still work without changes, but that's only thanks to some specific compatibility shims. These will go away in libpqxx 9 at the latest, so I urge you to update your code.

Lifetime rule changes

In libpqxx 7.x, when you converted a C++ value to an SQL string or vice versa, you could count on having a valid output for at least as long as you kept the object in which you stored it. The details depend on the type, e.g. pqxx::string_traits<bool>::to_string() returns a string constant, either "true" or "false".

We're tightening that up in 8.x. Now, the output remains valid for at least as long as you keep the object in which you stored it, and also keep the original in place. This streamlines some trivial conversions. For example, converting a std::string_view or a C-style string to SQL is now just a matter of returning a view on that same data. It's just the same data in both C++ and SQL.

It also enables some conversions that weren't possible before. You can now convert an SQL string to a std::string_view or a C-style string pointer, because the conversion is allowed to refer to its input data.

Most code won't need to care about this change. A calling function is usually either done very quickly with its converted value, or it immediately arranges for more permanent storage. If you call pqxx::to_string() for example, you get a std::string containing the SQL string. All that changes in that case is the conversion process skipping an internal copy step, making it a bit faster.

Using conversions

There is now a richer, more complete set of global conversion functions. This means you can convert C++ values to SQL strings without calling their pqxx::string_traits functions yourself.

The options are, from easiest to most efficient:

• pqxx::to_string() returns the SQL value as a std::string.
• pqxx::into_buf() renders the SQL v...

7.10.3: Fix MSVC C++17 build

This is a quick update to 7.10.2. It fixes some problems with building libpqxx as C++17 in Microsoft Visual Studio.

The main changes however are what you already got with 7.10.3:

• Fixes to CMake & pkgconfig setup.
• Better handling of weirdness in connection strings.
• Can't pass parameters to stream().
• row::as_tuple() is now public.
• Fixed potential garbage at end of type names in error messages.
• Build fix on macOS related to std::free().
• Fix to workaround when compiler does not support to_chars() for floating point.
• Avoid crash in affected_rows() for empty result.
• Work with compilers that support concepts but not ranges.