Documenting the Frontend/Backend Protocol update criteria

Hi,

Currently the criteria on updating the F/B protocol is undefined. This makes it hard to update the protocol going forward. It makes it also hard to write library/driver/application implementations that will be more future proof to future server versions.

Ideally the documentation for 9.4 (backport?) would say what kind of things are allowed to change within the v3 protocol, and thus implies what kind of changes need a new v4 protocol. Is there some wishlist page of items to do differently for v4 already?

I did find the following text in the documentation: "binary representations for complex data types might change across server versions". But having more specific rules would help, especially since it seems to be there just to scare: so far changes have been strongly discouraged.

An example to consider: some binary formats have flags (arrays) or version (jsonb) field. We should explicitly say that clients must reject any unknown bits/versions that they do not know about. This guarantees they detect small format updates instead of silently accepting then and possibly returning corrupt data.

My motivation:

Two years ago accidentally I opened a discussion on how to do updates to the binary encoding of data in the protocol [1]. I would like to reopen the discussion now since the jsonb 'binary' encoding is just a version '1' + text json. The result from the last discussion was that having a version or flags as part of the binary format is not enough, since drivers/libraries (fixable) and applications (unfixable) are depending on the current encoding.
And if we add a new bit to the flags or bump the version number we will break backward compatibility.

To summarize the previous discussion:
- there are currently no written rules for modifying the binary encoding formats
- bytea modification was done with a GUC, but GUC was seen as a bad solution in general
- my proposal was to add a minor format version number was not good either since any per session state would be problematic for connection poolers

[1]: http://grokbase.com/t/postgresql/pgsql-hackers/11bwhv1esa/add-minor-version-to-v3-protocol-to-allow-changes-without-breaking-backwards-compatibility

Jan Urbański made a presentation titled 'Postgres on the wire',
subtitle 'A look at the PostgreSQL wire protocol'. I hope this
covers some of your interest. Presentation slide deck is available
at
http://www.pgcon.org/2014/schedule/attachments/330_postgres-for-the-wire.pdf

Hope it helps;
---
Koichi Suzuki

2014-06-02 7:22 GMT+09:00 Mikko Tiihonen <Mikko(dot)Tiihonen(at)nitorcreations(dot)com>:
> Hi,
>
> Currently the criteria on updating the F/B protocol is undefined. This makes
> it hard to update the protocol going forward. It makes it also hard to write
> library/driver/application implementations that will be more future proof to
> future server versions.
>
> Ideally the documentation for 9.4 (backport?) would say what kind of things
> are allowed to change within the v3 protocol, and thus implies what kind of
> changes need a new v4 protocol. Is there some wishlist page of items to do
> differently for v4 already?
>
> I did find the following text in the documentation: "binary representations
> for complex data types might change across server versions". But having more
> specific rules would help, especially since it seems to be there just to
> scare: so far changes have been strongly discouraged.
>
> An example to consider: some binary formats have flags (arrays) or version
> (jsonb) field. We should explicitly say that clients must reject any unknown
> bits/versions that they do not know about. This guarantees they detect small
> format updates instead of silently accepting then and possibly returning
> corrupt data.
>
> My motivation:
>
> Two years ago accidentally I opened a discussion on how to do updates to the
> binary encoding of data in the protocol [1]. I would like to reopen the
> discussion now since the jsonb 'binary' encoding is just a version '1' +
> text json. The result from the last discussion was that having a version or
> flags as part of the binary format is not enough, since drivers/libraries
> (fixable) and applications (unfixable) are depending on the current
> encoding.
> And if we add a new bit to the flags or bump the version number we will
> break backward compatibility.
>
> To summarize the previous discussion:
> - there are currently no written rules for modifying the binary encoding
> formats
> - bytea modification was done with a GUC, but GUC was seen as a bad solution
> in general
> - my proposal was to add a minor format version number was not good either
> since any per session state would be problematic for connection poolers
>
> [1]:
> http://grokbase.com/t/postgresql/pgsql-hackers/11bwhv1esa/add-minor-version-to-v3-protocol-to-allow-changes-without-breaking-backwards-compatibility