Re: Protocol forced to V2 in low-memory conditions?

One of our customers seems to be running into exactly the issue
hypothesized about by Tom here:

http://www.postgresql.org/message-id/8040.1314403175@sss.pgh.pa.us

That is, the server is in low-memory conditions, and the client
occasionally issues an error saying it can't complete a certain action due
to the protocol version:

PG::UnableToSend: function requires at least protocol version 3.0: SELECT
...

The server is 9.2.4, and the client is the Ruby pg gem using a 9.2.4 libpq.

Was the possibility of an inadvertent protocol downgrade addressed as part
of that patch? I read through the thread, but it wasn't entirely clear.

Thanks,
Maciek

Maciek Sakrejda <m(dot)sakrejda(at)gmail(dot)com> writes:
> One of our customers seems to be running into exactly the issue
> hypothesized about by Tom here:
> http://www.postgresql.org/message-id/8040.1314403175@sss.pgh.pa.us
> Was the possibility of an inadvertent protocol downgrade addressed as part
> of that patch? I read through the thread, but it wasn't entirely clear.

No, a quick look at report_fork_failure_to_client shows it still always
sends V2 protocol. We fixed some of the lesser issues discussed in that
thread, but I don't think we ever agreed how to deal with this one.

I've been thinking of late that it might be time to retire libpq's
support for V2 protocol (other than in the specific context of the first
error message received while trying to make a connection). If we did
that, we'd remove the code path that thinks it should downgrade to V2
protocol, and thus fix this problem by removing code not adding more.

However, that doesn't sound like a back-patchable solution, and also
it remains unclear whether non-libpq clients such as JDBC have an issue
with this.

regards, tom lane

On Tue, Sep 10, 2013 at 5:29 AM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> Maciek Sakrejda <m(dot)sakrejda(at)gmail(dot)com> writes:
>> One of our customers seems to be running into exactly the issue
>> hypothesized about by Tom here:
>> http://www.postgresql.org/message-id/8040.1314403175@sss.pgh.pa.us
>> Was the possibility of an inadvertent protocol downgrade addressed as part
>> of that patch? I read through the thread, but it wasn't entirely clear.
>
> No, a quick look at report_fork_failure_to_client shows it still always
> sends V2 protocol. We fixed some of the lesser issues discussed in that
> thread, but I don't think we ever agreed how to deal with this one.
>
> I've been thinking of late that it might be time to retire libpq's
> support for V2 protocol (other than in the specific context of the first
> error message received while trying to make a connection). If we did
> that, we'd remove the code path that thinks it should downgrade to V2
> protocol, and thus fix this problem by removing code not adding more.
>
> However, that doesn't sound like a back-patchable solution, and also
> it remains unclear whether non-libpq clients such as JDBC have an issue
> with this.

It's probably worth polling for that. I believe the jdbc driver at
least has code for it, but I don't know if it's a requirement at this
point.

If it is, that might be a "10.0 release" feature, in how it would
break things :(

--
Magnus Hagander
Me: http://www.hagander.net/
Work: http://www.redpill-linpro.com/

On 2013-09-10 12:31:22 +0200, Magnus Hagander wrote:
> On Tue, Sep 10, 2013 at 5:29 AM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> > Maciek Sakrejda <m(dot)sakrejda(at)gmail(dot)com> writes:
> >> One of our customers seems to be running into exactly the issue
> >> hypothesized about by Tom here:
> >> http://www.postgresql.org/message-id/8040.1314403175@sss.pgh.pa.us
> >> Was the possibility of an inadvertent protocol downgrade addressed as part
> >> of that patch? I read through the thread, but it wasn't entirely clear.
> >
> > No, a quick look at report_fork_failure_to_client shows it still always
> > sends V2 protocol. We fixed some of the lesser issues discussed in that
> > thread, but I don't think we ever agreed how to deal with this one.
> >
> > I've been thinking of late that it might be time to retire libpq's
> > support for V2 protocol (other than in the specific context of the first
> > error message received while trying to make a connection). If we did
> > that, we'd remove the code path that thinks it should downgrade to V2
> > protocol, and thus fix this problem by removing code not adding more.
> >
> > However, that doesn't sound like a back-patchable solution, and also
> > it remains unclear whether non-libpq clients such as JDBC have an issue
> > with this.
>
> It's probably worth polling for that. I believe the jdbc driver at
> least has code for it, but I don't know if it's a requirement at this
> point.

Yes, it has code for it and I think it's still used pretty frequently to
circumvent prepared statement planning problems (misestimation,
indeterminate types). So I think we need convincing reasons to break
their usage.

Greetings,

Andres Freund

--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

Andres Freund <andres(at)2ndquadrant(dot)com> writes:
> On 2013-09-10 12:31:22 +0200, Magnus Hagander wrote:
>> On Tue, Sep 10, 2013 at 5:29 AM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
>>> I've been thinking of late that it might be time to retire libpq's
>>> support for V2 protocol (other than in the specific context of the first
>>> error message received while trying to make a connection).

>> It's probably worth polling for that. I believe the jdbc driver at
>> least has code for it, but I don't know if it's a requirement at this
>> point.

> Yes, it has code for it and I think it's still used pretty frequently to
> circumvent prepared statement planning problems (misestimation,
> indeterminate types). So I think we need convincing reasons to break
> their usage.

Note that I was proposing removing libpq's support for V2 connections.
Not the backend's.

regards, tom lane

On Tue, Sep 10, 2013 at 3:53 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> Andres Freund <andres(at)2ndquadrant(dot)com> writes:
>> On 2013-09-10 12:31:22 +0200, Magnus Hagander wrote:
>>> On Tue, Sep 10, 2013 at 5:29 AM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
>>>> I've been thinking of late that it might be time to retire libpq's
>>>> support for V2 protocol (other than in the specific context of the first
>>>> error message received while trying to make a connection).
>
>>> It's probably worth polling for that. I believe the jdbc driver at
>>> least has code for it, but I don't know if it's a requirement at this
>>> point.
>
>> Yes, it has code for it and I think it's still used pretty frequently to
>> circumvent prepared statement planning problems (misestimation,
>> indeterminate types). So I think we need convincing reasons to break
>> their usage.
>
> Note that I was proposing removing libpq's support for V2 connections.
> Not the backend's.

Oh. I blame the fact that we call the backend site libpq as well :)

Anyway. In that case, it seems a lot more reasonable. But definitely
not something backpatchable. But it's been a very long time since we
had a supported backend version that didn't speak v3.

The possible thing to consider there is if there's a common pg fork
that uses v2 only, that would then no longer be compatible with the
standard libpq. I have no idea if such a thing exists, and I'm not
sure we even care if it does, given how far behind they're lagging in
that case...

--
Magnus Hagander
Me: http://www.hagander.net/
Work: http://www.redpill-linpro.com/

On Tue, Sep 10, 2013 at 9:53 AM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> Andres Freund <andres(at)2ndquadrant(dot)com> writes:
>> On 2013-09-10 12:31:22 +0200, Magnus Hagander wrote:
>>> On Tue, Sep 10, 2013 at 5:29 AM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
>>>> I've been thinking of late that it might be time to retire libpq's
>>>> support for V2 protocol (other than in the specific context of the first
>>>> error message received while trying to make a connection).
>
>>> It's probably worth polling for that. I believe the jdbc driver at
>>> least has code for it, but I don't know if it's a requirement at this
>>> point.
>
>> Yes, it has code for it and I think it's still used pretty frequently to
>> circumvent prepared statement planning problems (misestimation,
>> indeterminate types). So I think we need convincing reasons to break
>> their usage.
>
> Note that I was proposing removing libpq's support for V2 connections.
> Not the backend's.

I vote against this. If we remove V2 support from libpq, then we'll
have no easy way to test that the backend's support still works. And
we've got too many people using V2 to think that it's OK not to have
an easy way of testing that. I think the question we ought to be
asking is: how can we get widely-used connectors to stop relying on V2
in the first place?

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On 09/11/2013 02:30 PM, Robert Haas wrote:
> On Tue, Sep 10, 2013 at 9:53 AM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
>> Andres Freund <andres(at)2ndquadrant(dot)com> writes:
>>> On 2013-09-10 12:31:22 +0200, Magnus Hagander wrote:
>>>> On Tue, Sep 10, 2013 at 5:29 AM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
>>>>> I've been thinking of late that it might be time to retire libpq's
>>>>> support for V2 protocol (other than in the specific context of the first
>>>>> error message received while trying to make a connection).
>>>> It's probably worth polling for that. I believe the jdbc driver at
>>>> least has code for it, but I don't know if it's a requirement at this
>>>> point.
>>> Yes, it has code for it and I think it's still used pretty frequently to
>>> circumvent prepared statement planning problems (misestimation,
>>> indeterminate types). So I think we need convincing reasons to break
>>> their usage.
>> Note that I was proposing removing libpq's support for V2 connections.
>> Not the backend's.
> I vote against this. If we remove V2 support from libpq, then we'll
> have no easy way to test that the backend's support still works. And
> we've got too many people using V2 to think that it's OK not to have
> an easy way of testing that. I think the question we ought to be
> asking is: how can we get widely-used connectors to stop relying on V2
> in the first place?
>

How is it tested now, and who is doing the testing?

cheers

andrew

On Tue, Sep 10, 2013 at 03:55:30PM +0200, Magnus Hagander wrote:
> On Tue, Sep 10, 2013 at 3:53 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> > Andres Freund <andres(at)2ndquadrant(dot)com> writes:
> >> On 2013-09-10 12:31:22 +0200, Magnus Hagander wrote:
> >>> On Tue, Sep 10, 2013 at 5:29 AM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> >>>> I've been thinking of late that it might be time to retire libpq's
> >>>> support for V2 protocol (other than in the specific context of the first
> >>>> error message received while trying to make a connection).
> >
> >>> It's probably worth polling for that. I believe the jdbc driver at
> >>> least has code for it, but I don't know if it's a requirement at this
> >>> point.
> >
> >> Yes, it has code for it and I think it's still used pretty frequently to
> >> circumvent prepared statement planning problems (misestimation,
> >> indeterminate types). So I think we need convincing reasons to break
> >> their usage.
> >
> > Note that I was proposing removing libpq's support for V2 connections.
> > Not the backend's.
>
> Oh. I blame the fact that we call the backend site libpq as well :)
>
> Anyway. In that case, it seems a lot more reasonable. But definitely
> not something backpatchable. But it's been a very long time since we
> had a supported backend version that didn't speak v3.
>
> The possible thing to consider there is if there's a common pg fork
> that uses v2 only, that would then no longer be compatible with the
> standard libpq. I have no idea if such a thing exists, and I'm not
> sure we even care if it does, given how far behind they're lagging
> in that case...

How could we care anyhow? It's not like we have the resources to
maintain all our current released versions anyhow, e.g. current
support for 8.4 is a good bit less solid than for 9.2.

Cheers,
David.
--
David Fetter <david(at)fetter(dot)org> http://fetter.org/
Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter
Skype: davidfetter XMPP: david(dot)fetter(at)gmail(dot)com
iCal: webcal://www.tripit.com/feed/ical/people/david74/tripit.ics

Remember to vote!
Consider donating to Postgres: http://www.postgresql.org/about/donate

Andrew Dunstan <andrew(at)dunslane(dot)net> writes:
> On 09/11/2013 02:30 PM, Robert Haas wrote:
>> On Tue, Sep 10, 2013 at 9:53 AM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
>>> Note that I was proposing removing libpq's support for V2 connections.
>>> Not the backend's.

>> I vote against this. If we remove V2 support from libpq, then we'll
>> have no easy way to test that the backend's support still works.

> How is it tested now, and who is doing the testing?

Exactly. The current support in libpq is nigh useless for testing
purposes, because there's no way to activate that code path on command.
It only runs if libpq (thinks it) is connecting to a pre-7.4 backend.

regards, tom lane

On Wed, Sep 11, 2013 at 2:35 PM, Andrew Dunstan <andrew(at)dunslane(dot)net> wrote:
>> I vote against this. If we remove V2 support from libpq, then we'll
>> have no easy way to test that the backend's support still works. And
>> we've got too many people using V2 to think that it's OK not to have
>> an easy way of testing that. I think the question we ought to be
>> asking is: how can we get widely-used connectors to stop relying on V2
>> in the first place?
>
> How is it tested now, and who is doing the testing?

I don't know that there's any automated testing, but if someone
reports a bug that can only be reproduced using the v2 protocol, the
first thing I'm going to try to do is reproduce that using libpq, just
as I would with any other connector malfunction. From what's being
said here it sounds like I might have to hack libpq a bit to get it to
speak the v2 protocol, but how long is that going to take? Less time
than setting up an environment that can run whatever crazy connector
the client is using and trying to figure out whether the client or the
server is to blame, for sure.

Don't get me wrong, I think that the v2 protocol should go away, but
the real issue is the connectors that are still relying on it as a
primary way of talking to the server, not libpq. We could force all
of those connectors to update or die by removing server support, and
once the server support is gone I don't care about libpq any more. Or
maybe there's some friendlier way to wean those connectors off v2.
But in the meantime I'm skeptical that removing the code from libpq
gets us anywhere.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

[ still catching up on old email ]

I wrote:
> Andrew Dunstan <andrew(at)dunslane(dot)net> writes:
>> On 09/11/2013 02:30 PM, Robert Haas wrote:
>>> On Tue, Sep 10, 2013 at 9:53 AM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
>>>> Note that I was proposing removing libpq's support for V2 connections.
>>>> Not the backend's.

>>> I vote against this. If we remove V2 support from libpq, then we'll
>>> have no easy way to test that the backend's support still works.

>> How is it tested now, and who is doing the testing?

> Exactly. The current support in libpq is nigh useless for testing
> purposes, because there's no way to activate that code path on command.
> It only runs if libpq (thinks it) is connecting to a pre-7.4 backend.

Actually ... there's another way we might deal with this. Suppose we
invent a libpq connection option to specify whether to use v2 or v3
protocol, defaulting to the latter, and then just remove the protocol-
fallback-during-connection code path. If there is anyone out there using
a modern libpq to talk to a pre-7.4 server, they can be told to invoke
the connection option. This gets rid of the unexpected-protocol-downgrade
problem in a reliable manner, and it also gives us a way to test V2 code
paths in both libpq and the backend, which Andrew is correct to finger as
something that goes nearly totally untested right now.

The main objections I can see to this are (1) it wouldn't provide
a back-patchable fix, and (2) it'd be adding more legacy code instead
of removing it. But the other approaches that we've talked about didn't
sound very back-patchable either, so I think only (2) really holds
much water.

regards, tom lane

On Fri, Nov 8, 2013 at 2:23 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> [ still catching up on old email ]
>
> I wrote:
>> Andrew Dunstan <andrew(at)dunslane(dot)net> writes:
>>> On 09/11/2013 02:30 PM, Robert Haas wrote:
>>>> On Tue, Sep 10, 2013 at 9:53 AM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
>>>>> Note that I was proposing removing libpq's support for V2 connections.
>>>>> Not the backend's.
>
>>>> I vote against this. If we remove V2 support from libpq, then we'll
>>>> have no easy way to test that the backend's support still works.
>
>>> How is it tested now, and who is doing the testing?
>
>> Exactly. The current support in libpq is nigh useless for testing
>> purposes, because there's no way to activate that code path on command.
>> It only runs if libpq (thinks it) is connecting to a pre-7.4 backend.
>
> Actually ... there's another way we might deal with this. Suppose we
> invent a libpq connection option to specify whether to use v2 or v3
> protocol, defaulting to the latter, and then just remove the protocol-
> fallback-during-connection code path. If there is anyone out there using
> a modern libpq to talk to a pre-7.4 server, they can be told to invoke
> the connection option. This gets rid of the unexpected-protocol-downgrade
> problem in a reliable manner, and it also gives us a way to test V2 code
> paths in both libpq and the backend, which Andrew is correct to finger as
> something that goes nearly totally untested right now.
>
> The main objections I can see to this are (1) it wouldn't provide
> a back-patchable fix, and (2) it'd be adding more legacy code instead
> of removing it. But the other approaches that we've talked about didn't
> sound very back-patchable either, so I think only (2) really holds
> much water.

This approach sounds promising to me.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company