Something blocking in libpq_gettext?

Often, when we get short-lived network problems (like when our network
admins break the firewall), postgres client apps will lock-up. They do
not recover once the network is back to normal, do not time-out, and do
not fail with any sort of error.

This has been happening since at least postgres 7.4.7. We are currently
running 8.0.3 (now upgrading to 8.0.8).

As near as we can tell, it looks like libpq blocked inside poll inside
libpq_gettext.

#0 0xffffe405 in __kernel_vsyscall ()
#1 0x005a31d4 in poll () from /lib/tls/libc.so.6
#2 0xf7fd71ff in libpq_gettext () from /usr/lib/libpq.so.3
#3 0xf7fd7331 in pqWaitTimed () from /usr/lib/libpq.so.3
#4 0xf7fd73a1 in pqWait () from /usr/lib/libpq.so.3
#5 0xf7fd53fb in PQgetResult () from /usr/lib/libpq.so.3
#6 0xf7fd5524 in PQgetResult () from /usr/lib/libpq.so.3
#7 0x081b43b3 in SQLInterface::execute (this=0xf7ce3080,
cmd=0xf7ce0074 "execute lock_games ( '100' )") at SQLInterface.cpp:138

Can anyone offer any solutions, suggestions, fixes? We cannot reproduce
this at will, but are willing to provide more information when next it
occurs.
__
Marc

Marc Munro wrote:
> Often, when we get short-lived network problems (like when our network
> admins break the firewall), postgres client apps will lock-up. They do
> not recover once the network is back to normal, do not time-out, and do
> not fail with any sort of error.
>
> This has been happening since at least postgres 7.4.7. We are currently
> running 8.0.3 (now upgrading to 8.0.8).
>
> As near as we can tell, it looks like libpq blocked inside poll inside
> libpq_gettext.
>
> #0 0xffffe405 in __kernel_vsyscall ()
> #1 0x005a31d4 in poll () from /lib/tls/libc.so.6
> #2 0xf7fd71ff in libpq_gettext () from /usr/lib/libpq.so.3
> #3 0xf7fd7331 in pqWaitTimed () from /usr/lib/libpq.so.3

Wow, that's strange. Maybe it's trying to fetch something in the
message catalogs. What are your locale settings?

--
Alvaro Herrera http://www.CommandPrompt.com/
PostgreSQL Replication, Consulting, Custom Development, 24x7 support

On Fri, 2006-08-25 at 15:10 -0400, Alvaro Herrera wrote:

> Wow, that's strange. Maybe it's trying to fetch something in the
> message catalogs. What are your locale settings?
>

I'm not sure exactly what you need to know. I do have this tho:

LANG=en_US.UTF-8

What other information would be helpful?

__
Marc

Marc Munro wrote:
> On Fri, 2006-08-25 at 15:10 -0400, Alvaro Herrera wrote:
>
> > Wow, that's strange. Maybe it's trying to fetch something in the
> > message catalogs. What are your locale settings?
>
> I'm not sure exactly what you need to know. I do have this tho:
>
> LANG=en_US.UTF-8
>
> What other information would be helpful?

SHOW lc_messages, I think. Is it something else than C? If so, it will
probably try to read the corresponding postgres.mo file. Do the hung
processes have that file open? (I'm not sure if you can find that out
from only the core file.)

--
Alvaro Herrera http://www.CommandPrompt.com/
The PostgreSQL Company - Command Prompt, Inc.

Marc Munro <marc(at)bloodnok(dot)com> writes:
> As near as we can tell, it looks like libpq blocked inside poll inside
> libpq_gettext.

> #0 0xffffe405 in __kernel_vsyscall ()
> #1 0x005a31d4 in poll () from /lib/tls/libc.so.6
> #2 0xf7fd71ff in libpq_gettext () from /usr/lib/libpq.so.3
> #3 0xf7fd7331 in pqWaitTimed () from /usr/lib/libpq.so.3
> #4 0xf7fd73a1 in pqWait () from /usr/lib/libpq.so.3
> #5 0xf7fd53fb in PQgetResult () from /usr/lib/libpq.so.3
> #6 0xf7fd5524 in PQgetResult () from /usr/lib/libpq.so.3
> #7 0x081b43b3 in SQLInterface::execute (this=3D0xf7ce3080,=20
> cmd=3D0xf7ce0074 "execute lock_games ( '100' )") at SQLInterface.cpp:138

That backtrace is silly on its face --- apparently you are using a
stripped executable and gdb is providing the nearest global symbol
rather than the actual function name. I think you can safely assume
however that you are looking at libpq waiting for input from the
backend. Does the kernel at each end still think the connection is
live? (Try netstat) If this is a common result from short-lived
network problems then you have a beef with the TCP stack at one end
or the other ... TCP is supposed to be more robust than that.

regards, tom lane

On Fri, 2006-08-25 at 15:42 -0400, Alvaro Herrera wrote:
> SHOW lc_messages, I think. Is it something else than C? If so, it will
> probably try to read the corresponding postgres.mo file. Do the hung
> processes have that file open? (I'm not sure if you can find that out
> from only the core file.)
>

Database=# SHOW lc_messages;
lc_messages
-------------
en_US.UTF-8
(1 row)

Database=#

We don't think we can find this from the core file either. Next time it
happens we will check whether postgres.mo is in use. Unfortunately we
are no longer seeing the problem. We'll let you know when we have more.

Thanks for the quick response.
__
Marc

On Fri, 2006-08-25 at 15:43 -0400, Tom Lane wrote:
> Marc Munro <marc(at)bloodnok(dot)com> writes:
> > As near as we can tell, it looks like libpq blocked inside poll inside
> > libpq_gettext.
>
> > #0 0xffffe405 in __kernel_vsyscall ()
> > #1 0x005a31d4 in poll () from /lib/tls/libc.so.6
> > #2 0xf7fd71ff in libpq_gettext () from /usr/lib/libpq.so.3
> > #3 0xf7fd7331 in pqWaitTimed () from /usr/lib/libpq.so.3
> > #4 0xf7fd73a1 in pqWait () from /usr/lib/libpq.so.3
> > #5 0xf7fd53fb in PQgetResult () from /usr/lib/libpq.so.3
> > #6 0xf7fd5524 in PQgetResult () from /usr/lib/libpq.so.3
> > #7 0x081b43b3 in SQLInterface::execute (this=3D0xf7ce3080,=20
> > cmd=3D0xf7ce0074 "execute lock_games ( '100' )") at SQLInterface.cpp:138
>
> That backtrace is silly on its face --- apparently you are using a
> stripped executable and gdb is providing the nearest global symbol
> rather than the actual function name. I think you can safely assume
> however that you are looking at libpq waiting for input from the
> backend. Does the kernel at each end still think the connection is
> live? (Try netstat) If this is a common result from short-lived
> network problems then you have a beef with the TCP stack at one end
> or the other ... TCP is supposed to be more robust than that.

Yes indeed, it is a stripped executable. We are having issues with rpms
and debug symbols and this is the best we have been able to do so far.
We will try netstat next time this happens, and we are (still) trying to
get proper debug information. More and better information will follow
when we have it.

__
Marc

Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> writes:

> If this is a common result from short-lived
> network problems then you have a beef with the TCP stack at one end
> or the other ... TCP is supposed to be more robust than that.

Or a beef with some firewall or router along the way. NAT routers are
particularly prone to breaking TCP's robustness guarantees.

--
greg