Re: display previous query string of idle-in-transaction

Lists: pgsql-hackers
From: Tatsuhito Kasahara <kasahara(dot)tatsuhito(at)oss(dot)ntt(dot)co(dot)jp>
To: pgsql-hackers(at)postgresql(dot)org
Subject: display previous query string of idle-in-transaction
Date: 2009-03-25 09:08:43
Message-ID: 49C9F49B.8020003@oss.ntt.co.jp
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

Hi.

Now, we can check the running query string by pg_stat_activity.current_query.
If we can also check previous query_string of idle-in-transaction,
it is useful for analysis of long transaction problem.

Long-transaction is a trouble, because it prevents defragmentation of HOT and VACUUM.
And long-transaction tends to be it in a state of "idle in transaction".
(BEGIN -> SOME SQL -> .... (long-transactin) ....)

So, I sometimes want to know what query (main cause) was done before
transaction which have been practiced for a long time.

I think that we are glad when we can confirm it in the following form.
# We will be able to use debug_query_string in postgres.c for this purpose.

=================================================================
=# SELECT current_query FROM pg_stat_activity
WHERE procpid <> pg_backend_pid();

current_query
---------------------------------------------------------------
<IDLE> in transaction [prev]: SELECT * FROM pg_class limit 1;

=================================================================

Thoughts?

Best regards.

--
Tatsuhito Kasahara
kasahara(dot)tatsuhito(at)oss(dot)ntt(dot)co(dot)jp


From: Gurjeet Singh <singh(dot)gurjeet(at)gmail(dot)com>
To: Tatsuhito Kasahara <kasahara(dot)tatsuhito(at)oss(dot)ntt(dot)co(dot)jp>
Cc: pgsql-hackers(at)postgresql(dot)org
Subject: Re: display previous query string of idle-in-transaction
Date: 2009-03-25 09:59:37
Message-ID: 65937bea0903250259t15698470k8a8c92501c6b4778@mail.gmail.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

This sure is a desirable feature. I have seen quite a few instances, where
the app is in 'IDLE in Transaction' state, and we are left with the only
choice of killing such processes from OS. (Remember pg_cancel_backend() does
not work for sessions in IDLE or IDLE in transaction state)

Also, it should be introduced as a new column, rather than tacking on the
existing string. Although this column will be of little use in cases where
current query is visible, but having a separate column looks like a cleaner
and simpler implementation.

Best regards,

2009/3/25 Tatsuhito Kasahara <kasahara(dot)tatsuhito(at)oss(dot)ntt(dot)co(dot)jp>

> Hi.
>
> Now, we can check the running query string by
> pg_stat_activity.current_query.
> If we can also check previous query_string of idle-in-transaction,
> it is useful for analysis of long transaction problem.
>
> Long-transaction is a trouble, because it prevents defragmentation of HOT
> and VACUUM.
> And long-transaction tends to be it in a state of "idle in transaction".
> (BEGIN -> SOME SQL -> .... (long-transactin) ....)
>
> So, I sometimes want to know what query (main cause) was done before
> transaction which have been practiced for a long time.
>
> I think that we are glad when we can confirm it in the following form.
> # We will be able to use debug_query_string in postgres.c for this purpose.
>
> =================================================================
> =# SELECT current_query FROM pg_stat_activity
> WHERE procpid <> pg_backend_pid();
>
> current_query
> ---------------------------------------------------------------
> <IDLE> in transaction [prev]: SELECT * FROM pg_class limit 1;
>
> =================================================================
>
> Thoughts?
>
> Best regards.
>
> --
> Tatsuhito Kasahara
> kasahara(dot)tatsuhito(at)oss(dot)ntt(dot)co(dot)jp
>
> --
> Sent via pgsql-hackers mailing list (pgsql-hackers(at)postgresql(dot)org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-hackers
>

--
gurjeet[(dot)singh](at)EnterpriseDB(dot)com
singh(dot)gurjeet(at){ gmail | hotmail | indiatimes | yahoo }.com

EnterpriseDB http://www.enterprisedb.com

Mail sent from my BlackLaptop device


From: hubert depesz lubaczewski <depesz(at)depesz(dot)com>
To: Tatsuhito Kasahara <kasahara(dot)tatsuhito(at)oss(dot)ntt(dot)co(dot)jp>
Cc: pgsql-hackers(at)postgresql(dot)org
Subject: Re: display previous query string of idle-in-transaction
Date: 2009-03-25 16:48:59
Message-ID: 20090325164859.GB30902@depesz.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

On Wed, Mar 25, 2009 at 06:08:43PM +0900, Tatsuhito Kasahara wrote:
> So, I sometimes want to know what query (main cause) was done before
> transaction which have been practiced for a long time.
> Thoughts?

I would love to get it, but when I suggested it some time in the past
Tom shot it down as bad idea.

http://archives.postgresql.org/message-id/20071016132131.GA4438@depesz.com

To be honest - I have yet to see case described by Ton (commit; begin;).

Best regards,

depesz

--
Linkedin: http://www.linkedin.com/in/depesz / blog: http://www.depesz.com/
jid/gtalk: depesz(at)depesz(dot)com / aim:depeszhdl / skype:depesz_hdl / gg:6749007


From: Guillaume Smet <guillaume(dot)smet(at)gmail(dot)com>
To: depesz(at)depesz(dot)com
Cc: Tatsuhito Kasahara <kasahara(dot)tatsuhito(at)oss(dot)ntt(dot)co(dot)jp>, pgsql-hackers(at)postgresql(dot)org
Subject: Re: display previous query string of idle-in-transaction
Date: 2009-03-25 17:07:24
Message-ID: 1d4e0c10903251007v500e762bqd3514a0c440b928c@mail.gmail.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

On Wed, Mar 25, 2009 at 5:48 PM, hubert depesz lubaczewski
<depesz(at)depesz(dot)com> wrote:
> I would love to get it, but when I suggested it some time in the past
> Tom shot it down as bad idea.
>
> http://archives.postgresql.org/message-id/20071016132131.GA4438@depesz.com

I agree with Tom here. I tracked a lot of idle in transaction problems
and you need the whole sequence of queries of all the backends to
really understand what happens.
The last query executed is mostly useless to solve this sort of
problem. At least, it was for me in every case I had to deal with.

--
Guillaume


From: Tatsuhito Kasahara <kasahara(dot)tatsuhito(at)oss(dot)ntt(dot)co(dot)jp>
To: Guillaume Smet <guillaume(dot)smet(at)gmail(dot)com>
Cc: depesz(at)depesz(dot)com, pgsql-hackers(at)postgresql(dot)org
Subject: Re: display previous query string of idle-in-transaction
Date: 2009-03-26 08:16:07
Message-ID: 49CB39C7.2000008@oss.ntt.co.jp
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

Guillaume Smet wrote:
> On Wed, Mar 25, 2009 at 5:48 PM, hubert depesz lubaczewski
> <depesz(at)depesz(dot)com> wrote:
>> I would love to get it, but when I suggested it some time in the past
>> Tom shot it down as bad idea.
>>
>> http://archives.postgresql.org/message-id/20071016132131.GA4438@depesz.com
>
> I agree with Tom here. I tracked a lot of idle in transaction problems
> and you need the whole sequence of queries of all the backends to
> really understand what happens.
> The last query executed is mostly useless to solve this sort of
> problem. At least, it was for me in every case I had to deal with.
I think so too. But last-query-string may be a useful *hint*.
It is a hard work to trace the whole sequence of queries.

In most cases, last query string is enough information to solve the
long transaction problem. And I want a easy way to get the informarion.
# last query string is also a key to trace queries.

"COMMIT;BEGIN;idle in transaction" is not a problem on 8.3 or later, but
"(BEGIN;)SOME SQL;idle in transaction" is a problem.

So, main purpose of displaying the last query string is ..
- check whether "idle in transaction (running long time) process
after SOME SQL" is exists or not.
- check the content of "SOME SQL".

best regards,

--
Tatsuhito Kasahara
kasahara(dot)tatsuhito(at)oss(dot)ntt(dot)co(dot)jp


From: ITAGAKI Takahiro <itagaki(dot)takahiro(at)oss(dot)ntt(dot)co(dot)jp>
To: Tatsuhito Kasahara <kasahara(dot)tatsuhito(at)oss(dot)ntt(dot)co(dot)jp>
Cc: Guillaume Smet <guillaume(dot)smet(at)gmail(dot)com>, depesz(at)depesz(dot)com, pgsql-hackers(at)postgresql(dot)org
Subject: Re: display previous query string of idle-in-transaction
Date: 2009-03-26 09:21:41
Message-ID: 20090326175406.AC8E.52131E4D@oss.ntt.co.jp
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers


Tatsuhito Kasahara <kasahara(dot)tatsuhito(at)oss(dot)ntt(dot)co(dot)jp> wrote:

> So, main purpose of displaying the last query string is ..
> - check whether "idle in transaction (running long time) process
> after SOME SQL" is exists or not.
> - check the content of "SOME SQL".

The feature could be achieved by an extension module using new executor
hooks in 8.4. It is just like contrib/pg_stat_statements; Backends store
their queries history in process-local or shared memory. If you store
queries in local memory, you could dump them at some of callback routines
called at the end of transaction. On the other hand, if you use shared
memory, it might be possitble to define a SRF function which return history
of queries:

CREATE FUNCTION query_history(backend_pid integer) RETURNS SETOF text;

Required memory for query history is not so much.
"1kB of query text * 100 connection * 10 queries" consumes just 1MB.

You can discard old queries at the end of transaction or out of memory.
If there are some interesting queries in it, you can dump them into
server logs. For example, logging configurations something like
'idle_in_transaction_min_duration' and 'total_transaction_min_duration'
will be possible.

I'm not sure this feature should be in the core or not.

Regards,
---
ITAGAKI Takahiro
NTT Open Source Software Center


From: Tatsuhito Kasahara <kasahara(dot)tatsuhito(at)oss(dot)ntt(dot)co(dot)jp>
To: ITAGAKI Takahiro <itagaki(dot)takahiro(at)oss(dot)ntt(dot)co(dot)jp>
Cc: Guillaume Smet <guillaume(dot)smet(at)gmail(dot)com>, depesz(at)depesz(dot)com, pgsql-hackers(at)postgresql(dot)org
Subject: Re: display previous query string of idle-in-transaction
Date: 2009-03-26 11:00:09
Message-ID: 49CB6039.9060306@oss.ntt.co.jp
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

ITAGAKI Takahiro wrote:
> The feature could be achieved by an extension module using new executor
> hooks in 8.4. It is just like contrib/pg_stat_statements;
Well, it is a good idea.

Displaying last-query-string may be useful, but it is not a feature for
general purpose. So, it may be an external module.

Now, I have two choice.

1. Displaying last-query-string by pg_stat_activity (suggested by Gurjeet):
Extends the pg_stat_activity to display "previous_query".
We can check only a previous query.

2. Displaying query-string-history by external module (suggested by Itagaki):
Using executor hook to get a query string and stores it in
shared memory.
We can check specified number of query string history.

I think I will try mainly 2.
# Or this feature may merge to pg_stat_statements.
Any comments welcome.

Best regards,
--
Tatsuhito Kasahara
kasahara(dot)tatsuhito(at)oss(dot)ntt(dot)co(dot)jp


From: Simon Riggs <simon(at)2ndQuadrant(dot)com>
To: Tatsuhito Kasahara <kasahara(dot)tatsuhito(at)oss(dot)ntt(dot)co(dot)jp>
Cc: pgsql-hackers(at)postgresql(dot)org
Subject: Re: display previous query string of idle-in-transaction
Date: 2009-03-26 18:14:33
Message-ID: 1238091273.16568.530.camel@ebony.2ndQuadrant
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers


On Wed, 2009-03-25 at 18:08 +0900, Tatsuhito Kasahara wrote:

> If we can also check previous query_string of idle-in-transaction,
> it is useful for analysis of long transaction problem.

I'm more interested in the problem itself. Why do you think there is a
problem and why does knowing this help you? I had similar problems
recently, so I'm interested in thoughts around this.

--
Simon Riggs www.2ndQuadrant.com
PostgreSQL Training, Services and Support


From: Tatsuhito Kasahara <kasahara(dot)tatsuhito(at)oss(dot)ntt(dot)co(dot)jp>
To: Simon Riggs <simon(at)2ndQuadrant(dot)com>
Cc: pgsql-hackers(at)postgresql(dot)org
Subject: Re: display previous query string of idle-in-transaction
Date: 2009-03-27 06:44:51
Message-ID: 49CC75E3.7010101@oss.ntt.co.jp
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

Simon Riggs wrote:
>> If we can also check previous query_string of idle-in-transaction,
>> it is useful for analysis of long transaction problem.
>
> I'm more interested in the problem itself. Why do you think there is a
> problem and why does knowing this help you? I had similar problems
> recently, so I'm interested in thoughts around this.
In many case, applications don't know what queries they are doing.
So, if a long transaction problem (unworking defragmnetation by HOT and
VACUUM) occured, I don't understand which application is the cause with
only "<IDLE> in transaction" message.

But if I can also check last query string, I guess which apllication
do that and point out the problem point.
# As I said, I think a long transaction is not a problem itself.

Does that answer your question ?

Best regards,

--
Tatsuhito Kasahara
kasahara(dot)tatsuhito(at)oss(dot)ntt(dot)co(dot)jp


From: Guillaume Smet <guillaume(dot)smet(at)gmail(dot)com>
To: Tatsuhito Kasahara <kasahara(dot)tatsuhito(at)oss(dot)ntt(dot)co(dot)jp>
Cc: Simon Riggs <simon(at)2ndquadrant(dot)com>, pgsql-hackers(at)postgresql(dot)org
Subject: Re: display previous query string of idle-in-transaction
Date: 2009-03-27 07:27:35
Message-ID: 1d4e0c10903270027k7e5ab5dduc9961f62ee89621d@mail.gmail.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

2009/3/27 Tatsuhito Kasahara <kasahara(dot)tatsuhito(at)oss(dot)ntt(dot)co(dot)jp>:
> But if I can also check last query string, I guess which apllication
> do that and point out the problem point.

Oh, I just understand why you want this patch. I usually have one
database per server so I didn't see your point.

Considering this usage, +1 for me.

--
Guillaume


From: Simon Riggs <simon(at)2ndQuadrant(dot)com>
To: Tatsuhito Kasahara <kasahara(dot)tatsuhito(at)oss(dot)ntt(dot)co(dot)jp>
Cc: pgsql-hackers(at)postgresql(dot)org
Subject: Re: display previous query string of idle-in-transaction
Date: 2009-03-27 07:36:58
Message-ID: 1238139418.16568.555.camel@ebony.2ndQuadrant
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers


On Fri, 2009-03-27 at 15:44 +0900, Tatsuhito Kasahara wrote:
> Simon Riggs wrote:
> >> If we can also check previous query_string of idle-in-transaction,
> >> it is useful for analysis of long transaction problem.
> >
> > I'm more interested in the problem itself. Why do you think there is a
> > problem and why does knowing this help you? I had similar problems
> > recently, so I'm interested in thoughts around this.
> In many case, applications don't know what queries they are doing.
> So, if a long transaction problem (unworking defragmnetation by HOT and
> VACUUM) occured, I don't understand which application is the cause with
> only "<IDLE> in transaction" message.
>
> But if I can also check last query string, I guess which apllication
> do that and point out the problem point.
> # As I said, I think a long transaction is not a problem itself.
>
> Does that answer your question ?

Not really. I want to understand the actual problem with
idle-in-transaction so we can consider all ways to solve it, rather than
just focus on one method.

--
Simon Riggs www.2ndQuadrant.com
PostgreSQL Training, Services and Support


From: Guillaume Smet <guillaume(dot)smet(at)gmail(dot)com>
To: Tatsuhito Kasahara <kasahara(dot)tatsuhito(at)oss(dot)ntt(dot)co(dot)jp>
Cc: Simon Riggs <simon(at)2ndquadrant(dot)com>, pgsql-hackers(at)postgresql(dot)org
Subject: Re: display previous query string of idle-in-transaction
Date: 2009-03-27 07:38:06
Message-ID: 1d4e0c10903270038s1785d37ai80b82e70f3287a24@mail.gmail.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

On Fri, Mar 27, 2009 at 8:27 AM, Guillaume Smet
<guillaume(dot)smet(at)gmail(dot)com> wrote:
> 2009/3/27 Tatsuhito Kasahara <kasahara(dot)tatsuhito(at)oss(dot)ntt(dot)co(dot)jp>:
>> But if I can also check last query string, I guess which apllication
>> do that and point out the problem point.
>
> Oh, I just understand why you want this patch. I usually have one
> database per server so I didn't see your point.

Thinking a bit more about it: the datname column in the
pg_stat_activity view gives you the database concerned and usename the
user used. So I still don't see your point: you can use different user
to distinguish the applications.

Moreover, if you're using connection pooling (which is more and more
common) and the same user for connecting to the database, you won't be
able to know if it's really the last query which causes the problem
(from my experience, it's usually not).

Being able to detect which application is running which query on the
very same database with the very same user seems like something not so
obvious and the use case seems to be pretty narrow. And IMHO, even if
we suppose you can make the difference between the applications with
only one query, you won't be able to limit your investigation to this
application.

So, in fact, I'm still not convinced. Could you detail a bit more how
you plan to use it?

--
Guillaume


From: Tatsuhito Kasahara <kasahara(dot)tatsuhito(at)oss(dot)ntt(dot)co(dot)jp>
To: Simon Riggs <simon(at)2ndQuadrant(dot)com>
Cc: pgsql-hackers(at)postgresql(dot)org
Subject: Re: display previous query string of idle-in-transaction
Date: 2009-03-27 07:49:56
Message-ID: 49CC8524.4090103@oss.ntt.co.jp
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

Simon Riggs wrote:
>> Does that answer your question ?
>
> Not really. I want to understand the actual problem with
> idle-in-transaction so we can consider all ways to solve it, rather than
> just focus on one method.
"idle in transaction timeout" feature may be one of the ways.
But I have no specific idea about it now.

--
Tatsuhito Kasahara
kasahara(dot)tatsuhito(at)oss(dot)ntt(dot)co(dot)jp


From: Simon Riggs <simon(at)2ndQuadrant(dot)com>
To: Tatsuhito Kasahara <kasahara(dot)tatsuhito(at)oss(dot)ntt(dot)co(dot)jp>
Cc: pgsql-hackers(at)postgresql(dot)org
Subject: Re: display previous query string of idle-in-transaction
Date: 2009-03-27 08:07:11
Message-ID: 1238141231.16568.574.camel@ebony.2ndQuadrant
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers


On Fri, 2009-03-27 at 16:49 +0900, Tatsuhito Kasahara wrote:
> Simon Riggs wrote:
> >> Does that answer your question ?
> >
> > Not really. I want to understand the actual problem with
> > idle-in-transaction so we can consider all ways to solve it, rather than
> > just focus on one method.
> "idle in transaction timeout" feature may be one of the ways.
> But I have no specific idea about it now.

Or take it further back still and think about why "idle in transaction"
occurs at all and fix *that*. Maybe not in Postgres at all, possibly in
the driver or even higher up client stack.

--
Simon Riggs www.2ndQuadrant.com
PostgreSQL Training, Services and Support


From: Guillaume Smet <guillaume(dot)smet(at)gmail(dot)com>
To: Simon Riggs <simon(at)2ndquadrant(dot)com>
Cc: Tatsuhito Kasahara <kasahara(dot)tatsuhito(at)oss(dot)ntt(dot)co(dot)jp>, pgsql-hackers(at)postgresql(dot)org
Subject: Re: display previous query string of idle-in-transaction
Date: 2009-03-27 09:57:44
Message-ID: 1d4e0c10903270257v5f0f2fdev6b742066e13fd974@mail.gmail.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

On Fri, Mar 27, 2009 at 9:07 AM, Simon Riggs <simon(at)2ndquadrant(dot)com> wrote:
> Or take it further back still and think about why "idle in transaction"
> occurs at all and fix *that*. Maybe not in Postgres at all, possibly in
> the driver or even higher up client stack.

From my experience, the main reasons for this problem are:
- a poorly coded connection pool (a lot of people think they really
should write their connection pooling solution themselves...);
- a poorly coded application;
- bugs in the code.

So you really should try to fix them in your application first.

That said, I have one case in mind where I wasn't able to fix
completely the connection pool and the application and we still
encounter idle in transaction connections from time to time.

For this sort of case, a timeout would be a nice solution.

--
Guillaume


From: Tatsuhito Kasahara <kasahara(dot)tatsuhito(at)oss(dot)ntt(dot)co(dot)jp>
To: Guillaume Smet <guillaume(dot)smet(at)gmail(dot)com>
Cc: Simon Riggs <simon(at)2ndquadrant(dot)com>, pgsql-hackers(at)postgresql(dot)org
Subject: Re: display previous query string of idle-in-transaction
Date: 2009-03-30 07:08:07
Message-ID: 49D06FD7.7030906@oss.ntt.co.jp
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

(Sorry for delay..)

Guillaume Smet wrote:
> Being able to detect which application is running which query on the
> very same database with the very same user seems like something not so
> obvious and the use case seems to be pretty narrow. And IMHO, even if
> we suppose you can make the difference between the applications with
> only one query, you won't be able to limit your investigation to this
> application.
Yes, I won't be able to *completely* detect which application is running
long transaction with a last query.

But, as I said, I can get a hint for guessing causes from it.
And, as Simon said, I can detect a problem point with collaboration
from other information (app's log, app's source, operation procedure, and so on).

> So, in fact, I'm still not convinced. Could you detail a bit more how
> you plan to use it?
Well, Now, I can't get enough information from pg_stat_activity.
So, I have to check huge logs or reproduce the same problem.
(They are annoying works.)

If I can check last and more queries, I can use it as a hint for narrowing
down problem points with app's log and so on.
# And search the point and fix (or suggesting action) it.

I hope it would be able to narrowing down problem points more easily.

Best regards,
--
Tatsuhito Kasahara
kasahara(dot)tatsuhito(at)oss(dot)ntt(dot)co(dot)jp


From: decibel <decibel(at)decibel(dot)org>
To: Simon Riggs <simon(at)2ndQuadrant(dot)com>
Cc: Tatsuhito Kasahara <kasahara(dot)tatsuhito(at)oss(dot)ntt(dot)co(dot)jp>, pgsql-hackers(at)postgresql(dot)org
Subject: Re: display previous query string of idle-in-transaction
Date: 2009-05-12 15:37:10
Message-ID: A90F3088-EC12-4802-BFC0-BE7BFEBA08F9@decibel.org
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

On Mar 27, 2009, at 2:36 AM, Simon Riggs wrote:
> Not really. I want to understand the actual problem with
> idle-in-transaction so we can consider all ways to solve it, rather
> than
> just focus on one method.

I have to distinct problems with idle in transaction. One is
reporting users / the tools they're using. I'll often find
transactions that have been open for minutes or hours. But, that's
not a big deal for me, because that's only impacting londiste slaves,
and I have no problem just killing those backends.

What does concern me is seeing idle in transaction from our web
servers that lasts anything more than a few fractions of a second.
Those cases worry me because I have to wonder if that's happening due
to bad code. Right now I can't think of any way to figure out if
that's the case other than a lot of complex logfile processing
(assuming that would even work). But if I knew what the previous
query was, I'd at least have half a chance to know what portion of
the code was responsible, and could then look at the code to see if
the idle state was expected or not.
--
Decibel!, aka Jim C. Nasby, Database Architect decibel(at)decibel(dot)org
Give your computer some brain candy! www.distributed.net Team #1828


From: Asko Oja <ascoja(at)gmail(dot)com>
To: decibel <decibel(at)decibel(dot)org>
Cc: Simon Riggs <simon(at)2ndquadrant(dot)com>, Tatsuhito Kasahara <kasahara(dot)tatsuhito(at)oss(dot)ntt(dot)co(dot)jp>, pgsql-hackers(at)postgresql(dot)org
Subject: Re: display previous query string of idle-in-transaction
Date: 2009-05-13 04:56:24
Message-ID: ecd779860905122156o48b3d07al75d2d21751c2a5bd@mail.gmail.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

After taking look at our monitoring system i think some hint about previous
SQL might be useful.

dba db70 db_name WARNING 1 long transactions, duration >
2690min user=postgres pid=7887 waiting=False query=<IDLE> in transaction

Currently i have no idea what exactly did i kill without digging in logs
which might have rotated anyway by now.

regards,
Asko

On Tue, May 12, 2009 at 6:37 PM, decibel <decibel(at)decibel(dot)org> wrote:

> On Mar 27, 2009, at 2:36 AM, Simon Riggs wrote:
>
>> Not really. I want to understand the actual problem with
>> idle-in-transaction so we can consider all ways to solve it, rather than
>> just focus on one method.
>>
>
>
> I have to distinct problems with idle in transaction. One is reporting
> users / the tools they're using. I'll often find transactions that have been
> open for minutes or hours. But, that's not a big deal for me, because that's
> only impacting londiste slaves, and I have no problem just killing those
> backends.
>
> What does concern me is seeing idle in transaction from our web servers
> that lasts anything more than a few fractions of a second. Those cases worry
> me because I have to wonder if that's happening due to bad code. Right now I
> can't think of any way to figure out if that's the case other than a lot of
> complex logfile processing (assuming that would even work). But if I knew
> what the previous query was, I'd at least have half a chance to know what
> portion of the code was responsible, and could then look at the code to see
> if the idle state was expected or not.
> --
> Decibel!, aka Jim C. Nasby, Database Architect decibel(at)decibel(dot)org
> Give your computer some brain candy! www.distributed.net Team #1828
>
>
>
>
> --
> Sent via pgsql-hackers mailing list (pgsql-hackers(at)postgresql(dot)org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-hackers
>


From: Greg Stark <stark(at)enterprisedb(dot)com>
To: Asko Oja <ascoja(at)gmail(dot)com>
Cc: decibel <decibel(at)decibel(dot)org>, Simon Riggs <simon(at)2ndquadrant(dot)com>, Tatsuhito Kasahara <kasahara(dot)tatsuhito(at)oss(dot)ntt(dot)co(dot)jp>, pgsql-hackers(at)postgresql(dot)org
Subject: Re: display previous query string of idle-in-transaction
Date: 2009-06-03 14:24:14
Message-ID: 4136ffa0906030724g72a56165s33819f3060002dcb@mail.gmail.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

Should this patch be on the commitfest page for 8.5? Or is there a
consensus already that it's a bad idea?

Personally I actually think this makes a lot of sense to do.

--
greg


From: "Kevin Grittner" <Kevin(dot)Grittner(at)wicourts(dot)gov>
To: "Greg Stark" <stark(at)enterprisedb(dot)com>, "Asko Oja" <ascoja(at)gmail(dot)com>
Cc: "Simon Riggs" <simon(at)2ndquadrant(dot)com>, "decibel" <decibel(at)decibel(dot)org>, "Tatsuhito Kasahara" <kasahara(dot)tatsuhito(at)oss(dot)ntt(dot)co(dot)jp>, <pgsql-hackers(at)postgresql(dot)org>
Subject: Re: display previous query string of idle-in-transaction
Date: 2009-06-03 14:33:44
Message-ID: 4A264377.EE98.0025.1@wicourts.gov
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

Greg Stark <stark(at)enterprisedb(dot)com> wrote:
> Should this patch be on the commitfest page for 8.5? Or is there a
> consensus already that it's a bad idea?
>
> Personally I actually think this makes a lot of sense to do.

+1

It at least gives one a reasonable chance to get a useful clue....

-Kevin


From: Tatsuhito Kasahara <kasahara(dot)tatsuhito(at)oss(dot)ntt(dot)co(dot)jp>
To: Kevin Grittner <Kevin(dot)Grittner(at)wicourts(dot)gov>
Cc: Greg Stark <stark(at)enterprisedb(dot)com>, Asko Oja <ascoja(at)gmail(dot)com>, Simon Riggs <simon(at)2ndquadrant(dot)com>, decibel <decibel(at)decibel(dot)org>, pgsql-hackers(at)postgresql(dot)org, "kasahara(OSSC)" <kasahara(dot)tatsuhito(at)oss(dot)ntt(dot)co(dot)jp>
Subject: Re: display previous query string of idle-in-transaction
Date: 2009-06-05 01:54:59
Message-ID: 4A287AF3.9000301@oss.ntt.co.jp
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

Kevin Grittner wrote:
> Greg Stark <stark(at)enterprisedb(dot)com> wrote:
>> Should this patch be on the commitfest page for 8.5? Or is there a
>> consensus already that it's a bad idea?
>>
>> Personally I actually think this makes a lot of sense to do.
>
> +1
>
> It at least gives one a reasonable chance to get a useful clue....
+1
# And I'm going to register this patch on the next commitfest page.

--
Tatsuhito Kasahara
kasahara(dot)tatsuhito(at)oss(dot)ntt(dot)co(dot)jp


From: Robert Haas <robertmhaas(at)gmail(dot)com>
To: Tatsuhito Kasahara <kasahara(dot)tatsuhito(at)oss(dot)ntt(dot)co(dot)jp>
Cc: Kevin Grittner <Kevin(dot)Grittner(at)wicourts(dot)gov>, Greg Stark <stark(at)enterprisedb(dot)com>, Asko Oja <ascoja(at)gmail(dot)com>, Simon Riggs <simon(at)2ndquadrant(dot)com>, decibel <decibel(at)decibel(dot)org>, pgsql-hackers(at)postgresql(dot)org
Subject: Re: display previous query string of idle-in-transaction
Date: 2009-06-05 02:22:41
Message-ID: 603c8f070906041922u7eaf0f76l8c2b7cbd3049c69b@mail.gmail.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

On Thu, Jun 4, 2009 at 9:54 PM, Tatsuhito
Kasahara<kasahara(dot)tatsuhito(at)oss(dot)ntt(dot)co(dot)jp> wrote:
> Kevin Grittner wrote:
>>
>> Greg Stark <stark(at)enterprisedb(dot)com> wrote:
>>>
>>> Should this patch be on the commitfest page for 8.5? Or is there a
>>> consensus already that it's a bad idea?
>>>
>>> Personally I actually think this makes a lot of sense to do.
>>
>>  +1
>>  It at least gives one a reasonable chance to get a useful clue....
>
> +1
> # And I'm going to register this patch on the next commitfest page.

I don't actually remember seeing actual code to implement this - is
this a patch, or just an idea? (I guess if this gets added to the
CommitFest page there will be a pointer to the code, but somehow I
can't find it at the moment.)

The only thing I don't like about this is that I think it's kind of a
hack to shove the <IDLE> in transaction designation and the query
string into the same database column. I've never liked having to
write:

select sum(1) from pg_stat_activity where current_query = '<IDLE> in
transaction';

...and I like it even less if I now have to modify that query to use
"like". We should find some way to represent this as structured
data... maybe make a separate column called "idle" that's a boolean,
or something, and let the query column contain the most recent query
(whether or not it's still executing).

...Robert


From: daveg <daveg(at)sonic(dot)net>
To: Robert Haas <robertmhaas(at)gmail(dot)com>
Cc: Tatsuhito Kasahara <kasahara(dot)tatsuhito(at)oss(dot)ntt(dot)co(dot)jp>, Kevin Grittner <Kevin(dot)Grittner(at)wicourts(dot)gov>, Greg Stark <stark(at)enterprisedb(dot)com>, Asko Oja <ascoja(at)gmail(dot)com>, Simon Riggs <simon(at)2ndquadrant(dot)com>, decibel <decibel(at)decibel(dot)org>, pgsql-hackers(at)postgresql(dot)org
Subject: Re: display previous query string of idle-in-transaction
Date: 2009-07-24 05:28:10
Message-ID: 20090724052810.GF18614@sonic.net
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

On Thu, Jun 04, 2009 at 10:22:41PM -0400, Robert Haas wrote:
> The only thing I don't like about this is that I think it's kind of a
> hack to shove the <IDLE> in transaction designation and the query
> string into the same database column. I've never liked having to
> write:
>
> select sum(1) from pg_stat_activity where current_query = '<IDLE> in
> transaction';
>
> ...and I like it even less if I now have to modify that query to use
> "like". We should find some way to represent this as structured
> data... maybe make a separate column called "idle" that's a boolean,
> or something, and let the query column contain the most recent query
> (whether or not it's still executing).

I like this idea a lot. Possibly it would be useful to have the end time
of the last query too, then one could find idle sessions that were old and
truly idle rather than just waiting for a busy client to send the next query.

select ... from pg_stat_activity
where idle
and last_statement_endtime < now() - interval '1 minute';

-dg

--
David Gould daveg(at)sonic(dot)net 510 536 1443 510 282 0869
If simplicity worked, the world would be overrun with insects.


From: "Kevin Grittner" <Kevin(dot)Grittner(at)wicourts(dot)gov>
To: "Robert Haas" <robertmhaas(at)gmail(dot)com>,"daveg" <daveg(at)sonic(dot)net>
Cc: "Simon Riggs" <simon(at)2ndquadrant(dot)com>, "decibel" <decibel(at)decibel(dot)org>, "Greg Stark" <stark(at)enterprisedb(dot)com>, "Asko Oja" <ascoja(at)gmail(dot)com>, "Tatsuhito Kasahara" <kasahara(dot)tatsuhito(at)oss(dot)ntt(dot)co(dot)jp>, <pgsql-hackers(at)postgresql(dot)org>
Subject: Re: display previous query string of idle-in-transaction
Date: 2009-07-24 14:47:06
Message-ID: 4A69831A0200002500028D0D@gw.wicourts.gov
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

daveg <daveg(at)sonic(dot)net> wrote:
> On Thu, Jun 04, 2009 at 10:22:41PM -0400, Robert Haas wrote:

>> maybe make a separate column called "idle" that's a boolean,
>> or something, and let the query column contain the most recent
>> query (whether or not it's still executing).

+1

> I like this idea a lot. Possibly it would be useful to have the end
> time of the last query too, then one could find idle sessions that
> were old and truly idle rather than just waiting for a busy client
> to send the next query.
>
> select ... from pg_stat_activity
> where idle
> and last_statement_endtime < now() - interval '1 minute';

+1

Of course, you might be more interested in those which are idle in a
transaction, but that's easily done with these changes -- just throw
in xact_start IS NULL.

-Kevin


From: Robert Haas <robertmhaas(at)gmail(dot)com>
To: Kevin Grittner <Kevin(dot)Grittner(at)wicourts(dot)gov>
Cc: daveg <daveg(at)sonic(dot)net>, Simon Riggs <simon(at)2ndquadrant(dot)com>, decibel <decibel(at)decibel(dot)org>, Greg Stark <stark(at)enterprisedb(dot)com>, Asko Oja <ascoja(at)gmail(dot)com>, Tatsuhito Kasahara <kasahara(dot)tatsuhito(at)oss(dot)ntt(dot)co(dot)jp>, pgsql-hackers(at)postgresql(dot)org
Subject: Re: display previous query string of idle-in-transaction
Date: 2009-07-24 14:58:15
Message-ID: 603c8f070907240758g34c1c54fp9a618e86adcc49a0@mail.gmail.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

On Fri, Jul 24, 2009 at 10:47 AM, Kevin
Grittner<Kevin(dot)Grittner(at)wicourts(dot)gov> wrote:
> daveg <daveg(at)sonic(dot)net> wrote:
>> On Thu, Jun 04, 2009 at 10:22:41PM -0400, Robert Haas wrote:
>
>>> maybe make a separate column called "idle" that's a boolean,
>>> or something, and let the query column contain the most recent
>>> query (whether or not it's still executing).
>
> +1
>
>> I like this idea a lot. Possibly it would be useful to have the end
>> time of the last query too, then one could find idle sessions that
>> were old and truly idle rather than just waiting for a busy client
>> to send the next query.
>>
>>    select ... from pg_stat_activity
>>      where idle
>>        and last_statement_endtime < now() - interval '1 minute';
>
> +1

Hmm, I don't think we'd need two columns for this, actually. You
could just have one column last_statement_endtime (not sure if it's
the best name, but something along those lines) which would be NULL if
the statement was still in progress and the appropriate timestamp if
not. You could infer idle from whether or not that column was NULL.

> Of course, you might be more interested in those which are idle in a
> transaction, but that's easily done with these changes -- just throw
> in xact_start IS NULL.

Surely if xact_start is NULL it is not in a transaction at all?

...Robert


From: "Kevin Grittner" <Kevin(dot)Grittner(at)wicourts(dot)gov>
To: "Robert Haas" <robertmhaas(at)gmail(dot)com>
Cc: "Simon Riggs" <simon(at)2ndquadrant(dot)com>, "decibel" <decibel(at)decibel(dot)org>, "Greg Stark" <stark(at)enterprisedb(dot)com>, "Asko Oja" <ascoja(at)gmail(dot)com>, "Tatsuhito Kasahara" <kasahara(dot)tatsuhito(at)oss(dot)ntt(dot)co(dot)jp>, <pgsql-hackers(at)postgresql(dot)org>,"daveg" <daveg(at)sonic(dot)net>
Subject: Re: display previous query string of idle-in-transaction
Date: 2009-07-24 15:06:32
Message-ID: 4A6987A80200002500028D18@gw.wicourts.gov
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

Robert Haas <robertmhaas(at)gmail(dot)com> wrote:

> Hmm, I don't think we'd need two columns for this, actually. You
> could just have one column last_statement_endtime (not sure if it's
> the best name, but something along those lines) which would be NULL
> if the statement was still in progress and the appropriate timestamp
> if not. You could infer idle from whether or not that column was
> NULL.

That would lose the ability to tell what the idle time was before the
latest statement began, but maybe that's not interesting enough to
justify another column....

>> Of course, you might be more interested in those which are idle in
>> a transaction, but that's easily done with these changes -- just
>> throw in xact_start IS NULL.
>
> Surely if xact_start is NULL it is not in a transaction at all?

That's exactly the point I was trying to make. Sorry if that appeared
to be saying anything else.

-Kevin


From: Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>
To: Robert Haas <robertmhaas(at)gmail(dot)com>
Cc: Kevin Grittner <Kevin(dot)Grittner(at)wicourts(dot)gov>, daveg <daveg(at)sonic(dot)net>, Simon Riggs <simon(at)2ndquadrant(dot)com>, decibel <decibel(at)decibel(dot)org>, Greg Stark <stark(at)enterprisedb(dot)com>, Asko Oja <ascoja(at)gmail(dot)com>, Tatsuhito Kasahara <kasahara(dot)tatsuhito(at)oss(dot)ntt(dot)co(dot)jp>, pgsql-hackers(at)postgresql(dot)org
Subject: Re: display previous query string of idle-in-transaction
Date: 2009-07-24 15:15:00
Message-ID: 27627.1248448500@sss.pgh.pa.us
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

Robert Haas <robertmhaas(at)gmail(dot)com> writes:
> Hmm, I don't think we'd need two columns for this, actually. You
> could just have one column last_statement_endtime (not sure if it's
> the best name, but something along those lines) which would be NULL if
> the statement was still in progress and the appropriate timestamp if
> not. You could infer idle from whether or not that column was NULL.

Yeah, but "where idle" or "where not idle" is a lot easier to type.
I think the extra column is justified on usability grounds. I'm also
not entirely convinced that we want last_statement_endtime, because
introducing that will cost us an extra kernel call per query in a lot of
scenarios. And gettimeofday() is not cheap everywhere.

Another question is that this proposal effectively redefines the
current_query column as not the "current" query, but something that
might be better be described as "latest_query". Should we change the
name? We'd probably break some client code if we did, but on the other
hand the semantics change might break such code anyway. Intentional
breakage might not be such a bad thing if it forces people to take a
fresh look at their code.

regards, tom lane


From: Robert Haas <robertmhaas(at)gmail(dot)com>
To: Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>
Cc: Kevin Grittner <Kevin(dot)Grittner(at)wicourts(dot)gov>, daveg <daveg(at)sonic(dot)net>, Simon Riggs <simon(at)2ndquadrant(dot)com>, decibel <decibel(at)decibel(dot)org>, Greg Stark <stark(at)enterprisedb(dot)com>, Asko Oja <ascoja(at)gmail(dot)com>, Tatsuhito Kasahara <kasahara(dot)tatsuhito(at)oss(dot)ntt(dot)co(dot)jp>, pgsql-hackers(at)postgresql(dot)org
Subject: Re: display previous query string of idle-in-transaction
Date: 2009-07-24 16:53:25
Message-ID: 603c8f070907240953i5be5557ah815b2f62dfb55c12@mail.gmail.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

On Fri, Jul 24, 2009 at 11:15 AM, Tom Lane<tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> Robert Haas <robertmhaas(at)gmail(dot)com> writes:
>> Hmm, I don't think we'd need two columns for this, actually.  You
>> could just have one column last_statement_endtime (not sure if it's
>> the best name, but something along those lines) which would be NULL if
>> the statement was still in progress and the appropriate timestamp if
>> not.  You could infer idle from whether or not that column was NULL.
>
> Yeah, but "where idle" or "where not idle" is a lot easier to type.
> I think the extra column is justified on usability grounds.  I'm also
> not entirely convinced that we want last_statement_endtime, because
> introducing that will cost us an extra kernel call per query in a lot of
> scenarios.  And gettimeofday() is not cheap everywhere.

I hate redundancy, but I don't care enough to argue about it.

> Another question is that this proposal effectively redefines the
> current_query column as not the "current" query, but something that
> might be better be described as "latest_query".  Should we change the
> name?  We'd probably break some client code if we did, but on the other
> hand the semantics change might break such code anyway.  Intentional
> breakage might not be such a bad thing if it forces people to take a
> fresh look at their code.

+1 for intentional breakage. I like the name, too.

...Robert


From: Peter Eisentraut <peter_e(at)gmx(dot)net>
To: pgsql-hackers(at)postgresql(dot)org
Cc: Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>, Robert Haas <robertmhaas(at)gmail(dot)com>, Kevin Grittner <Kevin(dot)Grittner(at)wicourts(dot)gov>, daveg <daveg(at)sonic(dot)net>, Simon Riggs <simon(at)2ndquadrant(dot)com>, decibel <decibel(at)decibel(dot)org>, Greg Stark <stark(at)enterprisedb(dot)com>, Asko Oja <ascoja(at)gmail(dot)com>, Tatsuhito Kasahara <kasahara(dot)tatsuhito(at)oss(dot)ntt(dot)co(dot)jp>
Subject: Re: display previous query string of idle-in-transaction
Date: 2009-07-28 11:43:10
Message-ID: 200907281443.10911.peter_e@gmx.net
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

On Friday 24 July 2009 18:15:00 Tom Lane wrote:
> Another question is that this proposal effectively redefines the
> current_query column as not the "current" query, but something that
> might be better be described as "latest_query". Should we change the
> name? We'd probably break some client code if we did, but on the other
> hand the semantics change might break such code anyway. Intentional
> breakage might not be such a bad thing if it forces people to take a
> fresh look at their code.

That breakage could be pretty widespread, though. Maybe have current_query
and last_query.