Re: Reducing stats collection overhead

[Lukas Kahwe Smith <smith(at)pooteeweet(dot)org>]

Tom Lane wrote:

> The first design that comes to mind is that at transaction end
> (pgstat_report_tabstat() time) we send a stats message only if at least
> X milliseconds have elapsed since we last sent one, where X is
> PGSTAT_STAT_INTERVAL or closely related to it.  We also make sure to
> flush stats out before process exit.  This approach ensures that in a
> lots-of-short-transactions scenario, we only need to send one stats
> message every X msec, not one per query.  The cost is possible delay of
> stats reports.  I claim that any transaction that makes a really sizable
> change in the stats will run longer than X msec and therefore will send
> its stats immediately.  Cases where a client does a small transaction
> after sleeping for awhile (more than X msec) will also send immediately.
> You might get a delay in reporting the last few transactions of a burst
> of short transactions, but how much does it matter?  So I think that
> complicating the design with, say, a timeout counter to force out the
> stats after a sleep interval is not necessary.  Doing so would add a
> couple of kernel calls to every client interaction so I'd really rather
> avoid that.

Well and if this delaying of updating the stats has an effect on query 
time, then it also increases the likelihood of going past the X msec 
limit of that previously "small" query. So its sort of "self fixing" 
with the only risk of one query getting overly long due to lack of stats 
updating.

regards,
Lukas