Re: Add min and max execute statement time in pg_stat_statement

From: Arne Scheffer <arne(dot)scheffer(at)uni-muenster(dot)de>
To: Andrew Dunstan <andrew(at)dunslane(dot)net>, David G Johnston <david(dot)g(dot)johnston(at)gmail(dot)com>, <pgsql-hackers(at)postgresql(dot)org>
Subject: Re: Add min and max execute statement time in pg_stat_statement
Date: 2015-01-21 14:16:11
Message-ID: permail-20150121141611fe5316b600000d92-scheffa@message-id.uni-muenster.de
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-hackers

> >>I don't understand. I'm following pretty exactly the calculations
> >>stated
> >>at &lt;http://www.johndcook.com/blog/standard_deviation/&gt;

> >>I'm not a statistician. Perhaps others who are more literate in

Maybe I'm mistaken here,
but I think, the algorithm is not that complicated.
I try to explain it further:

Comments appreciated.

Definition var_samp = Sum of squared differences /n-1
Definition stddev_samp = sqrt(var_samp)

Example N=4

1.) Sum of squared differences
1_4Sum(Xi-XM4)²
=
2.) adding nothing
1_4Sum(Xi-XM4)²
+0
+0
+0
=
3.) nothing changed
1_4Sum(Xi-XM4)²
+(-1_3Sum(Xi-XM3)²+1_3Sum(Xi-XM3)²)
+(-1_2Sum(Xi-XM2)²+1_2Sum(Xi-XM3)²)
+(-1_1Sum(Xi-XM2)²+1_1Sum(Xi-XM3)²)

=
4.) parts reordered
(1_4Sum(Xi-XM4)²-1_3Sum(Xi-XM3)²)
+(1_3Sum(Xi-XM3)²-1_2Sum(Xi-XM2)²)
+(1_2Sum(Xi-XM2)²-1_1Sum(Xi-XM2)²)
+1_1Sum(X1-XM1)²
=
5.)
(X4-XM4)(X4-XM3)
+ (X3-XM3)(X3-XM2)
+ (X2-XM2)(X2-XM1)
+ (X1-XM1)²
=
6.) XM1=X1 => There it is - The iteration part of Welfords Algorithm (in
reverse order)
(X4-XM4)(X4-XM3)
+ (X3-XM3)(X3-XM2)
+ (X2-XM2)(X2-X1)
+ 0

The missing piece is 4.) to 5.)
it's algebra, look at e.g.:
http://jonisalonen.com/2013/deriving-welfords-method-for-computing-variance/

> Thanks. Still not quite sure what to do, though :-) I guess in the
> end we want the answer to come up with similar results to the builtin
> stddev SQL function. I'll try to set up a test program, to see if we do.

If you want to go this way:
Maybe this is one of the very few times, you have to use a small sample
;-)

VlG-Arne

> cheers

> andrew

> --
> Sent via pgsql-hackers mailing list (pgsql-hackers(at)postgresql(dot)org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-hackers

In response to

Responses

Browse pgsql-hackers by date

  From Date Subject
Next Message Arne Scheffer 2015-01-21 14:27:03 Re: Add min and max execute statement time in pg_stat_statement
Previous Message Amit Kapila 2015-01-21 14:10:04 Re: parallel mode and parallel contexts