Re: Selecting K random rows - efficiently!

From: cluster <skrald(at)amossen(dot)dk>
To: pgsql-general(at)postgresql(dot)org
Cc: john(at)mitre(dot)org
Subject: Re: Selecting K random rows - efficiently!
Date: 2007-10-26 19:32:30
Message-ID: 472240CE.5000102@amossen.dk
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-general

> All you're doing is picking random =subsequences= from the same
> permutation of the original data.

You have some good points in your reply. I am very much aware of this
non-random behavior you point out for the "static random-value column"
approach but at least it is fast, which is a requirement. :-(
However, if the life time of the individual rows are short, the
behaviour is, luckily, sufficiently random for my specific purpose.

I furthermore realize that the only way to get truly random samples is
to ORDER BY random(), but this is an unacceptable slow method for large
data sets.
Even though it is not trivial at all, there ARE indeed algorithms out
there [1,2] for picking random sub sets from a result set but these are
(sadly) not implemented in postgresql.

References:
[1] http://portal.acm.org/citation.cfm?id=304206

[2] http://compstat.chonbuk.ac.kr/Sisyphus/CurrentStudy/Sampling/vldb86.pdf

In response to

Browse pgsql-general by date

  From Date Subject
Next Message Rainer Bauer 2007-10-26 19:35:05 Re: 8.2.3: Server crashes on Windows using Eclipse/Junit
Previous Message Tom Lane 2007-10-26 19:22:44 Re: change format of logging statements?