Re: Compression and on-disk sorting

["Jim C. Nasby" <jnasby(at)pervasive(dot)com>]

On Fri, May 26, 2006 at 12:35:36PM -0400, Tom Lane wrote:
> "Jim C. Nasby" <jnasby(at)pervasive(dot)com> writes:
> > Something else worth mentioning is that sort performance is worse with
> > larger work_mem for all cases except the old HEAD, prior to the
> > tuplesort.c changes. It looks like whatever was done to fix that will
> > need to be adjusted/rethought pending the outcome of using compression.
> 
> Please clarify.  What are you comparing here exactly, and what cases did
> you test?

Sorry, forgot to put the url in:
http://jim.nasby.net/misc/pgsqlcompression/compress_sort.txt

But the meat is:
                                        -- work_mem --
                        Scale           2000    20000
not compressed          150             805.7   797.7
not compressed          3000            17820   17436
compressed              150             371.4   400.1
compressed              3000            8152    8537
compressed, no headers  3000            7325    7876

Performance degrades with more work_mem any time compression is used. I
thought I had data on just your tuplesort.c change without compression,
but I guess I don't. :( I can run that tonight if desired.

As for the code, the 3 things I've tested are HEAD as of 5/17/06 with no
patches (labeld 'not compressed'); that code with the compression patch
(compressed), and that code with both the compression patch and your change to
tuplesort.c that removes tuple headers from the sorted data (compressed, no
headers).
-- 
Jim C. Nasby, Sr. Engineering Consultant      jnasby(at)pervasive(dot)com
Pervasive Software      http://pervasive.com    work: 512-231-6117
vcard: http://jim.nasby.net/pervasive.vcf       cell: 512-569-9461