Re: pluggable compression support

On 06/15/2013 02:02 PM, Andres Freund wrote:
> On 2013-06-15 13:11:47 +0200, Hannu Krosing wrote:
>> If it were truly pluggable - that is just load a .dll, set a GUG and start
>> using it
> Ok. I officially rechristen the patchset to 'extensible compression
> support'.
>
>> - then the selection of algorithms would be much
>> wider as several slow-but-high-compression ones under GPL could be
>> used as well, similar to how currently PostGIS works.
>> gzip and bzip2 are the first two that came in mind, but I am sure there
>> are more.
> gzip barely has a higher compression ratio than lz4 and is a magnitude
> slower decompressing, so I don't think it's interesting.
> I personally think bzip2 is too slow to be useful, even for
> decompression.
with low compression settings gzip and bzip2 seem to decompress at the
same speed :
http://pokecraft.first-world.info/wiki/Quick_Benchmark:_Gzip_vs_Bzip2_vs_LZMA_vs_XZ_vs_LZ4_vs_LZO

(an interesting thing there is memory usage, but I guess it is just an
artefact of outer layers around the algorithm)

and if better compression translates to more speed depends heavily on
disk speeds :

http://www.citusdata.com/blog/64-zfs-compression claims quite big
performance increases from using gzip, even with its slow decompression"

> What might be useful is something like lzma, but it's
> implementation is so complex that I don't really want to touch it.
>
>>> In the past, we've had a great deal of speculation about that legal
>>> question from people who are not lawyers. Maybe it would be valuable
>>> to get some opinions from people who ARE lawyers.
>> Making a truly pluggable compression API delegates this question
>> to end users.
>>
>> Delegation is good, as it lets you get done more :)
> No. It increases the features complexity by a magnitude. That's not
> good. And it means that about nobody but a few expert users will benefit
> from it, so I am pretty strongly opposed.
>
> You suddently need to solve the question of how the identifiers for
> compression formats are allocated and preserved across pg_upgrade and
> across machines.
This is something similar we already need to do for any non-builtin type
OID.
>
> Greetings,
>
> Andres Freund
>

--
Hannu Krosing
PostgreSQL Consultant
Performance, Scalability and High Availability
2ndQuadrant Nordic OÜ