Re: libpq compression

On Mon, Jun 25, 2012 at 3:38 PM, Euler Taveira <euler(at)timbira(dot)com> wrote:
> On 25-06-2012 16:45, Florian Pflug wrote:
>> Agreed. If we extend the protocol to support compression, and not rely
>> on SSL, then let's pick one of these LZ77-style compressors, and let's
>> integrate it into our tree.
>>
> If we have an option to have it out of our tree, good; if not, let's integrate
> it. I, particularly, don't see a compelling reason to integrate compression
> code in our tree, I mean, if we want to support more than one algorithm, it is
> clear that the overhead for maintain the compression code is too high (a lot
> of my-new-compression-algorithms).
>
>> We should then also make it possible to enable compression only for
>> the server -> client direction. Since those types of compressions are
>> usually pretty easy to decompress, that reduces the amount to work
>> non-libpq clients have to put in to take advantage of compression.
>>
> I don't buy this idea. My use case (data load) will not be covered if we only
> enable server -> client compression. I figure that there are use cases for
> server -> client compression (replication, for example) but also there are
> important use cases for client -> server (data load, for example). If you
> implement decompression, why not code compression code too?

+1. lz4, which is looking like a strong candidate, has c#, java,
etc. which are the main languages that are going to lag behind in
terms of protocol support. I don't think you're saving a lot by going
only one way (although you could make a case for the client to signal
interest in compression separately from decompression?)

another point:
It's been obvious for years now that zlib is somewhat of a dog in
terms of cpu usage for what it gives you. however, raw performance #s
are not the whole story -- it would be interesting to compress real
world protocol messages to/from the server in various scenarios to see
how compression would work, in particular with OLTP type queries --
for example pgbench runs, etc. That would speak a lot more to the
benefits than canned benchmarks.

merlin

merlin