Re: libpq compression

Hi,

There was already some discussion about compressing libpq data [1][2][3].
Recently, I faced a scenario that would become less problematic if we have had
compression support. The scenario is frequent data load (aka COPY) over
slow/unstable links. It should be executed in a few hundreds of PostgreSQL
servers all over Brazil. Someone could argue that I could use ssh tunnel to
solve the problem but (i) it is complex because it involves a different port
in the firewall and (ii) it's an opportunity to improve other scenarios like
reducing bandwidth consumption during replication or normal operation over
slow/unstable links.

AFAICS there aren't objections about implementing compression in libpq. The
problem is what algorithm use for compression. I mean, there is a lot of
patents in this area. As others spotted at [4], we should not implement
algorithms that possibly infringe patents in core. Derivated products are free
to plug whatever algorithms they want. There will be an API to do it.

This work will be sponsored by a company that is interested in this feature.

=== Design ===

- algorithm: zlib, bzip2, (another patent free and bsd licensed?)
- compiled-in option: --with-bzip2
- PGCOMPRESSMODE env
* disable: only try non-compressed connection (default)
* prefer: try compressed connection; if that fails, try a non-compressed
connection
* require: only try compressed connection
- PGCOMPRESSALGO env
* zlib
* bzip2
- compressmode and compressalgo string connection
- compress all data
- compress before send() and decompress after recv()

I am all ears for improving this design. Some of my choices are based on my
research in compression at protocols and PostgreSQL internals.

Keep in mind that I prefer compressing all data instead of a selected set of
messages because (i) every new data message could be coded with compression
support and (ii) avoid that the protocol code turns into a spaghetti.

I'll try to post a patch soon with the ideas discussed at this thread.

[1] http://archives.postgresql.org/pgsql-hackers/2012-03/msg00929.php
[2] http://archives.postgresql.org/pgsql-hackers/2011-01/msg00337.php
[3] http://archives.postgresql.org/pgsql-hackers/2002-03/msg00664.php
[4] http://archives.postgresql.org/pgsql-performance/2009-08/msg00053.php

--
Euler Taveira de Oliveira - Timbira http://www.timbira.com.br/
PostgreSQL: Consultoria, Desenvolvimento, Suporte 24x7 e Treinamento

Euler Taveira <euler(at)timbira(dot)com> writes:
> There was already some discussion about compressing libpq data [1][2][3].
> Recently, I faced a scenario that would become less problematic if we have had
> compression support. The scenario is frequent data load (aka COPY) over
> slow/unstable links. It should be executed in a few hundreds of PostgreSQL
> servers all over Brazil. Someone could argue that I could use ssh tunnel to
> solve the problem but (i) it is complex because it involves a different port
> in the firewall and (ii) it's an opportunity to improve other scenarios like
> reducing bandwidth consumption during replication or normal operation over
> slow/unstable links.

I still think that pushing this off to openssl (not an ssh tunnel, but
the underlying transport library) would be an adequate solution.
If you are shoving data over a connection that is long enough to need
compression, the odds that every bit of it is trustworthy seem pretty
small, so you need encryption too.

We do need the ability to tell openssl to use compression. We don't
need to implement it ourselves, nor to bring a bunch of new library
dependencies into our builds. I especially think that importing bzip2
is a pretty bad idea --- it's not only a new dependency, but bzip2's
compression versus speed tradeoff is entirely inappropriate for this
use-case.

regards, tom lane

Euler Taveira wrote:
> There was already some discussion about compressing libpq data
[1][2][3].
> Recently, I faced a scenario that would become less problematic if we
have had
> compression support. The scenario is frequent data load (aka COPY)
over
> slow/unstable links. It should be executed in a few hundreds of
PostgreSQL
> servers all over Brazil. Someone could argue that I could use ssh
tunnel to
> solve the problem but (i) it is complex because it involves a
different port
> in the firewall and (ii) it's an opportunity to improve other
scenarios like
> reducing bandwidth consumption during replication or normal operation
over
> slow/unstable links.

Maybe I'm missing something obvious, but shouldn't a regular SSL
connection (sslmode=require) do what you are asking for?

At least from OpenSSL 0.9.8 on, data get compressed by default.
You don't need an extra port in the firewall for that.

Yours,
Laurenz Albe

On 14-06-2012 02:19, Tom Lane wrote:
> I still think that pushing this off to openssl (not an ssh tunnel, but
> the underlying transport library) would be an adequate solution.
> If you are shoving data over a connection that is long enough to need
> compression, the odds that every bit of it is trustworthy seem pretty
> small, so you need encryption too.
>
I don't want to pay the SSL connection overhead. Also I just want compression,
encryption is not required. OpenSSL give us encryption with/without
compression; we need an option to obtain compression in non-SSL connections.

> We do need the ability to tell openssl to use compression. We don't
> need to implement it ourselves, nor to bring a bunch of new library
> dependencies into our builds. I especially think that importing bzip2
> is a pretty bad idea --- it's not only a new dependency, but bzip2's
> compression versus speed tradeoff is entirely inappropriate for this
> use-case.
>
I see, the idea is that bzip2 would be a compiled-in option (not enabled by
default) just to give another compression option. I don't have a strong
opinion about including it as another dependency. We already depend on zlib
and implementing compression using it won't add another dependency.

What do you think about adding a hook at libpq to load an extension that does
the compression? That way we don't add another dependency at libpq and also a
lot of extensions could be coded to cover a variety of algorithms without
putting us in trouble because of patent infringement.

--
Euler Taveira de Oliveira - Timbira http://www.timbira.com.br/
PostgreSQL: Consultoria, Desenvolvimento, Suporte 24x7 e Treinamento

On Jun14, 2012, at 15:28 , Euler Taveira wrote:
> On 14-06-2012 02:19, Tom Lane wrote:
>> I still think that pushing this off to openssl (not an ssh tunnel, but
>> the underlying transport library) would be an adequate solution.
>> If you are shoving data over a connection that is long enough to need
>> compression, the odds that every bit of it is trustworthy seem pretty
>> small, so you need encryption too.
>>
> I don't want to pay the SSL connection overhead. Also I just want compression,
> encryption is not required. OpenSSL give us encryption with/without
> compression; we need an option to obtain compression in non-SSL connections.

AFAIR, openssl supports a NULL cipher which doesn't do any encryption. We
could have a connection parameter, say compress=on, which selects that
cipher (unless sslmode is set to prefer or higher, of course).

SSL NULL-chipher connections would be treated like unencrypted connections
when matching against pg_hba.conf.

best regards,
Florian Pflug

On Thu, Jun 14, 2012 at 10:14 AM, Florian Pflug <fgp(at)phlo(dot)org> wrote:
> On Jun14, 2012, at 15:28 , Euler Taveira wrote:
>> On 14-06-2012 02:19, Tom Lane wrote:
>>> I still think that pushing this off to openssl (not an ssh tunnel, but
>>> the underlying transport library) would be an adequate solution.
>>> If you are shoving data over a connection that is long enough to need
>>> compression, the odds that every bit of it is trustworthy seem pretty
>>> small, so you need encryption too.
>>>
>> I don't want to pay the SSL connection overhead. Also I just want compression,
>> encryption is not required. OpenSSL give us encryption with/without
>> compression; we need an option to obtain compression in non-SSL connections.
>
> AFAIR, openssl supports a NULL cipher which doesn't do any encryption. We
> could have a connection parameter, say compress=on, which selects that
> cipher (unless sslmode is set to prefer or higher, of course).
>
> SSL NULL-chipher connections would be treated like unencrypted connections
> when matching against pg_hba.conf.
>
> best regards,
> Florian Pflug
>
>
> --
> Sent via pgsql-hackers mailing list (pgsql-hackers(at)postgresql(dot)org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-hackers

It doesn't sound like there is a lot of support for this idea, but I
think it would be nice to get something like lz4
(http://code.google.com/p/lz4/) or snappy
(http://code.google.com/p/snappy/) support. Both are BSD-ish licensed.
It could be useful for streaming replication as well. A hook (as Euler
mentioned) might be a nice compromise.

On Thu, Jun 14, 2012 at 9:57 AM, Phil Sorber <phil(at)omniti(dot)com> wrote:
> It doesn't sound like there is a lot of support for this idea, but I
> think it would be nice to get something like lz4
> (http://code.google.com/p/lz4/) or snappy
> (http://code.google.com/p/snappy/) support. Both are BSD-ish licensed.
> It could be useful for streaming replication as well. A hook (as Euler
> mentioned) might be a nice compromise.

There is a lot of support for the idea: it's one of the more requested
features. I think a well thought out framework that bypassed the
dependency issues via plugging might get some serious traction.
Emphasis 'on well thought out' :-).

merlin

Euler Taveira <euler(at)timbira(dot)com> writes:
> On 14-06-2012 02:19, Tom Lane wrote:
>> ... I especially think that importing bzip2
>> is a pretty bad idea --- it's not only a new dependency, but bzip2's
>> compression versus speed tradeoff is entirely inappropriate for this
>> use-case.

> I see, the idea is that bzip2 would be a compiled-in option (not enabled by
> default) just to give another compression option.

I'm not particularly thrilled with "let's have more compression options
just to have options". Each such option you add is another source of
fail-to-connect incompatibilities (when either the client or the server
doesn't have it). Moreover, while there are a lot of compression
algorithms out there, a lot of them are completely unsuited for this
use-case. If memory serves, bzip2 for example requires fairly large
data blocks in order to get decent compression, which means you are
either going to get terrible compression or suffer very bad latency
when trying to apply it to a connection data stream.

So I've got very little patience with the idea of "let's put in some
hooks and then great things will happen". It would be far better all
around if we supported exactly one, well-chosen, method. But really
I still don't see a reason not to let openssl do it for us.

regards, tom lane

On Thu, Jun 14, 2012 at 1:43 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> So I've got very little patience with the idea of "let's put in some
> hooks and then great things will happen".  It would be far better all
> around if we supported exactly one, well-chosen, method.  But really
> I still don't see a reason not to let openssl do it for us.

Well, for toast compression the right choice is definitely one of the
lz based algorithms (not libz!). For transport compression you have
the case of sending large data over very slow and/or expensive links
in which case you want to use bzip type methods. But in the majority
of cases I'd probably be using lz there too. So if I had to pick just
one, there you go. But which one? the lz algorithm with arguably the
best pedigree (lzo) is gnu but there are many other decent candidates,
some of which have really tiny implementations.

merlin

On Thu, Jun 14, 2012 at 02:38:02PM -0500, Merlin Moncure wrote:
> On Thu, Jun 14, 2012 at 1:43 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> > So I've got very little patience with the idea of "let's put in some
> > hooks and then great things will happen".  It would be far better all
> > around if we supported exactly one, well-chosen, method.  But really
> > I still don't see a reason not to let openssl do it for us.
>
> Well, for toast compression the right choice is definitely one of the
> lz based algorithms (not libz!). For transport compression you have
> the case of sending large data over very slow and/or expensive links
> in which case you want to use bzip type methods. But in the majority
> of cases I'd probably be using lz there too. So if I had to pick just
> one, there you go. But which one? the lz algorithm with arguably the
> best pedigree (lzo) is gnu but there are many other decent candidates,
> some of which have really tiny implementations.
>
> merlin
>

+1 for a very fast compressor/de-compressor. lz4 from Google has
a BSD license and at 8.5X faster compression than zlib(-1) and
5X faster de-compression the zlib (-1), 2X faster than LZO even
would be my pick.

Regards,
Ken

On Thu, Jun 14, 2012 at 02:43:04PM -0400, Tom Lane wrote:
> Euler Taveira <euler(at)timbira(dot)com> writes:
> > On 14-06-2012 02:19, Tom Lane wrote:
> >> ... I especially think that importing bzip2
> >> is a pretty bad idea --- it's not only a new dependency, but bzip2's
> >> compression versus speed tradeoff is entirely inappropriate for this
> >> use-case.
>
> > I see, the idea is that bzip2 would be a compiled-in option (not enabled by
> > default) just to give another compression option.
>
> I'm not particularly thrilled with "let's have more compression options
> just to have options". Each such option you add is another source of
> fail-to-connect incompatibilities (when either the client or the server
> doesn't have it). Moreover, while there are a lot of compression
> algorithms out there, a lot of them are completely unsuited for this
> use-case. If memory serves, bzip2 for example requires fairly large
> data blocks in order to get decent compression, which means you are
> either going to get terrible compression or suffer very bad latency
> when trying to apply it to a connection data stream.
>
> So I've got very little patience with the idea of "let's put in some
> hooks and then great things will happen". It would be far better all
> around if we supported exactly one, well-chosen, method. But really
> I still don't see a reason not to let openssl do it for us.

Do we just need to document SSL's NULL encryption option?

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ It's impossible for everything to be true. +

On Fri, Jun 15, 2012 at 10:19 AM, Bruce Momjian <bruce(at)momjian(dot)us> wrote:
> On Thu, Jun 14, 2012 at 02:43:04PM -0400, Tom Lane wrote:
>> Euler Taveira <euler(at)timbira(dot)com> writes:
>> > On 14-06-2012 02:19, Tom Lane wrote:
>> >> ...  I especially think that importing bzip2
>> >> is a pretty bad idea --- it's not only a new dependency, but bzip2's
>> >> compression versus speed tradeoff is entirely inappropriate for this
>> >> use-case.
>>
>> > I see, the idea is that bzip2 would be a compiled-in option (not enabled by
>> > default) just to give another compression option.
>>
>> I'm not particularly thrilled with "let's have more compression options
>> just to have options".  Each such option you add is another source of
>> fail-to-connect incompatibilities (when either the client or the server
>> doesn't have it).  Moreover, while there are a lot of compression
>> algorithms out there, a lot of them are completely unsuited for this
>> use-case.  If memory serves, bzip2 for example requires fairly large
>> data blocks in order to get decent compression, which means you are
>> either going to get terrible compression or suffer very bad latency
>> when trying to apply it to a connection data stream.

Agreed. I think there's probably arguments to be had for supporting
compression without openssl (see below), but I don't think we need to
have a whole set of potentially incompatible ways of doing it. Picking
one that's good for the common case and not completely crap for the
corner cases would be a better choice (meaning bzip2 is probably a
very bad choice).

>> So I've got very little patience with the idea of "let's put in some
>> hooks and then great things will happen".  It would be far better all
>> around if we supported exactly one, well-chosen, method.  But really
>> I still don't see a reason not to let openssl do it for us.
>
> Do we just need to document SSL's NULL encryption option?

Does the SSL NULL encryption+compression thing work if you're not
using openssl?

For one thing, some of us still hold a hope to support non-openssl
libraries in both libpq and server side, so it's something that would
need to be supported by the standard and thus available in most
libraries not to invalidate that.

Second, we also have things like the JDBC driver and the .Net driver
that don't use libpq. the JDBC driver uses the native java ssl
support, AFAIK. Does that one support the compression, and does it
support controlling it?

--
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/

On Jun15, 2012, at 07:50 , Magnus Hagander wrote:
>>> So I've got very little patience with the idea of "let's put in some
>>> hooks and then great things will happen". It would be far better all
>>> around if we supported exactly one, well-chosen, method. But really
>>> I still don't see a reason not to let openssl do it for us.
>>
>> Do we just need to document SSL's NULL encryption option?
>
> Does the SSL NULL encryption+compression thing work if you're not
> using openssl?

The compression support is defined in RFC 3749, and according to
http://en.wikipedia.org/wiki/Comparison_of_TLS_Implementations it's
supported in openssl and gnutls.

gnutls also seems to support a NULL cipher - gnutls-cli on my Ubuntu
10.04 box prints

Ciphers: AES-256-CBC, AES-128-CBC, 3DES-CBC, DES-CBC, ARCFOUR-128,
ARCFOUR-40, RC2-40, CAMELLIA-256-CBC, CAMELLIA-128-CBC, NULL.

> For one thing, some of us still hold a hope to support non-openssl
> libraries in both libpq and server side, so it's something that would
> need to be supported by the standard and thus available in most
> libraries not to invalidate that.

Well, it's a standard a least, and both openssl and gnutls seem to
support it. Are there any other ssl implementations beside gnutls and
openssl that we need to worry about?

> Second, we also have things like the JDBC driver and the .Net driver
> that don't use libpq. the JDBC driver uses the native java ssl
> support, AFAIK. Does that one support the compression, and does it
> support controlling it?

Java uses pluggable providers with standardized interfaces for most
things related to encryption. SSL support is provided by JSSE
(Java Secure Socket Extension). The JSSE implementation included with
the oracle JRE doesn't seem to support compression according to the
wikipedia page quoted above. But chances are that there exists an
alternative implementation which does.

best regards,
Florian Pflug

On Fri, Jun 15, 2012 at 5:52 PM, Florian Pflug <fgp(at)phlo(dot)org> wrote:
> On Jun15, 2012, at 07:50 , Magnus Hagander wrote:
>>>> So I've got very little patience with the idea of "let's put in some
>>>> hooks and then great things will happen".  It would be far better all
>>>> around if we supported exactly one, well-chosen, method.  But really
>>>> I still don't see a reason not to let openssl do it for us.
>>>
>>> Do we just need to document SSL's NULL encryption option?
>>
>> Does the SSL NULL encryption+compression thing work if you're not
>> using openssl?
>
> The compression support is defined in RFC 3749, and according to
> http://en.wikipedia.org/wiki/Comparison_of_TLS_Implementations it's
> supported in openssl and gnutls.
>
> gnutls also seems to support a NULL cipher - gnutls-cli on my Ubuntu
> 10.04 box prints
>
> Ciphers: AES-256-CBC, AES-128-CBC, 3DES-CBC, DES-CBC, ARCFOUR-128,
> ARCFOUR-40, RC2-40, CAMELLIA-256-CBC, CAMELLIA-128-CBC, NULL.

ah, thanks for looking that up for me!

The other big one to consider would be GNUTLS - which also has support
for compression, I see.

I guess a related question is if they all alow us to turn it *off*,
which we now do support on openssl :) gnutls does, I didn't look into
nss.

>> For one thing, some of us still hold a hope to support non-openssl
>> libraries in both libpq and server side, so it's something that would
>> need to be supported by the standard and thus available in most
>> libraries not to invalidate that.
>
> Well, it's a standard a least, and both openssl and gnutls seem to
> support it. Are there any other ssl implementations beside gnutls and
> openssl that we need to worry about?

NSS would be the big one, an din theory microsoft schannel if we were
to go there (that would give us access to easy use of the windows
certificate store so ther emight be a reason - but not a very big one,
to support that).

>> Second, we also have things like the JDBC driver and the .Net driver
>> that don't use libpq. the JDBC driver uses the native java ssl
>> support, AFAIK. Does that one support the compression, and does it
>> support controlling it?
>
> Java uses pluggable providers with standardized interfaces for most
> things related to encryption. SSL support is provided by JSSE
> (Java Secure Socket Extension). The JSSE implementation included with
> the oracle JRE doesn't seem to support compression according to the
> wikipedia page quoted above. But chances are that there exists an
> alternative implementation which does.

Yeah, but that alone is IMO a rather big blocker for claiming that
this is the only way to do it :( And I think the fact that that
wikipedia page doesn't list any other ones, is a sign that there might
not be a lot of other choices out there in reality - expecially not
opensource...

--
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/

On Jun15, 2012, at 12:09 , Magnus Hagander wrote:
> On Fri, Jun 15, 2012 at 5:52 PM, Florian Pflug <fgp(at)phlo(dot)org> wrote:
>> On Jun15, 2012, at 07:50 , Magnus Hagander wrote:
>>> Second, we also have things like the JDBC driver and the .Net driver
>>> that don't use libpq. the JDBC driver uses the native java ssl
>>> support, AFAIK. Does that one support the compression, and does it
>>> support controlling it?
>>
>> Java uses pluggable providers with standardized interfaces for most
>> things related to encryption. SSL support is provided by JSSE
>> (Java Secure Socket Extension). The JSSE implementation included with
>> the oracle JRE doesn't seem to support compression according to the
>> wikipedia page quoted above. But chances are that there exists an
>> alternative implementation which does.
>
> Yeah, but that alone is IMO a rather big blocker for claiming that
> this is the only way to do it :( And I think the fact that that
> wikipedia page doesn't list any other ones, is a sign that there might
> not be a lot of other choices out there in reality - expecially not
> opensource…

Hm, but things get even harder for the JDBC and .NET folks if we go
with a third-party compression method. Or would we require that the
existence of a free Java (and maybe .NET) implementation of such a
method would be an absolute must?

The way I see it, if we use SSL-based compression then non-libpq clients
there's at least a chance of those clients being able to use it easily
(if their SSL implementation supports it). If we go with a third-party
compression method, they *all* need to add yet another dependency, or may
even need to re-implement the compression method in their implementation
language of choice.

best regards,
Florian Pflug

On Fri, Jun 15, 2012 at 5:48 AM, Florian Pflug <fgp(at)phlo(dot)org> wrote:
> On Jun15, 2012, at 12:09 , Magnus Hagander wrote:
>> On Fri, Jun 15, 2012 at 5:52 PM, Florian Pflug <fgp(at)phlo(dot)org> wrote:
>>> On Jun15, 2012, at 07:50 , Magnus Hagander wrote:
>>>> Second, we also have things like the JDBC driver and the .Net driver
>>>> that don't use libpq. the JDBC driver uses the native java ssl
>>>> support, AFAIK. Does that one support the compression, and does it
>>>> support controlling it?
>>>
>>> Java uses pluggable providers with standardized interfaces for most
>>> things related to encryption. SSL support is provided by JSSE
>>> (Java Secure Socket Extension). The JSSE implementation included with
>>> the oracle JRE doesn't seem to support compression according to the
>>> wikipedia page quoted above. But chances are that there exists an
>>> alternative implementation which does.
>>
>> Yeah, but that alone is IMO a rather big blocker for claiming that
>> this is the only way to do it :( And I think the fact that that
>> wikipedia page doesn't list any other ones, is a sign that there might
>> not be a lot of other choices out there in reality - expecially not
>> opensource…
>
> Hm, but things get even harder for the JDBC and .NET folks if we go
> with a third-party compression method. Or would we require that the
> existence of a free Java (and maybe .NET) implementation of such a
> method would be an absolute must?
>
> The way I see it, if we use SSL-based compression then non-libpq clients
> there's at least a chance of those clients being able to use it easily
> (if their SSL implementation supports it). If we go with a third-party
> compression method, they *all* need to add yet another dependency, or may
> even need to re-implement the compression method in their implementation
> language of choice.

hm, that's a really excellent point.

merlin

On Fri, Jun 15, 2012 at 07:18:34AM -0500, Merlin Moncure wrote:
> On Fri, Jun 15, 2012 at 5:48 AM, Florian Pflug <fgp(at)phlo(dot)org> wrote:
> > On Jun15, 2012, at 12:09 , Magnus Hagander wrote:
> >> On Fri, Jun 15, 2012 at 5:52 PM, Florian Pflug <fgp(at)phlo(dot)org> wrote:
> >>> On Jun15, 2012, at 07:50 , Magnus Hagander wrote:
> >>>> Second, we also have things like the JDBC driver and the .Net driver
> >>>> that don't use libpq. the JDBC driver uses the native java ssl
> >>>> support, AFAIK. Does that one support the compression, and does it
> >>>> support controlling it?
> >>>
> >>> Java uses pluggable providers with standardized interfaces for most
> >>> things related to encryption. SSL support is provided by JSSE
> >>> (Java Secure Socket Extension). The JSSE implementation included with
> >>> the oracle JRE doesn't seem to support compression according to the
> >>> wikipedia page quoted above. But chances are that there exists an
> >>> alternative implementation which does.
> >>
> >> Yeah, but that alone is IMO a rather big blocker for claiming that
> >> this is the only way to do it :( And I think the fact that that
> >> wikipedia page doesn't list any other ones, is a sign that there might
> >> not be a lot of other choices out there in reality - expecially not
> >> opensource…
> >
> > Hm, but things get even harder for the JDBC and .NET folks if we go
> > with a third-party compression method. Or would we require that the
> > existence of a free Java (and maybe .NET) implementation of such a
> > method would be an absolute must?
> >
> > The way I see it, if we use SSL-based compression then non-libpq clients
> > there's at least a chance of those clients being able to use it easily
> > (if their SSL implementation supports it). If we go with a third-party
> > compression method, they *all* need to add yet another dependency, or may
> > even need to re-implement the compression method in their implementation
> > language of choice.
>
> hm, that's a really excellent point.
>
> merlin
>

I agree and think that the SSL-based compression is an excellent default
compression scheme. The plugable compression approach allows for the
choice of the most appropriate compression implementation based on the
application needs. It really addresses corner cases such as high-
performance system.

Regards,
Ken

On Fri, Jun 15, 2012 at 6:48 PM, Florian Pflug <fgp(at)phlo(dot)org> wrote:
> On Jun15, 2012, at 12:09 , Magnus Hagander wrote:
>> On Fri, Jun 15, 2012 at 5:52 PM, Florian Pflug <fgp(at)phlo(dot)org> wrote:
>>> On Jun15, 2012, at 07:50 , Magnus Hagander wrote:
>>>> Second, we also have things like the JDBC driver and the .Net driver
>>>> that don't use libpq. the JDBC driver uses the native java ssl
>>>> support, AFAIK. Does that one support the compression, and does it
>>>> support controlling it?
>>>
>>> Java uses pluggable providers with standardized interfaces for most
>>> things related to encryption. SSL support is provided by JSSE
>>> (Java Secure Socket Extension). The JSSE implementation included with
>>> the oracle JRE doesn't seem to support compression according to the
>>> wikipedia page quoted above. But chances are that there exists an
>>> alternative implementation which does.
>>
>> Yeah, but that alone is IMO a rather big blocker for claiming that
>> this is the only way to do it :( And I think the fact that that
>> wikipedia page doesn't list any other ones, is a sign that there might
>> not be a lot of other choices out there in reality - expecially not
>> opensource…
>
> Hm, but things get even harder for the JDBC and .NET folks if we go
> with a third-party compression method. Or would we require that the
> existence of a free Java (and maybe .NET) implementation of such a
> method would be an absolute must?

As long as a free implementation exists, it can be ported to
Java/.Net. Sure, it takes more work, but it *can be done*.

> The way I see it, if we use SSL-based compression then non-libpq clients
> there's at least a chance of those clients being able to use it easily
> (if their SSL implementation supports it). If we go with a third-party
> compression method, they *all* need to add yet another dependency, or may
> even need to re-implement the compression method in their implementation
> language of choice.

I only partially agree. If there *is* no third party SSL libary that
does support it, then they're stuck reimplementing an *entire SSL
library*, which is surely many orders of magnitude more work, and
suddenly steps into writing encryption code which is a lot more
sensitive. Basically if they have to do that, then they're stuck
*never* being able to fix the problem.

If we can prove such a third party library *exists*, that makes it
different. But from what I can tell so far, I haven't seen a single
one - let alone one that supports compression.

--
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/

On 15.06.2012 17:39, Magnus Hagander wrote:
> On Fri, Jun 15, 2012 at 6:48 PM, Florian Pflug<fgp(at)phlo(dot)org> wrote:
>> The way I see it, if we use SSL-based compression then non-libpq clients
>> there's at least a chance of those clients being able to use it easily
>> (if their SSL implementation supports it). If we go with a third-party
>> compression method, they *all* need to add yet another dependency, or may
>> even need to re-implement the compression method in their implementation
>> language of choice.
>
> I only partially agree. If there *is* no third party SSL libary that
> does support it, then they're stuck reimplementing an *entire SSL
> library*, which is surely many orders of magnitude more work, and
> suddenly steps into writing encryption code which is a lot more
> sensitive.

You could write a dummy SSL implementation that only does compression,
not encryption. Ie. only support the 'null' encryption method. That
should be about the same amount of work as writing an implementation of
compression using whatever protocol we would decide to use for
negotiating the compression.

--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com

On Fri, Jun 15, 2012 at 10:56 PM, Heikki Linnakangas
<heikki(dot)linnakangas(at)enterprisedb(dot)com> wrote:
> On 15.06.2012 17:39, Magnus Hagander wrote:
>>
>> On Fri, Jun 15, 2012 at 6:48 PM, Florian Pflug<fgp(at)phlo(dot)org>  wrote:
>>>
>>> The way I see it, if we use SSL-based compression then non-libpq clients
>>>
>>> there's at least a chance of those clients being able to use it easily
>>> (if their SSL implementation supports it). If we go with a third-party
>>> compression method, they *all* need to add yet another dependency, or may
>>> even need to re-implement the compression method in their implementation
>>> language of choice.
>>
>>
>> I only partially agree. If there *is* no third party SSL libary that
>> does support it, then they're stuck reimplementing an *entire SSL
>> library*, which is surely many orders of magnitude more work, and
>> suddenly steps into writing encryption code which is a lot more
>> sensitive.
>
>
> You could write a dummy SSL implementation that only does compression, not
> encryption. Ie. only support the 'null' encryption method. That should be
> about the same amount of work as writing an implementation of compression
> using whatever protocol we would decide to use for negotiating the
> compression.

Sure, but then what do you do if you actually want both?

--
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/

On 15.06.2012 17:58, Magnus Hagander wrote:
> On Fri, Jun 15, 2012 at 10:56 PM, Heikki Linnakangas
> <heikki(dot)linnakangas(at)enterprisedb(dot)com> wrote:
>> On 15.06.2012 17:39, Magnus Hagander wrote:
>>>
>>> On Fri, Jun 15, 2012 at 6:48 PM, Florian Pflug<fgp(at)phlo(dot)org> wrote:
>>>>
>>>> The way I see it, if we use SSL-based compression then non-libpq clients
>>>>
>>>> there's at least a chance of those clients being able to use it easily
>>>> (if their SSL implementation supports it). If we go with a third-party
>>>> compression method, they *all* need to add yet another dependency, or may
>>>> even need to re-implement the compression method in their implementation
>>>> language of choice.
>>>
>>> I only partially agree. If there *is* no third party SSL libary that
>>> does support it, then they're stuck reimplementing an *entire SSL
>>> library*, which is surely many orders of magnitude more work, and
>>> suddenly steps into writing encryption code which is a lot more
>>> sensitive.
>>
>> You could write a dummy SSL implementation that only does compression, not
>> encryption. Ie. only support the 'null' encryption method. That should be
>> about the same amount of work as writing an implementation of compression
>> using whatever protocol we would decide to use for negotiating the
>> compression.
>
> Sure, but then what do you do if you actually want both?

Umm, then you use a real SSL libray, not the dummy one?

--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com

On Fri, Jun 15, 2012 at 11:24 PM, Heikki Linnakangas
<heikki(dot)linnakangas(at)enterprisedb(dot)com> wrote:
> On 15.06.2012 17:58, Magnus Hagander wrote:
>>
>> On Fri, Jun 15, 2012 at 10:56 PM, Heikki Linnakangas
>> <heikki(dot)linnakangas(at)enterprisedb(dot)com>  wrote:
>>>
>>> On 15.06.2012 17:39, Magnus Hagander wrote:
>>>>
>>>>
>>>> On Fri, Jun 15, 2012 at 6:48 PM, Florian Pflug<fgp(at)phlo(dot)org>    wrote:
>>>>>
>>>>>
>>>>> The way I see it, if we use SSL-based compression then non-libpq
>>>>> clients
>>>>>
>>>>> there's at least a chance of those clients being able to use it easily
>>>>> (if their SSL implementation supports it). If we go with a third-party
>>>>> compression method, they *all* need to add yet another dependency, or
>>>>> may
>>>>> even need to re-implement the compression method in their
>>>>> implementation
>>>>> language of choice.
>>>>
>>>>
>>>> I only partially agree. If there *is* no third party SSL libary that
>>>> does support it, then they're stuck reimplementing an *entire SSL
>>>> library*, which is surely many orders of magnitude more work, and
>>>> suddenly steps into writing encryption code which is a lot more
>>>> sensitive.
>>>
>>>
>>> You could write a dummy SSL implementation that only does compression,
>>> not
>>> encryption. Ie. only support the 'null' encryption method. That should be
>>> about the same amount of work as writing an implementation of compression
>>> using whatever protocol we would decide to use for negotiating the
>>> compression.
>>
>>
>> Sure, but then what do you do if you actually want both?
>
>
> Umm, then you use a real SSL libray, not the dummy one?

But (in this scenario, and so far nobody has proven it to be wrong)
there exists no real SSL library that does support compression.

--
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/

Magnus Hagander <magnus(at)hagander(dot)net> writes:
> On Fri, Jun 15, 2012 at 11:24 PM, Heikki Linnakangas
> <heikki(dot)linnakangas(at)enterprisedb(dot)com> wrote:
>> Umm, then you use a real SSL libray, not the dummy one?

> But (in this scenario, and so far nobody has proven it to be wrong)
> there exists no real SSL library that does support compression.

I do not think it is incumbent on this project to rectify that problem
... especially when nobody has proven that such a library exists (and
is not obsolete, nor are its authors busy fixing the lack so as to be
interoperable with openssl).

regards, tom lane

On Fri, Jun 15, 2012 at 11:28:48PM +0800, Magnus Hagander wrote:
> On Fri, Jun 15, 2012 at 11:24 PM, Heikki Linnakangas
> <heikki(dot)linnakangas(at)enterprisedb(dot)com> wrote:
> > On 15.06.2012 17:58, Magnus Hagander wrote:
> >>
> >> On Fri, Jun 15, 2012 at 10:56 PM, Heikki Linnakangas
> >> <heikki(dot)linnakangas(at)enterprisedb(dot)com>  wrote:
> >>>
> >>> On 15.06.2012 17:39, Magnus Hagander wrote:
> >>>>
> >>>>
> >>>> On Fri, Jun 15, 2012 at 6:48 PM, Florian Pflug<fgp(at)phlo(dot)org>    wrote:
> >>>>>
> >>>>>
> >>>>> The way I see it, if we use SSL-based compression then non-libpq
> >>>>> clients
> >>>>>
> >>>>> there's at least a chance of those clients being able to use it easily
> >>>>> (if their SSL implementation supports it). If we go with a third-party
> >>>>> compression method, they *all* need to add yet another dependency, or
> >>>>> may
> >>>>> even need to re-implement the compression method in their
> >>>>> implementation
> >>>>> language of choice.
> >>>>
> >>>>
> >>>> I only partially agree. If there *is* no third party SSL libary that
> >>>> does support it, then they're stuck reimplementing an *entire SSL
> >>>> library*, which is surely many orders of magnitude more work, and
> >>>> suddenly steps into writing encryption code which is a lot more
> >>>> sensitive.
> >>>
> >>>
> >>> You could write a dummy SSL implementation that only does compression,
> >>> not
> >>> encryption. Ie. only support the 'null' encryption method. That should be
> >>> about the same amount of work as writing an implementation of compression
> >>> using whatever protocol we would decide to use for negotiating the
> >>> compression.
> >>
> >>
> >> Sure, but then what do you do if you actually want both?
> >
> >
> > Umm, then you use a real SSL libray, not the dummy one?
>
> But (in this scenario, and so far nobody has proven it to be wrong)
> there exists no real SSL library that does support compression.
gnutls and openssl both support compression:

http://www.gnu.org/software/gnutls/manual/html_node/Compression-algorithms-used-in-the-record-layer.html
http://www.openssl.org/docs/apps/enc.html

-Ryan Kelly

>
> --
>  Magnus Hagander
>  Me: http://www.hagander.net/
>  Work: http://www.redpill-linpro.com/
>
> --
> Sent via pgsql-hackers mailing list (pgsql-hackers(at)postgresql(dot)org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-hackers

On 15.06.2012 18:28, Magnus Hagander wrote:
> On Fri, Jun 15, 2012 at 11:24 PM, Heikki Linnakangas
> <heikki(dot)linnakangas(at)enterprisedb(dot)com> wrote:
>> On 15.06.2012 17:58, Magnus Hagander wrote:
>>>
>>> On Fri, Jun 15, 2012 at 10:56 PM, Heikki Linnakangas
>>> <heikki(dot)linnakangas(at)enterprisedb(dot)com> wrote:
>>>>
>>>> You could write a dummy SSL implementation that only does compression,
>>>> not
>>>> encryption. Ie. only support the 'null' encryption method. That should be
>>>> about the same amount of work as writing an implementation of compression
>>>> using whatever protocol we would decide to use for negotiating the
>>>> compression.
>>>
>>> Sure, but then what do you do if you actually want both?
>>
>> Umm, then you use a real SSL libray, not the dummy one?
>
> But (in this scenario, and so far nobody has proven it to be wrong)
> there exists no real SSL library that does support compression.

Oh, I see. Then you're screwed. But I think the right solution to that
is to write/extend a Java SSL implementation to support compression, not
to invent our own in PostgreSQL. The JDK is open source nowadays.

--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com

On 15-06-2012 11:39, Magnus Hagander wrote:
> As long as a free implementation exists, it can be ported to
> Java/.Net. Sure, it takes more work, but it *can be done*.
>
Good point. IMHO, if there isn't a solution that cover all PostgreSQL (it
seems it is not), we should pick the most appropriate one for *libpq* and let
other drivers implement it at their time.

> I only partially agree. If there *is* no third party SSL libary that
> does support it, then they're stuck reimplementing an *entire SSL
> library*, which is surely many orders of magnitude more work, and
> suddenly steps into writing encryption code which is a lot more
> sensitive. Basically if they have to do that, then they're stuck
> *never* being able to fix the problem.
>
> If we can prove such a third party library *exists*, that makes it
> different. But from what I can tell so far, I haven't seen a single
> one - let alone one that supports compression.
>
Using SSL-based compression could be a solution but I would like to emphasize
that (i) I'm obligated to use cryptography library to compress data, (ii) I'm
paying the price for SSL overhead and (iii) it will confuse people when we
said that for compression we need a SSL connection or (iv) even transform the
libpq communication code into a spaghetti to support compression using SSL in
non-SSL connections).

I see the point in not adding another dependencies or reinventing the wheel
but I see more drawbacks than benefits in adopting a SSL-based compression. I
like the Farina's idea in supporting compression outside libpq but I'm ok with
adding a standard algorithm for compression (problem is that in the future
others could want to add another interesting compression algorithms).

--
Euler Taveira de Oliveira - Timbira http://www.timbira.com.br/
PostgreSQL: Consultoria, Desenvolvimento, Suporte 24x7 e Treinamento

On 15-06-2012 11:10, ktm(at)rice(dot)edu wrote:
> I agree and think that the SSL-based compression is an excellent default
> compression scheme. The plugable compression approach allows for the
> choice of the most appropriate compression implementation based on the
> application needs. It really addresses corner cases such as high-
> performance system.
>
That is my opinion too. I'm free to use to most appropriate algorithm for
compression. It is just a matter of coding an interface for my favorite
compression algorithm. We could even add some algorithms in a new contrib module.

--
Euler Taveira de Oliveira - Timbira http://www.timbira.com.br/
PostgreSQL: Consultoria, Desenvolvimento, Suporte 24x7 e Treinamento

Euler Taveira <euler(at)timbira(dot)com> writes:
> I see the point in not adding another dependencies or reinventing the wheel
> but I see more drawbacks than benefits in adopting a SSL-based compression.

In the end, judging this tradeoff is a matter of opinion, but I come to
the opposite conclusion. Transport-level compression is not part of the
core competence of this project. As such, if we have an opportunity to
farm out that work to other projects (particularly ones that we are
already relying on), we should do so. Not expend our limited resources
on re-inventing this wheel, which we'd be more likely than not to do so
less well than it's already been done.

To draw an analogy: on the basis of the arguments that have been made
about how some users might not have access to an SSL library
implementing feature X, we should drop our use of OpenSSL entirely and
re-implement transport encryption from scratch, incompatibly with OpenSSL.
Now that's obviously ridiculous, not least because it does nothing at
all to ease the pain of people who need a non-C implementation. But
arguing that we should not use OpenSSL's compression features because
some people might need to use a different SSL implementation doesn't
seem to me to be any different from that.

regards, tom lane

I wrote:
> Euler Taveira <euler(at)timbira(dot)com> writes:
>> I see the point in not adding another dependencies or reinventing the wheel
>> but I see more drawbacks than benefits in adopting a SSL-based compression.

> In the end, judging this tradeoff is a matter of opinion, but I come to
> the opposite conclusion.

BTW, there is an additional technical argument that I don't think has
been made yet. Assume that we implement our own transport compression,
and then somebody runs an SSL connection using a recent OpenSSL version
(in which, IIRC, SSL-level compression is enabled by default). Now,
OpenSSL is trying to compress already-compressed data. That's not
merely a waste of cycles but is very likely to be counterproductive,
ie recompressed data usually gets larger not smaller.

We could possibly address this by adding control logic to tell OpenSSL
not to compress ... but that's almost exactly the code you don't want
to write, just making a different option selection. And I wonder
whether SSL implementations that don't support compression will accept
a set-the-compression-option call at all.

regards, tom lane

On Fri, Jun 15, 2012 at 12:48:24PM +0200, Florian Pflug wrote:
> > Yeah, but that alone is IMO a rather big blocker for claiming that
> > this is the only way to do it :( And I think the fact that that
> > wikipedia page doesn't list any other ones, is a sign that there might
> > not be a lot of other choices out there in reality - expecially not
> > opensource…
>
> Hm, but things get even harder for the JDBC and .NET folks if we go
> with a third-party compression method. Or would we require that the
> existence of a free Java (and maybe .NET) implementation of such a
> method would be an absolute must?
>
> The way I see it, if we use SSL-based compression then non-libpq clients
> there's at least a chance of those clients being able to use it easily
> (if their SSL implementation supports it). If we go with a third-party
> compression method, they *all* need to add yet another dependency, or may
> even need to re-implement the compression method in their implementation
> language of choice.

Does OpenSSL use hardware acceleration for its compression? I know it
often does for encryption --- that would be a big reason to do
compression at the SSL layer.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ It's impossible for everything to be true. +

On Sat, Jun 16, 2012 at 12:03 AM, Heikki Linnakangas
<heikki(dot)linnakangas(at)enterprisedb(dot)com> wrote:
> On 15.06.2012 18:28, Magnus Hagander wrote:
>>
>> On Fri, Jun 15, 2012 at 11:24 PM, Heikki Linnakangas
>> <heikki(dot)linnakangas(at)enterprisedb(dot)com>  wrote:
>>>
>>> On 15.06.2012 17:58, Magnus Hagander wrote:
>>>>
>>>>
>>>> On Fri, Jun 15, 2012 at 10:56 PM, Heikki Linnakangas
>>>> <heikki(dot)linnakangas(at)enterprisedb(dot)com>    wrote:
>>>>>
>>>>>
>>>>> You could write a dummy SSL implementation that only does compression,
>>>>> not
>>>>> encryption. Ie. only support the 'null' encryption method. That should
>>>>> be
>>>>> about the same amount of work as writing an implementation of
>>>>> compression
>>>>> using whatever protocol we would decide to use for negotiating the
>>>>> compression.
>>>>
>>>>
>>>> Sure, but then what do you do if you actually want both?
>>>
>>>
>>> Umm, then you use a real SSL libray, not the dummy one?
>>
>>
>> But (in this scenario, and so far nobody has proven it to be wrong)
>> there exists no real SSL library that does support compression.
>
>
> Oh, I see. Then you're screwed. But I think the right solution to that is to
> write/extend a Java SSL implementation to support compression, not to invent
> our own in PostgreSQL. The JDK is open source nowadays.

I don't have any personal experience with it, but it's my
understanding that it's only opensource in the "published opensource
product" sense. Meaning it's not really something that solicits (or
even accepts? ast least not easily...) contributions from the outside.
And forgive me for being negative, but I think you're going to have an
even harder time to get Oracle to accept a contribution if the
motivation for having it is to make the PostgreSQL driver work
better...

--
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/

On Sat, Jun 16, 2012 at 4:04 AM, Euler Taveira <euler(at)timbira(dot)com> wrote:
> On 15-06-2012 11:39, Magnus Hagander wrote:
>> As long as a free implementation exists, it can be ported to
>> Java/.Net. Sure, it takes more work, but it *can be done*.
>>
> Good point. IMHO, if there isn't a solution that cover all PostgreSQL (it
> seems it is not), we should pick the most appropriate one for *libpq* and let
> other drivers implement it at their time.

Fair enough if we decide that - but we should make that decision
knowing that we're leaving the JDBC and .Net people in a bad position
where they are not likely to be able to implement his.

The JDBC people have a theoretical chance if the JDK is open. The .Net
people are stuck with schannel that doesn't support it at this point.
It might well do in the future (since it's in the standard); but
they're at the mercy of Microsoft.

--
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/

On Sat, Jun 16, 2012 at 6:37 AM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> I wrote:
>> Euler Taveira <euler(at)timbira(dot)com> writes:
>>> I see the point in not adding another dependencies or reinventing the wheel
>>> but I see more drawbacks than benefits in adopting a SSL-based compression.
>
>> In the end, judging this tradeoff is a matter of opinion, but I come to
>> the opposite conclusion.
>
> BTW, there is an additional technical argument that I don't think has
> been made yet.  Assume that we implement our own transport compression,
> and then somebody runs an SSL connection using a recent OpenSSL version
> (in which, IIRC, SSL-level compression is enabled by default).  Now,
> OpenSSL is trying to compress already-compressed data.  That's not
> merely a waste of cycles but is very likely to be counterproductive,
> ie recompressed data usually gets larger not smaller.
>
> We could possibly address this by adding control logic to tell OpenSSL
> not to compress ... but that's almost exactly the code you don't want
> to write, just making a different option selection.  And I wonder
> whether SSL implementations that don't support compression will accept
> a set-the-compression-option call at all.

Yes, but there's also a lot of such awkward logic we need to add if we
*do* go with the SSL library doing the compression:

For example, we can no longer trust the SSL library to always do
encryption, since we specifically want to support null encryption.
Meaning we need to teach pg_hba to treat a connection with null
encryption as hostnossl, even if it's an openssl-backed connection,
and mirrored. And in libpq, we have to make sure that a requiressl
connection *does* fail even if we have ssl, when we're using null
encryption. And we currently have no way to specify different
encryption options on a per-host basis, which is something we'd have
to do (e.g. i want to be able to say that "subnet x requires
encryption with these encryptions methods" and "subnet y doesn't
require encryption but should do compression". Which in the easiest
first look would require ssl_ciphers to be controllable from
pg_hba.conf - but that doesn't work since we don't get to pg_hba.conf
until after we've negotiated the SSL mode...

So there's quite a bit of complexity that needs to be put in there
just to deal with the fact that we're using SSL to do compression, if
we want to support it in a way that's not hackish.

--
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/

On 16-06-2012 00:43, Magnus Hagander wrote:
> For example, we can no longer trust the SSL library to always do
> encryption, since we specifically want to support null encryption.
> Meaning we need to teach pg_hba to treat a connection with null
> encryption as hostnossl, even if it's an openssl-backed connection,
> and mirrored. And in libpq, we have to make sure that a requiressl
> connection *does* fail even if we have ssl, when we're using null
> encryption. And we currently have no way to specify different
> encryption options on a per-host basis, which is something we'd have
> to do (e.g. i want to be able to say that "subnet x requires
> encryption with these encryptions methods" and "subnet y doesn't
> require encryption but should do compression". Which in the easiest
> first look would require ssl_ciphers to be controllable from
> pg_hba.conf - but that doesn't work since we don't get to pg_hba.conf
> until after we've negotiated the SSL mode...
>
> So there's quite a bit of complexity that needs to be put in there
> just to deal with the fact that we're using SSL to do compression, if
> we want to support it in a way that's not hackish.
>
That's exactly the complexity I wouldn't add to the code. I'm in favor of
experimenting an standard algorithm (zlib, for example -- let's say, it is the
easiest way to implement it) or even hooks (libpq and backend -- that seems to
be complex but less than openssl-backed connection just for compression).

--
Euler Taveira de Oliveira - Timbira http://www.timbira.com.br/
PostgreSQL: Consultoria, Desenvolvimento, Suporte 24x7 e Treinamento

On Sat, Jun 16, 2012 at 6:39 AM, Magnus Hagander <magnus(at)hagander(dot)net> wrote:
> On Sat, Jun 16, 2012 at 4:04 AM, Euler Taveira <euler(at)timbira(dot)com> wrote:
>> On 15-06-2012 11:39, Magnus Hagander wrote:
>>> As long as a free implementation exists, it can be ported to
>>> Java/.Net. Sure, it takes more work, but it *can be done*.
>>>
>> Good point. IMHO, if there isn't a solution that cover all PostgreSQL (it
>> seems it is not), we should pick the most appropriate one for *libpq* and let
>> other drivers implement it at their time.
>
> Fair enough if we decide that - but we should make that decision
> knowing that we're leaving the JDBC and .Net people in a bad position
> where they are not likely to be able to implement his.
>
> The JDBC people have a theoretical chance if the JDK is open. The .Net
> people are stuck with schannel that doesn't support it at this point.
> It might well do in the future (since it's in the standard); but
> they're at the mercy of Microsoft.

Both Java and C# are open-source enough that anybody can
take existing SSL implementation and add compression to it,
then distribute it as improved SSL library.

--
marko

Marko Kreen <markokr(at)gmail(dot)com> writes:
> On Sat, Jun 16, 2012 at 6:39 AM, Magnus Hagander <magnus(at)hagander(dot)net> wrote:
>> Fair enough if we decide that - but we should make that decision
>> knowing that we're leaving the JDBC and .Net people in a bad position
>> where they are not likely to be able to implement his.
>>
>> The JDBC people have a theoretical chance if the JDK is open. The .Net
>> people are stuck with schannel that doesn't support it at this point.
>> It might well do in the future (since it's in the standard); but
>> they're at the mercy of Microsoft.

> Both Java and C# are open-source enough that anybody can
> take existing SSL implementation and add compression to it,
> then distribute it as improved SSL library.

Possibly more to the point: that is work they might have to do, if
nobody else steps up to the plate --- and if they do end up doing it,
it could benefit other projects too. On the other hand, if we
roll-our-own transport compression solution, that is work they *will*
have to do, with no chance of sharing the effort with other projects.

BTW, as far as the .Net case goes, it took only a moment's googling
to find this:
http://openssl-net.sourceforge.net/
which is a .Net wrapper around real OpenSSL. It doesn't appear to
provide wrappers for the compression selection functions, but surely
that's just a lack of round tuits, not that it would take more than
five minutes to add them.

regards, tom lane

Magnus Hagander <magnus(at)hagander(dot)net> writes:
> Yes, but there's also a lot of such awkward logic we need to add if we
> *do* go with the SSL library doing the compression:

> For example, we can no longer trust the SSL library to always do
> encryption, since we specifically want to support null encryption.

True, but are you sure we don't need to do that anyway? What happens
today, if a non-libpq client connects with SSL and specifies null
encryption?

> And we currently have no way to specify different
> encryption options on a per-host basis, which is something we'd have
> to do (e.g. i want to be able to say that "subnet x requires
> encryption with these encryptions methods" and "subnet y doesn't
> require encryption but should do compression".

[ shrug... ] Having that sort of control over a homebrew compression
solution will *also* require a lot of control logic that does not exist
today.

> So there's quite a bit of complexity that needs to be put in there
> just to deal with the fact that we're using SSL to do compression, if
> we want to support it in a way that's not hackish.

It's not obvious to me that we actually *need* anything except the
ability to recognize that a null-encrypted SSL connection probably
shouldn't be treated as matching a hostssl line; which is not something
that requires any fundamental rearrangements, since it only requires an
after-the-fact check of what was selected. Things like "subnet x
requires encryption with these encryption methods" are features that are
sensible with our existing feature set. But we don't have that now and
nobody has asked for it, so I think you are moving the goalposts rather
unfairly by claiming that a compression-related patch needs to add it.

regards, tom lane

On Sat, Jun 16, 2012 at 12:55 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> Magnus Hagander <magnus(at)hagander(dot)net> writes:
>> Yes, but there's also a lot of such awkward logic we need to add if we
>> *do* go with the SSL library doing the compression:
>
>> For example, we can no longer trust the SSL library to always do
>> encryption, since we specifically want to support null encryption.
>
> True, but are you sure we don't need to do that anyway?  What happens
> today, if a non-libpq client connects with SSL and specifies null
> encryption?

openssl rejects the connection unless you have explicitly allowed NULL
encryption in ssl_ciphers.

Which is the only sensible default.

>> And we currently have no way to specify different
>> encryption options on a per-host basis, which is something we'd have
>> to do (e.g. i want to be able to say that "subnet x requires
>> encryption with these encryptions methods" and "subnet y doesn't
>> require encryption but should do compression".
>
> [ shrug... ]  Having that sort of control over a homebrew compression
> solution will *also* require a lot of control logic that does not exist
> today.

The important part isn't really being able to control the compression
in this. It's that we're overloading a "convenience feature"
(compression) in the settings of a security feature (encryption).
Which leads to both complex processing, and also a fairly high risk of
accidentally configuring what you wouldn't want unless we change the
interface to make it look like separate things even if they aren't.

>> So there's quite a bit of complexity that needs to be put in there
>> just to deal with the fact that we're using SSL to do compression, if
>> we want to support it in a way that's not hackish.
>
> It's not obvious to me that we actually *need* anything except the
> ability to recognize that a null-encrypted SSL connection probably
> shouldn't be treated as matching a hostssl line; which is not something
> that requires any fundamental rearrangements, since it only requires an
> after-the-fact check of what was selected.  Things like "subnet x
> requires encryption with these encryption methods" are features that are
> sensible with our existing feature set.  But we don't have that now and
> nobody has asked for it, so I think you are moving the goalposts rather
> unfairly by claiming that a compression-related patch needs to add it.

Maybe I spelled it out wrong. It does require it insofar that if we
want to use this for compression, we must *always* enable openssl on
the connection. So the "with these encryption method" boils down to
"NULL encryption only" or "whatever other standards I have for
encryption". We don't need the ability to change the "whatever other
standards" per subnet, but we need to control the
accept-NULL-encryption on a per subnet basis.

It also risks some level of information leak - assuming someone
connects with NULL encryption and we don't support it, unless we do
something particular about it, the error message will go out in
cleartext. Today, you will get a client generated error message and no
actual message crosses the wire in cleartext.

It's not that we can't deal with those things. It's just that it's
going to take some work, and some careful thought about exactly which
parts can be exposed over NULL encrypted connections.

--
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/

On Sat, Jun 16, 2012 at 12:40 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> Marko Kreen <markokr(at)gmail(dot)com> writes:
>> On Sat, Jun 16, 2012 at 6:39 AM, Magnus Hagander <magnus(at)hagander(dot)net> wrote:
>>> Fair enough if we decide that - but we should make that decision
>>> knowing that we're leaving the JDBC and .Net people in a bad position
>>> where they are not likely to be able to implement his.
>>>
>>> The JDBC people have a theoretical chance if the JDK is open. The .Net
>>> people are stuck with schannel that doesn't support it at this point.
>>> It might well do in the future (since it's in the standard); but
>>> they're at the mercy of Microsoft.
>
>> Both Java and C# are open-source enough that anybody can
>> take existing SSL implementation and add compression to it,
>> then distribute it as improved SSL library.
>
> Possibly more to the point: that is work they might have to do, if
> nobody else steps up to the plate --- and if they do end up doing it,
> it could benefit other projects too.  On the other hand, if we
> roll-our-own transport compression solution, that is work they *will*
> have to do, with no chance of sharing the effort with other projects.

True - provided said upstream (Oracle in the Java case) are interested
in accepting the patches...

If they end up having to port one of the compressoin algorithms, let's
dake LZ4 as an example, then they can certainly release that as open
source under a compatible license, thus making it available to others.

Though that's not necessarily that relevant - LZ4 already has a C#
implementation for .net, a JNI wrapper for Java.
Snappy even has a native Java implementation.

So if we went down that road, there wouldn't *be* a need to implement
it. Just the protocol parts itself, which are - compared to
implementing the actual compression in either scheme - trivial.

> BTW, as far as the .Net case goes, it took only a moment's googling
> to find this:
> http://openssl-net.sourceforge.net/
> which is a .Net wrapper around real OpenSSL.  It doesn't appear to
> provide wrappers for the compression selection functions, but surely
> that's just a lack of round tuits, not that it would take more than
> five minutes to add them.

that would then loose all the advantages that npgsql get from
schannel, such as integrated certificate management. So it can be done
- but it would AFAICT require a fairly large rearchitecture of how
security is handled, it would add a license-incompatible requirement,
and it would loose other features. But it can be done.

--
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/

Magnus Hagander <magnus(at)hagander(dot)net> writes:
> On Sat, Jun 16, 2012 at 12:55 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
>> It's not obvious to me that we actually *need* anything except the
>> ability to recognize that a null-encrypted SSL connection probably
>> shouldn't be treated as matching a hostssl line; which is not something
>> that requires any fundamental rearrangements, since it only requires an
>> after-the-fact check of what was selected.

> Maybe I spelled it out wrong. It does require it insofar that if we
> want to use this for compression, we must *always* enable openssl on
> the connection. So the "with these encryption method" boils down to
> "NULL encryption only" or "whatever other standards I have for
> encryption". We don't need the ability to change the "whatever other
> standards" per subnet, but we need to control the
> accept-NULL-encryption on a per subnet basis.

After sleeping on it, I wonder if we couldn't redefine the existing
"list of acceptable ciphers" option as the "list of ciphers that are
considered to provide encrypted transport". So you'd be allowed to
connect with SSL using any unapproved cipher (including NULL), the
backend just considers it as equivalent to a non-SSL connection for
pg_hba purposes. Then no change is needed in any configuration stuff.

regards, tom lane

On Sat, Jun 16, 2012 at 11:15:30AM -0400, Tom Lane wrote:
> Magnus Hagander <magnus(at)hagander(dot)net> writes:
> > On Sat, Jun 16, 2012 at 12:55 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> >> It's not obvious to me that we actually *need* anything except the
> >> ability to recognize that a null-encrypted SSL connection probably
> >> shouldn't be treated as matching a hostssl line; which is not something
> >> that requires any fundamental rearrangements, since it only requires an
> >> after-the-fact check of what was selected.
>
> > Maybe I spelled it out wrong. It does require it insofar that if we
> > want to use this for compression, we must *always* enable openssl on
> > the connection. So the "with these encryption method" boils down to
> > "NULL encryption only" or "whatever other standards I have for
> > encryption". We don't need the ability to change the "whatever other
> > standards" per subnet, but we need to control the
> > accept-NULL-encryption on a per subnet basis.
>
> After sleeping on it, I wonder if we couldn't redefine the existing
> "list of acceptable ciphers" option as the "list of ciphers that are
> considered to provide encrypted transport". So you'd be allowed to
> connect with SSL using any unapproved cipher (including NULL), the
> backend just considers it as equivalent to a non-SSL connection for
> pg_hba purposes. Then no change is needed in any configuration stuff.
>
> regards, tom lane
>

+1 That is nice and clean.

Regards,
Ken

On Sat, Jun 16, 2012 at 11:15 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> Magnus Hagander <magnus(at)hagander(dot)net> writes:
>> On Sat, Jun 16, 2012 at 12:55 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
>>> It's not obvious to me that we actually *need* anything except the
>>> ability to recognize that a null-encrypted SSL connection probably
>>> shouldn't be treated as matching a hostssl line; which is not something
>>> that requires any fundamental rearrangements, since it only requires an
>>> after-the-fact check of what was selected.
>
>> Maybe I spelled it out wrong. It does require it insofar that if we
>> want to use this for compression, we must *always* enable openssl on
>> the connection. So the "with these encryption method" boils down to
>> "NULL encryption only" or "whatever other standards I have for
>> encryption". We don't need the ability to change the "whatever other
>> standards" per subnet, but we need to control the
>> accept-NULL-encryption on a per subnet basis.
>
> After sleeping on it, I wonder if we couldn't redefine the existing
> "list of acceptable ciphers" option as the "list of ciphers that are
> considered to provide encrypted transport".  So you'd be allowed to
> connect with SSL using any unapproved cipher (including NULL), the
> backend just considers it as equivalent to a non-SSL connection for
> pg_hba purposes.  Then no change is needed in any configuration stuff.

That seems reasonable. In looking at our current defaults, two things hit me:

Is there a reason why we don't have a parameter on the client
mirroring ssl_ciphers?

and

Shouldn't our default SSL methods include !aNULL, meaning by default
we exclude all ciphers that don't provide authentication (which means
they can be man-in-the-middle'd). AFACIT, eNULL/NULL is disabled by
default unless explicitly enabled, but aNULL isn't..

I don't think it matters from a pure security perspective since we
look inside the actual cert anyway (which shouldn't work with these
methods, I think), but it seems like a wrong default.

That, or just have DEFAULT as being the default (which in current
openssl means ALL:!aNULL:!eNULL.

--
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/

Magnus Hagander <magnus(at)hagander(dot)net> writes:
> Is there a reason why we don't have a parameter on the client
> mirroring ssl_ciphers?

Dunno, do we need one? I am not sure what the cipher negotiation process
looks like or which side has the freedom to choose.

> That, or just have DEFAULT as being the default (which in current
> openssl means ALL:!aNULL:!eNULL.

If our default isn't the same as the underlying default, I have to
question why not. But are you sure this "!" notation will work with
all openssl versions?

regards, tom lane

On Sun, Jun 17, 2012 at 11:42 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> Magnus Hagander <magnus(at)hagander(dot)net> writes:
>> Is there a reason why we don't have a parameter on the client
>> mirroring ssl_ciphers?
>
> Dunno, do we need one?  I am not sure what the cipher negotiation process
> looks like or which side has the freedom to choose.

I haven't looked into the details, but it seems reasonable that
*either* side should be able to at least define a list of ciphers it
*doens't* want to talk with.

Do we need it - well, it makes sense for the client to be able to say
"I won't trust 56-bit encryption" before it sends over the password,
imo..

>> That, or just have DEFAULT as being the default (which in current
>> openssl means ALL:!aNULL:!eNULL.
>
> If our default isn't the same as the underlying default, I have to
> question why not.

Yeah, that's exaclty what I'm questioning here..

>  But are you sure this "!" notation will work with
> all openssl versions?

Uh. We have the ! notation in our default *now*. What openssl also
supports is the text "DEFAULT", which is currently the equivalent of
"ALL!aNULL!eNULL". The question, which is valid of course, should be
if "DEFAULT" works with all openssl versions.

It would seem reasonable it does, but I haven't investigated.

--
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/

On Sun, Jun 17, 2012 at 4:45 PM, Magnus Hagander <magnus(at)hagander(dot)net> wrote:
> On Sun, Jun 17, 2012 at 11:42 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
>> Magnus Hagander <magnus(at)hagander(dot)net> writes:
>>> Is there a reason why we don't have a parameter on the client
>>> mirroring ssl_ciphers?
>>
>> Dunno, do we need one?  I am not sure what the cipher negotiation process
>> looks like or which side has the freedom to choose.
>
> I haven't looked into the details, but it seems reasonable that
> *either* side should be able to at least define a list of ciphers it
> *doens't* want to talk with.
>
> Do we need it - well, it makes sense for the client to be able to say
> "I won't trust 56-bit encryption" before it sends over the password,
> imo..

I would certainly like to see that.

--
Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On Jun16, 2012, at 17:15 , Tom Lane wrote:
> Magnus Hagander <magnus(at)hagander(dot)net> writes:
>> On Sat, Jun 16, 2012 at 12:55 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
>>> It's not obvious to me that we actually *need* anything except the
>>> ability to recognize that a null-encrypted SSL connection probably
>>> shouldn't be treated as matching a hostssl line; which is not something
>>> that requires any fundamental rearrangements, since it only requires an
>>> after-the-fact check of what was selected.
>
>> Maybe I spelled it out wrong. It does require it insofar that if we
>> want to use this for compression, we must *always* enable openssl on
>> the connection. So the "with these encryption method" boils down to
>> "NULL encryption only" or "whatever other standards I have for
>> encryption". We don't need the ability to change the "whatever other
>> standards" per subnet, but we need to control the
>> accept-NULL-encryption on a per subnet basis.
>
> After sleeping on it, I wonder if we couldn't redefine the existing
> "list of acceptable ciphers" option as the "list of ciphers that are
> considered to provide encrypted transport". So you'd be allowed to
> connect with SSL using any unapproved cipher (including NULL), the
> backend just considers it as equivalent to a non-SSL connection for
> pg_hba purposes. Then no change is needed in any configuration stuff.

Would we still tell openssl to only negotiate ciphers in the configured
list of available ciphers + NULL? If not, what happens if a connection
happens to use a cipher that is actually stronger than any cipher on
the "list of acceptable ciphers" list? The DBA wouldn't necessarily be
aware that such a cipher even exists, since it could have been made
available by an openssl upgrade…

But if we restrict the negotiable ciphers to the configure list + NULL,
then we're good I think.

best regards,
Florian Pflug

Florian Pflug <fgp(at)phlo(dot)org> writes:
> Would we still tell openssl to only negotiate ciphers in the configured
> list of available ciphers + NULL? If not, what happens if a connection
> happens to use a cipher that is actually stronger than any cipher on
> the "list of acceptable ciphers" list? The DBA wouldn't necessarily be
> aware that such a cipher even exists, since it could have been made
> available by an openssl upgrade

So? If the DBA has gone so far as to list specific ciphers, who are
we to second guess his judgment? It's not for us to decide that cipher
X is "stronger" than the ones he listed.

> But if we restrict the negotiable ciphers to the configure list + NULL,
> then we're good I think.

The fly in the ointment with any of these ideas is that the "configure
list" is not a list of exact cipher names, as per Magnus' comment that
the current default includes tests like "!aNULL". I am not sure that
we know how to evaluate such conditions if we are applying an
after-the-fact check on the selected cipher. Does OpenSSL expose any
API for evaluating whether a selected cipher meets such a test?

regards, tom lane

On 17-06-2012 12:42, Tom Lane wrote:
> If our default isn't the same as the underlying default, I have to
> question why not. But are you sure this "!" notation will work with
> all openssl versions?
>
What is all for you? It seems we don't claim support for an specific version
or later in docs or even configure. But looking at an old version (0.9.7,
2003-12-31), it seems to support "!" notation.

--
Euler Taveira de Oliveira - Timbira http://www.timbira.com.br/
PostgreSQL: Consultoria, Desenvolvimento, Suporte 24x7 e Treinamento

On 17-06-2012 12:45, Magnus Hagander wrote:
> Uh. We have the ! notation in our default *now*. What openssl also
> supports is the text "DEFAULT", which is currently the equivalent of
> "ALL!aNULL!eNULL". The question, which is valid of course, should be
> if "DEFAULT" works with all openssl versions.
>
AFAICS, "DEFAULT" works for ancient openssl versions (~10 years ago).

--
Euler Taveira de Oliveira - Timbira http://www.timbira.com.br/
PostgreSQL: Consultoria, Desenvolvimento, Suporte 24x7 e Treinamento

On 17-06-2012 12:45, Magnus Hagander wrote:
> On Sun, Jun 17, 2012 at 11:42 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
>> Magnus Hagander <magnus(at)hagander(dot)net> writes:
>>> Is there a reason why we don't have a parameter on the client
>>> mirroring ssl_ciphers?
>>
>> Dunno, do we need one? I am not sure what the cipher negotiation process
>> looks like or which side has the freedom to choose.
>
Both. Client sends a cipher list and the server determines which cipher is
used getting the first supported cipher in the client list.

> I haven't looked into the details, but it seems reasonable that
> *either* side should be able to at least define a list of ciphers it
> *doens't* want to talk with.
>
+1.

--
Euler Taveira de Oliveira - Timbira http://www.timbira.com.br/
PostgreSQL: Consultoria, Desenvolvimento, Suporte 24x7 e Treinamento

On Sun, Jun 17, 2012 at 12:29:53PM -0400, Tom Lane wrote:
> The fly in the ointment with any of these ideas is that the "configure
> list" is not a list of exact cipher names, as per Magnus' comment that
> the current default includes tests like "!aNULL". I am not sure that
> we know how to evaluate such conditions if we are applying an
> after-the-fact check on the selected cipher. Does OpenSSL expose any
> API for evaluating whether a selected cipher meets such a test?

I'm not sure whether there's an API for it, but you can certainly check
manually with "openssl ciphers -v", for example:

$ openssl ciphers -v 'ALL:!ADH:RC4+RSA:+HIGH:+MEDIUM:+LOW:+SSLv2:+EXP'
NULL-SHA SSLv3 Kx=RSA Au=RSA Enc=None Mac=SHA1
NULL-MD5 SSLv3 Kx=RSA Au=RSA Enc=None Mac=MD5

...etc...

So unless the openssl includes the code twice there must be a way to
extract the list from the library.

Have a nice ay,
--
Martijn van Oosterhout <kleptog(at)svana(dot)org> http://svana.org/kleptog/
> He who writes carelessly confesses thereby at the very outset that he does
> not attach much importance to his own thoughts.
-- Arthur Schopenhauer

On Mon, Jun 18, 2012 at 1:42 PM, Martijn van Oosterhout
<kleptog(at)svana(dot)org> wrote:
> On Sun, Jun 17, 2012 at 12:29:53PM -0400, Tom Lane wrote:
>> The fly in the ointment with any of these ideas is that the "configure
>> list" is not a list of exact cipher names, as per Magnus' comment that
>> the current default includes tests like "!aNULL".  I am not sure that
>> we know how to evaluate such conditions if we are applying an
>> after-the-fact check on the selected cipher.  Does OpenSSL expose any
>> API for evaluating whether a selected cipher meets such a test?
>
> I'm not sure whether there's an API for it, but you can certainly check
> manually with "openssl ciphers -v", for example:
>
> $ openssl ciphers -v 'ALL:!ADH:RC4+RSA:+HIGH:+MEDIUM:+LOW:+SSLv2:+EXP'
> NULL-SHA                SSLv3 Kx=RSA      Au=RSA  Enc=None      Mac=SHA1
> NULL-MD5                SSLv3 Kx=RSA      Au=RSA  Enc=None      Mac=MD5
>
> ...etc...
>
> So unless the openssl includes the code twice there must be a way to
> extract the list from the library.

There doubtless is, but I'd being willing to wager that you won't be
able to figure out the exact method without reading the source code
for 'opennssl ciphers' to see how it was done there, and most likely
you'll find that at least one of the functions they use has no man
page. Documentation isn't their strong point.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On Jun19, 2012, at 17:36 , Robert Haas wrote:
> On Mon, Jun 18, 2012 at 1:42 PM, Martijn van Oosterhout
> <kleptog(at)svana(dot)org> wrote:
>> On Sun, Jun 17, 2012 at 12:29:53PM -0400, Tom Lane wrote:
>>> The fly in the ointment with any of these ideas is that the "configure
>>> list" is not a list of exact cipher names, as per Magnus' comment that
>>> the current default includes tests like "!aNULL". I am not sure that
>>> we know how to evaluate such conditions if we are applying an
>>> after-the-fact check on the selected cipher. Does OpenSSL expose any
>>> API for evaluating whether a selected cipher meets such a test?
>>
>> I'm not sure whether there's an API for it, but you can certainly check
>> manually with "openssl ciphers -v", for example:
>>
>> $ openssl ciphers -v 'ALL:!ADH:RC4+RSA:+HIGH:+MEDIUM:+LOW:+SSLv2:+EXP'
>> NULL-SHA SSLv3 Kx=RSA Au=RSA Enc=None Mac=SHA1
>> NULL-MD5 SSLv3 Kx=RSA Au=RSA Enc=None Mac=MD5
>>
>> ...etc...
>>
>> So unless the openssl includes the code twice there must be a way to
>> extract the list from the library.
>
> There doubtless is, but I'd being willing to wager that you won't be
> able to figure out the exact method without reading the source code
> for 'opennssl ciphers' to see how it was done there, and most likely
> you'll find that at least one of the functions they use has no man
> page. Documentation isn't their strong point.

Yes, unfortunately.

I wonder though if shouldn't restrict the allowed ciphers list to being
a simple list of supported ciphers. If our goal is to support multiple
SSL libraries transparently then surely having openssl-specific syntax
in the config file isn't exactly great anyway...

best regards,
Florian Pflug

Excerpts from Florian Pflug's message of mié jun 20 06:35:29 -0400 2012:
> On Jun19, 2012, at 17:36 , Robert Haas wrote:
> > On Mon, Jun 18, 2012 at 1:42 PM, Martijn van Oosterhout
> > <kleptog(at)svana(dot)org> wrote:
> >> On Sun, Jun 17, 2012 at 12:29:53PM -0400, Tom Lane wrote:
> >>> The fly in the ointment with any of these ideas is that the "configure
> >>> list" is not a list of exact cipher names, as per Magnus' comment that
> >>> the current default includes tests like "!aNULL". I am not sure that
> >>> we know how to evaluate such conditions if we are applying an
> >>> after-the-fact check on the selected cipher. Does OpenSSL expose any
> >>> API for evaluating whether a selected cipher meets such a test?
> >>
> >> I'm not sure whether there's an API for it, but you can certainly check
> >> manually with "openssl ciphers -v", for example:
> >>
> >> $ openssl ciphers -v 'ALL:!ADH:RC4+RSA:+HIGH:+MEDIUM:+LOW:+SSLv2:+EXP'
> >> NULL-SHA SSLv3 Kx=RSA Au=RSA Enc=None Mac=SHA1
> >> NULL-MD5 SSLv3 Kx=RSA Au=RSA Enc=None Mac=MD5
> >>
> >> ...etc...
> >>
> >> So unless the openssl includes the code twice there must be a way to
> >> extract the list from the library.
> >
> > There doubtless is, but I'd being willing to wager that you won't be
> > able to figure out the exact method without reading the source code
> > for 'opennssl ciphers' to see how it was done there, and most likely
> > you'll find that at least one of the functions they use has no man
> > page. Documentation isn't their strong point.
>
> Yes, unfortunately.

I looked at the code (apps/ciphers.c) and it looks pretty easy to obtain
the list of ciphers starting from the stringified configuration
parameter and iterate on them. The problem is figuring out whether any
given cipher meets some criteria; all the stuff that the command prints
after the cipher name comes from a "get cipher description" API call and
it doesn't look like there's any simple way of getting the individual
bits in some better form (assuming we don't want to parse the
description string).

Now if the cipher name is enough for whatever it is that we want, then
that looks easy.

--
Álvaro Herrera <alvherre(at)commandprompt(dot)com>
The PostgreSQL Company - Command Prompt, Inc.
PostgreSQL Replication, Consulting, Custom Development, 24x7 support

Florian Pflug <fgp(at)phlo(dot)org> writes:
> I wonder though if shouldn't restrict the allowed ciphers list to being
> a simple list of supported ciphers. If our goal is to support multiple
> SSL libraries transparently then surely having openssl-specific syntax
> in the config file isn't exactly great anyway...

No, we don't want to go there, because then we'd have to worry about
keeping the default list in sync with what's supported by the particular
version of the particular library we chance to be using. That's about
as far from transparent as you can get. A notation like "DEFAULT"
is really quite ideal for our purposes in that respect.

regards, tom lane

On Jun20, 2012, at 17:34 , Tom Lane wrote:
> Florian Pflug <fgp(at)phlo(dot)org> writes:
>> I wonder though if shouldn't restrict the allowed ciphers list to being
>> a simple list of supported ciphers. If our goal is to support multiple
>> SSL libraries transparently then surely having openssl-specific syntax
>> in the config file isn't exactly great anyway...
>
> No, we don't want to go there, because then we'd have to worry about
> keeping the default list in sync with what's supported by the particular
> version of the particular library we chance to be using. That's about
> as far from transparent as you can get. A notation like "DEFAULT"
> is really quite ideal for our purposes in that respect.

No argument with that, but does that mean we have to allow the full
syntax supported by OpenSSL (i.e., those +,-,! prefixes)? Maybe we could
map an empty list to DEFAULT and otherwise interpret it as a list of
ciphers?

It'd make the whole NULL-cipher business easy, because once we know that
the cipher specified doesn't contain !NULL (which removes NULL *permanently*),
we can simply append NULL to allow "all these ciphers plus NULL".

best regards,
Florian Pflug

Alvaro Herrera <alvherre(at)commandprompt(dot)com> writes:
> I looked at the code (apps/ciphers.c) and it looks pretty easy to obtain
> the list of ciphers starting from the stringified configuration
> parameter and iterate on them.

Do you mean that it will produce an expansion of the set of ciphers
meeting criteria like "!aNULL"? If so, I think we are set; we can
easily check to see if the active cipher is in that list, no?

regards, tom lane

Excerpts from Tom Lane's message of mié jun 20 11:49:51 -0400 2012:
>
> Alvaro Herrera <alvherre(at)commandprompt(dot)com> writes:
> > I looked at the code (apps/ciphers.c) and it looks pretty easy to obtain
> > the list of ciphers starting from the stringified configuration
> > parameter and iterate on them.
>
> Do you mean that it will produce an expansion of the set of ciphers
> meeting criteria like "!aNULL"?

Attached is a simple program that does that. You pass 'ALL:!aNULL' as
its first arg and it produces such a list.

> If so, I think we are set; we can
> easily check to see if the active cipher is in that list, no?

Great.

--
Álvaro Herrera <alvherre(at)commandprompt(dot)com>
The PostgreSQL Company - Command Prompt, Inc.
PostgreSQL Replication, Consulting, Custom Development, 24x7 support

On Wed, Jun 20, 2012 at 12:35 PM, Florian Pflug <fgp(at)phlo(dot)org> wrote:
> On Jun19, 2012, at 17:36 , Robert Haas wrote:
>> On Mon, Jun 18, 2012 at 1:42 PM, Martijn van Oosterhout
>> <kleptog(at)svana(dot)org> wrote:
>>> On Sun, Jun 17, 2012 at 12:29:53PM -0400, Tom Lane wrote:
>>>> The fly in the ointment with any of these ideas is that the "configure
>>>> list" is not a list of exact cipher names, as per Magnus' comment that
>>>> the current default includes tests like "!aNULL".  I am not sure that
>>>> we know how to evaluate such conditions if we are applying an
>>>> after-the-fact check on the selected cipher.  Does OpenSSL expose any
>>>> API for evaluating whether a selected cipher meets such a test?
>>>
>>> I'm not sure whether there's an API for it, but you can certainly check
>>> manually with "openssl ciphers -v", for example:
>>>
>>> $ openssl ciphers -v 'ALL:!ADH:RC4+RSA:+HIGH:+MEDIUM:+LOW:+SSLv2:+EXP'
>>> NULL-SHA                SSLv3 Kx=RSA      Au=RSA  Enc=None      Mac=SHA1
>>> NULL-MD5                SSLv3 Kx=RSA      Au=RSA  Enc=None      Mac=MD5
>>>
>>> ...etc...
>>>
>>> So unless the openssl includes the code twice there must be a way to
>>> extract the list from the library.
>>
>> There doubtless is, but I'd being willing to wager that you won't be
>> able to figure out the exact method without reading the source code
>> for 'opennssl ciphers' to see how it was done there, and most likely
>> you'll find that at least one of the functions they use has no man
>> page.  Documentation isn't their strong point.
>
> Yes, unfortunately.
>
> I wonder though if shouldn't restrict the allowed ciphers list to being
> a simple list of supported ciphers. If our goal is to support multiple
> SSL libraries transparently then surely having openssl-specific syntax
> in the config file isn't exactly great anyway...

That is a very good point. Before we design *another* feature that
relies on it, we should verify if the syntax is compatible in the
other libraries that would be interesting (gnutls and NSS primarily),
and if it's not that at least the *functionality* exists ina
compatible way. So we don't put ourselves in a position where we can't
proceed.

And yes, that's vapourware so far. But I know at least Claes (added to
CC) has said he wants to work on it during this summer, and I've
promised to help him out and review as well, if/when he gets that far.
But even without that, we should try to keep the door to these other
library implementations as open as possible.

--
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/

On Jun20, 2012, at 18:42 , Magnus Hagander wrote:
> That is a very good point. Before we design *another* feature that
> relies on it, we should verify if the syntax is compatible in the
> other libraries that would be interesting (gnutls and NSS primarily),
> and if it's not that at least the *functionality* exists ina
> compatible way. So we don't put ourselves in a position where we can't
> proceed.

Hm, here's another problem with relying on SSL/TLS for compression.
RFC2246, which defines TLS 1.0, explicitly states that

"TLS_NULL_WITH_NULL_NULL is specified and is the initial state of a
TLS connection during the first handshake on that channel, but MUST
NOT be negotiated, as it provides no more protection than an
unsecured connection." [RFC2246, A.5. The Cipher Suite]

and that paragraph is still present in RFC5246 (TLS 1.2). The other
cipher suits without actual encryption seem to be

TLS_RSA_WITH_NULL_MD5
TLS_RSA_WITH_NULL_SHA
TLS_RSA_WITH_NULL_SHA256 (TLS 1.2)

Unless I'm missing something, that leaves us with no way of skipping the
initial RSA handshake and also (more importantly) of computing a MD5
or SHA digest of every packet sent.

I'm starting to think that relying on SSL/TLS for compression of
unencrypted connections might not be such a good idea after all. We'd
be using the protocol in a way it quite clearly never was intended to
be used...

best regards,
Florian Pflug

On Wed, Jun 20, 2012 at 10:05 PM, Florian Pflug <fgp(at)phlo(dot)org> wrote:
> I'm starting to think that relying on SSL/TLS for compression of
> unencrypted connections might not be such a good idea after all. We'd
> be using the protocol in a way it quite clearly never was intended to
> be used...

Maybe, but what is the argument that we should avoid
on encryption+compression at the same time?

AES is quite lightweight compared to compression, so should
be no problem in situations where you care about compression.

RSA is noticeable, but only for short connections.
Thus easily solvable with connection pooling.

And for really special compression needs you can always
create a UDF that does custom compression for you.

So what exactly is the situation we need to solve
with postgres-specific protocol compression?

--
marko

On Jun20, 2012, at 22:40 , Marko Kreen wrote:
> On Wed, Jun 20, 2012 at 10:05 PM, Florian Pflug <fgp(at)phlo(dot)org> wrote:
>> I'm starting to think that relying on SSL/TLS for compression of
>> unencrypted connections might not be such a good idea after all. We'd
>> be using the protocol in a way it quite clearly never was intended to
>> be used...
>
> Maybe, but what is the argument that we should avoid
> on encryption+compression at the same time?

Mostly orthogonality, I think. I personally could live supporting
compression only together with encryption. We *shouldn't* require people
to create an RSA certificate just to use compression, though. I guess we
could do that by using one of the cipher suits that use Diffie-Hellman
key exchange (TLS_DH_anon_WITH_AES_128_CBC_SHA for example), but that
again requires fiddling with the allowed ciphers list. Or we could
auto-generate a self-signed RSA certificate if none is available. Which
is doable as long as we're going to stick with OpenSSL or GnuTLS, but if
we eventually want to support more SSL libraries server-side, this could
be a road blocker...

> AES is quite lightweight compared to compression, so should
> be no problem in situations where you care about compression.

Hm, yeah, that's especially true since DEFLATE (zlib) seems to be the
only universally supported compression method for TLS, which is rather
slow anyway (At least compared to alternatives such as LZO or Google's
Snappy).

> So what exactly is the situation we need to solve
> with postgres-specific protocol compression?

All we really want IMHO is a way to enable compression which requires
no more than specifying compress=on or the like in the connection
string. The million dollar question is, what is the easiest way to
get there? I initially though that relying on TLS for compression should
be relatively straight-forward, but it's starting to look rather messy…

best regards,
Florian Pflug

On 20-06-2012 17:40, Marko Kreen wrote:
> On Wed, Jun 20, 2012 at 10:05 PM, Florian Pflug <fgp(at)phlo(dot)org> wrote:
>> I'm starting to think that relying on SSL/TLS for compression of
>> unencrypted connections might not be such a good idea after all. We'd
>> be using the protocol in a way it quite clearly never was intended to
>> be used...
>
> Maybe, but what is the argument that we should avoid
> on encryption+compression at the same time?
>
> AES is quite lightweight compared to compression, so should
> be no problem in situations where you care about compression.
>
If we could solve compression problem without AES that will turn things
easier. Compression-only via encryption is a weird manner to solve the problem
in the user's POV.

> RSA is noticeable, but only for short connections.
> Thus easily solvable with connection pooling.
>
RSA overhead is not the main problem. SSL/TLS setup is.

> And for really special compression needs you can always
> create a UDF that does custom compression for you.
>
You have to own the code to modify it; it is not always an option.

> So what exactly is the situation we need to solve
> with postgres-specific protocol compression?
>
Compression only support. Why do I need to set up SSL/TLS just for compression?

IMHO SSL/TLS use is no different from relying in another library to handle
compression for the protocol and more it is compression-specific. That way, we
could implement another algorithms in such library without needing to modify
libpq code. Using SSL/TLS you are bounded by what SSL/TLS software products
decide to use as compression algorithms. I'll be happy to maintain the code
iif it is postgres-specific or even as close as possible to core.

--
Euler Taveira de Oliveira - Timbira http://www.timbira.com.br/
PostgreSQL: Consultoria, Desenvolvimento, Suporte 24x7 e Treinamento

On Fri, Jun 22, 2012 at 12:38 PM, Euler Taveira <euler(at)timbira(dot)com> wrote:
> On 20-06-2012 17:40, Marko Kreen wrote:
>> On Wed, Jun 20, 2012 at 10:05 PM, Florian Pflug <fgp(at)phlo(dot)org> wrote:
>>> I'm starting to think that relying on SSL/TLS for compression of
>>> unencrypted connections might not be such a good idea after all. We'd
>>> be using the protocol in a way it quite clearly never was intended to
>>> be used...
>>
>> Maybe, but what is the argument that we should avoid
>> on encryption+compression at the same time?
>>
>> AES is quite lightweight compared to compression, so should
>> be no problem in situations where you care about compression.
>>
> If we could solve compression problem without AES that will turn things
> easier. Compression-only via encryption is a weird manner to solve the problem
> in the user's POV.
>
>> RSA is noticeable, but only for short connections.
>> Thus easily solvable with connection pooling.
>>
> RSA overhead is not the main problem. SSL/TLS setup is.
>
>> And for really special compression needs you can always
>> create a UDF that does custom compression for you.
>>
> You have to own the code to modify it; it is not always an option.
>
>> So what exactly is the situation we need to solve
>> with postgres-specific protocol compression?
>>
> Compression only support. Why do I need to set up SSL/TLS just for compression?
>
> IMHO SSL/TLS use is no different from relying in another library to handle
> compression for the protocol and more it is compression-specific. That way, we
> could implement another algorithms in such library without needing to modify
> libpq code. Using SSL/TLS you are bounded by what SSL/TLS software products
> decide to use as compression algorithms. I'll be happy to maintain the code
> iif it is postgres-specific or even as close as possible to core.

I guess my feeling on this is that, so far as I can see, supporting
compression via OpenSSL involves work and trade-offs, and supporting
it without depending on OpenSSL also involves work, and trade-offs.
So it's not real evident to me that we should prefer one to the other
on general principle. It seems to me that a lot might come down to
performance. If someone can demonstrate that using an external
library involves gets significantly better compression, chews up
significantly less CPU time, and/or is significantly less code than
supporting this via OpenSSL, then maybe we ought to consider it.
OpenSSL is kind of an ugly piece of code, and all things being equal
depending on it more heavily would not be my first choice.

On the other hand, this does not seem to me to be a situation where we
should accept a patch to use an external library just because someone
takes the time to write one, because there is also a non-trivial cost
for the entire project to depending on more things; or if the
compression code gets put into the backend, then there's a cost to us
to maintain that code inside our source base. So I think we really
need someone to try this both ways and compare. Right now it seems
like we're mostly speculating on how well either approach would work,
which is not as good as having some experimental results. If, for
example, someone can demonstrate that an awesomebsdlz compresses 10x
as fast as OpenSSL... that'd be pretty compelling.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On Mon, Jun 25, 2012 at 4:04 AM, Robert Haas <robertmhaas(at)gmail(dot)com> wrote:
> On Fri, Jun 22, 2012 at 12:38 PM, Euler Taveira <euler(at)timbira(dot)com> wrote:
>> On 20-06-2012 17:40, Marko Kreen wrote:
>>> On Wed, Jun 20, 2012 at 10:05 PM, Florian Pflug <fgp(at)phlo(dot)org> wrote:
>>>> I'm starting to think that relying on SSL/TLS for compression of
>>>> unencrypted connections might not be such a good idea after all. We'd
>>>> be using the protocol in a way it quite clearly never was intended to
>>>> be used...
>>>
>>> Maybe, but what is the argument that we should avoid
>>> on encryption+compression at the same time?
>>>
>>> AES is quite lightweight compared to compression, so should
>>> be no problem in situations where you care about compression.
>>>
>> If we could solve compression problem without AES that will turn things
>> easier. Compression-only via encryption is a weird manner to solve the problem
>> in the user's POV.
>>
>>> RSA is noticeable, but only for short connections.
>>> Thus easily solvable with connection pooling.
>>>
>> RSA overhead is not the main problem. SSL/TLS setup is.
>>
>>> And for really special compression needs you can always
>>> create a UDF that does custom compression for you.
>>>
>> You have to own the code to modify it; it is not always an option.
>>
>>> So what exactly is the situation we need to solve
>>> with postgres-specific protocol compression?
>>>
>> Compression only support. Why do I need to set up SSL/TLS just for compression?
>>
>> IMHO SSL/TLS use is no different from relying in another library to handle
>> compression for the protocol and more it is compression-specific. That way, we
>> could implement another algorithms in such library without needing to modify
>> libpq code. Using SSL/TLS you are bounded by what SSL/TLS software products
>> decide to use as compression algorithms. I'll be happy to maintain the code
>> iif it is postgres-specific or even as close as possible to core.
>
> I guess my feeling on this is that, so far as I can see, supporting
> compression via OpenSSL involves work and trade-offs, and supporting
> it without depending on OpenSSL also involves work, and trade-offs.

Nice summary :)

> So it's not real evident to me that we should prefer one to the other
> on general principle.  It seems to me that a lot might come down to
> performance.  If someone can demonstrate that using an external
> library involves gets significantly better compression, chews up
> significantly less CPU time, and/or is significantly less code than
> supporting this via OpenSSL, then maybe we ought to consider it.

I think we should, yes. But as you say, we need to know first. It's
also a question of if one of these compression schemes are trivial
enough that we could embed the code rather than rely on it externally
- I have no idea if that's even remotely possibe, but that would move
the goalposts a bit too.

> OpenSSL is kind of an ugly piece of code, and all things being equal
> depending on it more heavily would not be my first choice.

Indeed.

But we should really stop saying "rely on openssl" and start saying
"rely on the ssl library". There are other SSL libraries which are not
quite so ugly, which we should eventually support.

That said, it's *still* a bit ugly to rely on the SSL library for this, IMO.

> On the other hand, this does not seem to me to be a situation where we
> should accept a patch to use an external library just because someone
> takes the time to write one, because there is also a non-trivial cost
> for the entire project to depending on more things; or if the
> compression code gets put into the backend, then there's a cost to us
> to maintain that code inside our source base.  So I think we really
> need someone to try this both ways and compare.  Right now it seems
> like we're mostly speculating on how well either approach would work,
> which is not as good as having some experimental results.  If, for
> example, someone can demonstrate that an awesomebsdlz compresses 10x
> as fast as OpenSSL...  that'd be pretty compelling.

Or that it takes less code/generates cleaner code...

--
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/

On Jun25, 2012, at 04:04 , Robert Haas wrote:
> If, for
> example, someone can demonstrate that an awesomebsdlz compresses 10x
> as fast as OpenSSL... that'd be pretty compelling.

That, actually, is demonstrably the case for at least Google's snappy.
(and LZO, but that's not an option since its license is GPL) They state in
their documentation that

In our tests, Snappy usually is faster than algorithms in the same class
(e.g. LZO, LZF, FastLZ, QuickLZ, etc.) while achieving comparable
compression ratios.

The only widely supported compression method for SSL seems to be DEFLATE,
which is also what gzip/zlib uses. I've benchmarked LZO against gzip/zlib
a few months ago, and LZO outperformed zlib in fast mode (i.e. gzip -1) by
an order of magnitude.

The compression ratio achieved by DEFLATE/gzip/zlib is much better, though.
The snappy documentation states

Typical compression ratios (based on the benchmark suite) are about
1.5-1.7x for plain text, about 2-4x for HTML, and of course 1.0x for
JPEGs, PNGs and other already-compressed data. Similar numbers for zlib
in its fastest mode are 2.6-2.8x, 3-7x and 1.0x, respectively.

Here are a few numbers for LZO vs. gzip. Snappy should be comparable to
LZO - I tested LZO because I still had the command-line compressor lzop
lying around on my machine, whereas I'd have needed to download and compile
snappy first.

$ dd if=/dev/random of=data bs=1m count=128
$ time gzip -1 < data > data.gz
real 0m6.189s
user 0m5.947s
sys 0m0.224s
$ time lzop < data > data.lzo
real 0m2.697s
user 0m0.295s
sys 0m0.224s
$ ls -lh data*
-rw-r--r-- 1 fgp staff 128M Jun 25 14:43 data
-rw-r--r-- 1 fgp staff 128M Jun 25 14:44 data.gz
-rw-r--r-- 1 fgp staff 128M Jun 25 14:44 data.lzo

$ dd if=/dev/zero of=zeros bs=1m count=128
$ time gzip -1 < zeros > zeros.gz
real 0m1.083s
user 0m1.019s
sys 0m0.052s
$ time lzop < zeros > zeros.lzo
real 0m0.186s
user 0m0.123s
sys 0m0.053s
$ ls -lh zeros*
-rw-r--r-- 1 fgp staff 128M Jun 25 14:47 zeros
-rw-r--r-- 1 fgp staff 572K Jun 25 14:47 zeros.gz
-rw-r--r-- 1 fgp staff 598K Jun 25 14:47 zeros.lzo

To summarize, on my 2.66 Ghz Core2 Duo Macbook Pro, LZO compresses about
350MB/s if the data is purely random, and about 800MB/s if the data
compresses extremely well. (Numbers based on user time since that indicates
the CPU time used, and ignores the IO overhead, which is substantial)

IMHO, the only compelling argument (and a very compelling one) to use
SSL compression was that it requires very little code on our side. We've
since discovered that it's not actually that simple, at least if we want
to support compression without authentication or encryption, and don't
want to restrict ourselves to using OpenSSL forever. So unless we give
up at least one of those requirements, the arguments for using
SSL-compression are rather thin, I think.

best regards,
Florian Pflug

On Mon, Jun 25, 2012 at 03:12:46PM +0200, Florian Pflug wrote:
> On Jun25, 2012, at 04:04 , Robert Haas wrote:
> > If, for
> > example, someone can demonstrate that an awesomebsdlz compresses 10x
> > as fast as OpenSSL... that'd be pretty compelling.
>
> That, actually, is demonstrably the case for at least Google's snappy.
> (and LZO, but that's not an option since its license is GPL) They state in
> their documentation that
>
> In our tests, Snappy usually is faster than algorithms in the same class
> (e.g. LZO, LZF, FastLZ, QuickLZ, etc.) while achieving comparable
> compression ratios.
>
> The only widely supported compression method for SSL seems to be DEFLATE,
> which is also what gzip/zlib uses. I've benchmarked LZO against gzip/zlib
> a few months ago, and LZO outperformed zlib in fast mode (i.e. gzip -1) by
> an order of magnitude.
>
> The compression ratio achieved by DEFLATE/gzip/zlib is much better, though.
> The snappy documentation states
>
> Typical compression ratios (based on the benchmark suite) are about
> 1.5-1.7x for plain text, about 2-4x for HTML, and of course 1.0x for
> JPEGs, PNGs and other already-compressed data. Similar numbers for zlib
> in its fastest mode are 2.6-2.8x, 3-7x and 1.0x, respectively.
>
> Here are a few numbers for LZO vs. gzip. Snappy should be comparable to
> LZO - I tested LZO because I still had the command-line compressor lzop
> lying around on my machine, whereas I'd have needed to download and compile
> snappy first.
>
> $ dd if=/dev/random of=data bs=1m count=128
> $ time gzip -1 < data > data.gz
> real 0m6.189s
> user 0m5.947s
> sys 0m0.224s
> $ time lzop < data > data.lzo
> real 0m2.697s
> user 0m0.295s
> sys 0m0.224s
> $ ls -lh data*
> -rw-r--r-- 1 fgp staff 128M Jun 25 14:43 data
> -rw-r--r-- 1 fgp staff 128M Jun 25 14:44 data.gz
> -rw-r--r-- 1 fgp staff 128M Jun 25 14:44 data.lzo
>
> $ dd if=/dev/zero of=zeros bs=1m count=128
> $ time gzip -1 < zeros > zeros.gz
> real 0m1.083s
> user 0m1.019s
> sys 0m0.052s
> $ time lzop < zeros > zeros.lzo
> real 0m0.186s
> user 0m0.123s
> sys 0m0.053s
> $ ls -lh zeros*
> -rw-r--r-- 1 fgp staff 128M Jun 25 14:47 zeros
> -rw-r--r-- 1 fgp staff 572K Jun 25 14:47 zeros.gz
> -rw-r--r-- 1 fgp staff 598K Jun 25 14:47 zeros.lzo
>
> To summarize, on my 2.66 Ghz Core2 Duo Macbook Pro, LZO compresses about
> 350MB/s if the data is purely random, and about 800MB/s if the data
> compresses extremely well. (Numbers based on user time since that indicates
> the CPU time used, and ignores the IO overhead, which is substantial)
>
> IMHO, the only compelling argument (and a very compelling one) to use
> SSL compression was that it requires very little code on our side. We've
> since discovered that it's not actually that simple, at least if we want
> to support compression without authentication or encryption, and don't
> want to restrict ourselves to using OpenSSL forever. So unless we give
> up at least one of those requirements, the arguments for using
> SSL-compression are rather thin, I think.
>
> best regards,
> Florian Pflug
>
+1 for http://code.google.com/p/lz4/ support. It has a BSD license too.
Using SSL libraries give all the complexity without any real benefit.

Regards,
Ken

On 24-06-2012 23:04, Robert Haas wrote:
> So I think we really
> need someone to try this both ways and compare. Right now it seems
> like we're mostly speculating on how well either approach would work,
> which is not as good as having some experimental results.
>
Not a problem. That's what I'm thinking too but I would like to make sure that
others don't object to general idea. Let me give it a try in both ideas...

--
Euler Taveira de Oliveira - Timbira http://www.timbira.com.br/
PostgreSQL: Consultoria, Desenvolvimento, Suporte 24x7 e Treinamento

Wasn't this more of an issue in de-coupling compression from encryption ?

On 25 Jun 2012, at 16:36, Euler Taveira wrote:

> On 24-06-2012 23:04, Robert Haas wrote:
>> So I think we really
>> need someone to try this both ways and compare. Right now it seems
>> like we're mostly speculating on how well either approach would work,
>> which is not as good as having some experimental results.
>>
> Not a problem. That's what I'm thinking too but I would like to make sure that
> others don't object to general idea. Let me give it a try in both ideas...
>
>
> --
> Euler Taveira de Oliveira - Timbira http://www.timbira.com.br/
> PostgreSQL: Consultoria, Desenvolvimento, Suporte 24x7 e Treinamento
>
> --
> Sent via pgsql-hackers mailing list (pgsql-hackers(at)postgresql(dot)org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-hackers
>

Magnus Hagander <magnus(at)hagander(dot)net> writes:
> Or that it takes less code/generates cleaner code...

So we're talking about some LZO things such as snappy from google, and
that would be another run time dependency IIUC.

I think it's time to talk about fastlz:

http://fastlz.org/
http://code.google.com/p/fastlz/source/browse/trunk/fastlz.c

551 lines of C code under MIT license, works also under windows

I guess it would be easy (and safe) enough to embed in our tree should
we decide going this way.

Regards,
--
Dimitri Fontaine
http://2ndQuadrant.fr PostgreSQL : Expertise, Formation et Support

On Jun25, 2012, at 21:21 , Dimitri Fontaine wrote:
> Magnus Hagander <magnus(at)hagander(dot)net> writes:
>> Or that it takes less code/generates cleaner code...
>
> So we're talking about some LZO things such as snappy from google, and
> that would be another run time dependency IIUC.
>
> I think it's time to talk about fastlz:
>
> http://fastlz.org/
> http://code.google.com/p/fastlz/source/browse/trunk/fastlz.c
>
> 551 lines of C code under MIT license, works also under windows
>
> I guess it would be easy (and safe) enough to embed in our tree should
> we decide going this way.

Agreed. If we extend the protocol to support compression, and not rely
on SSL, then let's pick one of these LZ77-style compressors, and let's
integrate it into our tree.

We should then also make it possible to enable compression only for
the server -> client direction. Since those types of compressions are
usually pretty easy to decompress, that reduces the amount to work
non-libpq clients have to put in to take advantage of compression.

best regards,
Florian Pflug

On Mon, Jun 25, 2012 at 3:45 PM, Florian Pflug <fgp(at)phlo(dot)org> wrote:
> On Jun25, 2012, at 21:21 , Dimitri Fontaine wrote:
>> Magnus Hagander <magnus(at)hagander(dot)net> writes:
>>> Or that it takes less code/generates cleaner code...
>>
>> So we're talking about some LZO things such as snappy from google, and
>> that would be another run time dependency IIUC.
>>
>> I think it's time to talk about fastlz:
>>
>>  http://fastlz.org/
>>  http://code.google.com/p/fastlz/source/browse/trunk/fastlz.c
>>
>>  551 lines of C code under MIT license, works also under windows
>>
>> I guess it would be easy (and safe) enough to embed in our tree should
>> we decide going this way.
>
> Agreed. If we extend the protocol to support compression, and not rely
> on SSL, then let's pick one of these LZ77-style compressors, and let's
> integrate it into our tree.
>
> We should then also make it possible to enable compression only for
> the server -> client direction. Since those types of compressions are
> usually pretty easy to decompress, that reduces the amount to work
> non-libpq clients have to put in to take advantage of compression.

+1

>
> best regards,
> Florian Pflug
>
>
> --
> Sent via pgsql-hackers mailing list (pgsql-hackers(at)postgresql(dot)org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-hackers

On Mon, Jun 25, 2012 at 09:45:26PM +0200, Florian Pflug wrote:
> On Jun25, 2012, at 21:21 , Dimitri Fontaine wrote:
> > Magnus Hagander <magnus(at)hagander(dot)net> writes:
> >> Or that it takes less code/generates cleaner code...
> >
> > So we're talking about some LZO things such as snappy from google, and
> > that would be another run time dependency IIUC.
> >
> > I think it's time to talk about fastlz:
> >
> > http://fastlz.org/
> > http://code.google.com/p/fastlz/source/browse/trunk/fastlz.c
> >
> > 551 lines of C code under MIT license, works also under windows
> >
> > I guess it would be easy (and safe) enough to embed in our tree should
> > we decide going this way.
>
> Agreed. If we extend the protocol to support compression, and not rely
> on SSL, then let's pick one of these LZ77-style compressors, and let's
> integrate it into our tree.
>
> We should then also make it possible to enable compression only for
> the server -> client direction. Since those types of compressions are
> usually pretty easy to decompress, that reduces the amount to work
> non-libpq clients have to put in to take advantage of compression.
>
> best regards,
> Florian Pflug
>

Here is the benchmark list from the Google lz4 page:

Name Ratio C.speed D.speed
LZ4 (r59) 2.084 330 915
LZO 2.05 1x_1 2.038 311 480
QuickLZ 1.5 -1 2.233 257 277
Snappy 1.0.5 2.024 227 729
LZF 2.076 197 465
FastLZ 2.030 190 420
zlib 1.2.5 -1 2.728 39 195
LZ4 HC (r66) 2.712 18 1020
zlib 1.2.5 -6 3.095 14 210

lz4 absolutely dominates on compression/decompression speed. While fastlz
is faster than zlib(-1) on compression, lz4 is almost 2X faster still.

Regards,
Ken

On 25-06-2012 16:45, Florian Pflug wrote:
> Agreed. If we extend the protocol to support compression, and not rely
> on SSL, then let's pick one of these LZ77-style compressors, and let's
> integrate it into our tree.
>
If we have an option to have it out of our tree, good; if not, let's integrate
it. I, particularly, don't see a compelling reason to integrate compression
code in our tree, I mean, if we want to support more than one algorithm, it is
clear that the overhead for maintain the compression code is too high (a lot
of my-new-compression-algorithms).

> We should then also make it possible to enable compression only for
> the server -> client direction. Since those types of compressions are
> usually pretty easy to decompress, that reduces the amount to work
> non-libpq clients have to put in to take advantage of compression.
>
I don't buy this idea. My use case (data load) will not be covered if we only
enable server -> client compression. I figure that there are use cases for
server -> client compression (replication, for example) but also there are
important use cases for client -> server (data load, for example). If you
implement decompression, why not code compression code too?

--
Euler Taveira de Oliveira - Timbira http://www.timbira.com.br/
PostgreSQL: Consultoria, Desenvolvimento, Suporte 24x7 e Treinamento

On 25-06-2012 14:30, Greg Jaskiewicz wrote:
> Wasn't this more of an issue in de-coupling compression from encryption ?
>
Let me give a try to take some conclusion. If we decide for an independent
compression code instead of an SSL-based one, that is a possibility to be
tested: SSL + SSL-based compression x SSL + our compression code. If the
latter is faster then we could discuss the possibility to disable compression
in our SSL code and put in documentation that it is recommended to enable
compression in SSL connections.

--
Euler Taveira de Oliveira - Timbira http://www.timbira.com.br/
PostgreSQL: Consultoria, Desenvolvimento, Suporte 24x7 e Treinamento

On Mon, Jun 25, 2012 at 3:38 PM, Euler Taveira <euler(at)timbira(dot)com> wrote:
> On 25-06-2012 16:45, Florian Pflug wrote:
>> Agreed. If we extend the protocol to support compression, and not rely
>> on SSL, then let's pick one of these LZ77-style compressors, and let's
>> integrate it into our tree.
>>
> If we have an option to have it out of our tree, good; if not, let's integrate
> it. I, particularly, don't see a compelling reason to integrate compression
> code in our tree, I mean, if we want to support more than one algorithm, it is
> clear that the overhead for maintain the compression code is too high (a lot
> of my-new-compression-algorithms).
>
>> We should then also make it possible to enable compression only for
>> the server -> client direction. Since those types of compressions are
>> usually pretty easy to decompress, that reduces the amount to work
>> non-libpq clients have to put in to take advantage of compression.
>>
> I don't buy this idea. My use case (data load) will not be covered if we only
> enable server -> client compression. I figure that there are use cases for
> server -> client compression (replication, for example) but also there are
> important use cases for client -> server (data load, for example). If you
> implement decompression, why not code compression code too?

+1. lz4, which is looking like a strong candidate, has c#, java,
etc. which are the main languages that are going to lag behind in
terms of protocol support. I don't think you're saving a lot by going
only one way (although you could make a case for the client to signal
interest in compression separately from decompression?)

another point:
It's been obvious for years now that zlib is somewhat of a dog in
terms of cpu usage for what it gives you. however, raw performance #s
are not the whole story -- it would be interesting to compress real
world protocol messages to/from the server in various scenarios to see
how compression would work, in particular with OLTP type queries --
for example pgbench runs, etc. That would speak a lot more to the
benefits than canned benchmarks.

merlin

merlin

On Mon, Jun 25, 2012 at 4:25 PM, ktm(at)rice(dot)edu <ktm(at)rice(dot)edu> wrote:
> On Mon, Jun 25, 2012 at 09:45:26PM +0200, Florian Pflug wrote:
>> On Jun25, 2012, at 21:21 , Dimitri Fontaine wrote:
>> > Magnus Hagander <magnus(at)hagander(dot)net> writes:
>> >> Or that it takes less code/generates cleaner code...
>> >
>> > So we're talking about some LZO things such as snappy from google, and
>> > that would be another run time dependency IIUC.
>> >
>> > I think it's time to talk about fastlz:
>> >
>> >  http://fastlz.org/
>> >  http://code.google.com/p/fastlz/source/browse/trunk/fastlz.c
>> >
>> >  551 lines of C code under MIT license, works also under windows
>> >
>> > I guess it would be easy (and safe) enough to embed in our tree should
>> > we decide going this way.
>>
>> Agreed. If we extend the protocol to support compression, and not rely
>> on SSL, then let's pick one of these LZ77-style compressors, and let's
>> integrate it into our tree.
>>
>> We should then also make it possible to enable compression only for
>> the server -> client direction. Since those types of compressions are
>> usually pretty easy to decompress, that reduces the amount to work
>> non-libpq clients have to put in to take advantage of compression.
>
> Here is the benchmark list from the Google lz4 page:
>
> Name            Ratio   C.speed D.speed
> LZ4 (r59)       2.084   330      915
> LZO 2.05 1x_1   2.038   311      480
> QuickLZ 1.5 -1  2.233   257      277
> Snappy 1.0.5    2.024   227      729
> LZF             2.076   197      465
> FastLZ          2.030   190      420
> zlib 1.2.5 -1   2.728    39      195
> LZ4 HC (r66)    2.712    18     1020
> zlib 1.2.5 -6   3.095    14      210
>
> lz4 absolutely dominates on compression/decompression speed. While fastlz
> is faster than zlib(-1) on compression, lz4 is almost 2X faster still.

At the risk of making everyone laugh at me, has anyone tested pglz? I
observe that if the compression ration and performance are good, we
might consider using it for this purpose, too, which would avoid
having to add dependencies. Conversely, if they are bad, and we
decide to support another algorithm, we might consider also using that
other algorithm, at least optionally, for the purposes for which we
now use pglz.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On 26-06-2012 12:23, Robert Haas wrote:
> At the risk of making everyone laugh at me, has anyone tested pglz? I
> observe that if the compression ration and performance are good, we
> might consider using it for this purpose, too, which would avoid
> having to add dependencies. Conversely, if they are bad, and we
> decide to support another algorithm, we might consider also using that
> other algorithm, at least optionally, for the purposes for which we
> now use pglz.
>
I'll remember to test it too.

--
Euler Taveira de Oliveira - Timbira http://www.timbira.com.br/
PostgreSQL: Consultoria, Desenvolvimento, Suporte 24x7 e Treinamento

Robert Haas <robertmhaas(at)gmail(dot)com> writes:
> On Mon, Jun 25, 2012 at 4:25 PM, ktm(at)rice(dot)edu <ktm(at)rice(dot)edu> wrote:
>> Here is the benchmark list from the Google lz4 page:
>>
>> Name Ratio C.speed D.speed
>> LZ4 (r59) 2.084 330 915
>> LZO 2.05 1x_1 2.038 311 480
>> QuickLZ 1.5 -1 2.233 257 277
>> Snappy 1.0.5 2.024 227 729
>> LZF 2.076 197 465
>> FastLZ 2.030 190 420
>> zlib 1.2.5 -1 2.728 39 195
>> LZ4 HC (r66) 2.712 18 1020
>> zlib 1.2.5 -6 3.095 14 210

>> lz4 absolutely dominates on compression/decompression speed. While fastlz
>> is faster than zlib(-1) on compression, lz4 is almost 2X faster still.

> At the risk of making everyone laugh at me, has anyone tested pglz?

Another point here is that those Google numbers (assuming that they
apply to our use-cases, a point not in evidence) utterly fail to make
the argument that zlib is not the thing to use. zlib is beating all
the others on compression ratio, often by 50%. Before making any
comparisons, you have to make some assumptions about what the network
speed is ... and unless it's pretty damn fast relative to your CPU speed
getting the better compression ratio is going to be more attractive than
saving some cycles.

regards, tom lane

On Mon, Jun 25, 2012 at 2:26 PM, Magnus Hagander <magnus(at)hagander(dot)net> wrote:
> On Mon, Jun 25, 2012 at 4:04 AM, Robert Haas <robertmhaas(at)gmail(dot)com> wrote:
>> On Fri, Jun 22, 2012 at 12:38 PM, Euler Taveira <euler(at)timbira(dot)com> wrote:
>>> On 20-06-2012 17:40, Marko Kreen wrote:
>>>> On Wed, Jun 20, 2012 at 10:05 PM, Florian Pflug <fgp(at)phlo(dot)org> wrote:
>>>>> I'm starting to think that relying on SSL/TLS for compression of
>>>>> unencrypted connections might not be such a good idea after all. We'd
>>>>> be using the protocol in a way it quite clearly never was intended to
>>>>> be used...
>>>>
>>>> Maybe, but what is the argument that we should avoid
>>>> on encryption+compression at the same time?
>>>>
>>>> AES is quite lightweight compared to compression, so should
>>>> be no problem in situations where you care about compression.
>>>>
>>> If we could solve compression problem without AES that will turn things
>>> easier. Compression-only via encryption is a weird manner to solve the problem
>>> in the user's POV.
>>>
>>>> RSA is noticeable, but only for short connections.
>>>> Thus easily solvable with connection pooling.
>>>>
>>> RSA overhead is not the main problem. SSL/TLS setup is.
>>>
>>>> And for really special compression needs you can always
>>>> create a UDF that does custom compression for you.
>>>>
>>> You have to own the code to modify it; it is not always an option.
>>>
>>>> So what exactly is the situation we need to solve
>>>> with postgres-specific protocol compression?
>>>>
>>> Compression only support. Why do I need to set up SSL/TLS just for compression?
>>>
>>> IMHO SSL/TLS use is no different from relying in another library to handle
>>> compression for the protocol and more it is compression-specific. That way, we
>>> could implement another algorithms in such library without needing to modify
>>> libpq code. Using SSL/TLS you are bounded by what SSL/TLS software products
>>> decide to use as compression algorithms. I'll be happy to maintain the code
>>> iif it is postgres-specific or even as close as possible to core.
>>
>> I guess my feeling on this is that, so far as I can see, supporting
>> compression via OpenSSL involves work and trade-offs, and supporting
>> it without depending on OpenSSL also involves work, and trade-offs.
>
> Nice summary :)
>
>> So it's not real evident to me that we should prefer one to the other
>> on general principle. It seems to me that a lot might come down to
>> performance. If someone can demonstrate that using an external
>> library involves gets significantly better compression, chews up
>> significantly less CPU time, and/or is significantly less code than
>> supporting this via OpenSSL, then maybe we ought to consider it.
>
> I think we should, yes. But as you say, we need to know first. It's
> also a question of if one of these compression schemes are trivial
> enough that we could embed the code rather than rely on it externally
> - I have no idea if that's even remotely possibe, but that would move
> the goalposts a bit too.

A followup on this thread. I was researching something else, and
stumbled across
http://www.openssl.org/docs/ssl/SSL_COMP_add_compression_method.html,
which says:

"
The TLS standard (or SSLv3) allows the integration of compression
methods into the communication. The TLS RFC does however not specify
compression methods or their corresponding identifiers, so there is
currently no compatible way to integrate compression with unknown
peers. It is therefore currently not recommended to integrate
compression into applications. Applications for non-public use may
agree on certain compression methods. Using different compression
methods with the same identifier will lead to connection failure.
"

Which I think is another reason to not go down that path as our
"official way to do compression".

--
Magnus Hagander
Me: http://www.hagander.net/
Work: http://www.redpill-linpro.com/

On Sun, Jun 17, 2012 at 11:45:54PM +0800, Magnus Hagander wrote:
> On Sun, Jun 17, 2012 at 11:42 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> > Magnus Hagander <magnus(at)hagander(dot)net> writes:
> >> Is there a reason why we don't have a parameter on the client
> >> mirroring ssl_ciphers?
> >
> > Dunno, do we need one?  I am not sure what the cipher negotiation process
> > looks like or which side has the freedom to choose.
>
> I haven't looked into the details, but it seems reasonable that
> *either* side should be able to at least define a list of ciphers it
> *doens't* want to talk with.
>
> Do we need it - well, it makes sense for the client to be able to say
> "I won't trust 56-bit encryption" before it sends over the password,
> imo..
>
>
> >> That, or just have DEFAULT as being the default (which in current
> >> openssl means ALL:!aNULL:!eNULL.
> >
> > If our default isn't the same as the underlying default, I have to
> > question why not.
>
> Yeah, that's exaclty what I'm questioning here..
>
> >  But are you sure this "!" notation will work with
> > all openssl versions?
>
> Uh. We have the ! notation in our default *now*. What openssl also
> supports is the text "DEFAULT", which is currently the equivalent of
> "ALL!aNULL!eNULL". The question, which is valid of course, should be
> if "DEFAULT" works with all openssl versions.
>
> It would seem reasonable it does, but I haven't investigated.

Do we want to change our ssl_ciphers default to 'DEFAULT'? Currently it
is 'ALL:!ADH:!LOW:!EXP:!MD5:@STRENGTH'.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ It's impossible for everything to be true. +