Compression on SSL links?

I may be blind - I don't see a way to enable this. OpenSSL "kinda"
supports this - does Postgres' SSL connectivity allow it to be
supported/enabled?

- Karl

Karl Denninger wrote:
> I may be blind - I don't see a way to enable this. OpenSSL "kinda"
> supports this - does Postgres' SSL connectivity allow it to be
> supported/enabled?

What are you asking, exactly?

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ It's impossible for everything to be true. +

On 13/08/2010 9:31 PM, Bruce Momjian wrote:
> Karl Denninger wrote:
>> I may be blind - I don't see a way to enable this. OpenSSL "kinda"
>> supports this - does Postgres' SSL connectivity allow it to be
>> supported/enabled?
>
> What are you asking, exactly?

As far as I can tell they're asking for transport-level compression,
using gzip or similar, in much the same way as SSL/TLS currently
provides transport-level encryption. Compression at the postgresql
protocol level or above, so it's invisible at the level of the libpq
APIs for executing statements and processing results, and doesn't change
SQL processing.

Since remote access is often combined with SSL, which is already
supported by libpq, using SSL-integrated compression seems pretty
promising if it's viable in practice. It'd avoid the pain of having to
add compression to the Pg protocol by putting it "outside" the current
protocol, in the SSL layer. Even better, compressing results before
encrypting them makes the encrypted traffic *much* stronger against
known-plaintext and pattern-based attacks. And, of course, compressing
the content costs CPU time but reduces the amount of data that must then
be compressed.

OpenSSL does provide some transparent crypto support. See:
http://www.openssl.org/docs/ssl/SSL_COMP_add_compression_method.html

That page notes that using compression as integrated into the SSL layer
has a few issues. The lack of compression method standardization
shouldn't be a problem for PostgreSQL, as PostgreSQL is free to define
compression methods and their names as an SSL-only extension to the
current postgresql-specific protocol. It looks like the negotiation
mechanisms to make this transparent to older clients are there, too, so
there's potential, though lots of testing would be required.

It's hard to know how widely the COMP_zlib method is enabled in deployed
copies of OpenSSL, but a configure test could probably detect its
availibility and conditionally enable it in libpq. Again, connections
would just fall back to uncompressed.

I'm curious to see if the Java SSL APIs can support SSL-level
compression, and if so whether the JDBC driver can be SSL+Gzip enabled.
I'll have a dig through the JCA (Java Cryptography Architecture) docs
and see what I can find out.

--
Craig Ringer

Craig Ringer wrote:
> On 13/08/2010 9:31 PM, Bruce Momjian wrote:
> > Karl Denninger wrote:
> >> I may be blind - I don't see a way to enable this. OpenSSL "kinda"
> >> supports this - does Postgres' SSL connectivity allow it to be
> >> supported/enabled?
> >
> > What are you asking, exactly?
>
> As far as I can tell they're asking for transport-level compression,
> using gzip or similar, in much the same way as SSL/TLS currently
> provides transport-level encryption. Compression at the postgresql
> protocol level or above, so it's invisible at the level of the libpq
> APIs for executing statements and processing results, and doesn't change
> SQL processing.
>
> Since remote access is often combined with SSL, which is already
> supported by libpq, using SSL-integrated compression seems pretty
> promising if it's viable in practice. It'd avoid the pain of having to
> add compression to the Pg protocol by putting it "outside" the current
> protocol, in the SSL layer. Even better, compressing results before
> encrypting them makes the encrypted traffic *much* stronger against
> known-plaintext and pattern-based attacks. And, of course, compressing
> the content costs CPU time but reduces the amount of data that must then
> be compressed.
>
> OpenSSL does provide some transparent crypto support. See:
> http://www.openssl.org/docs/ssl/SSL_COMP_add_compression_method.html

I thought all SSL traffic was compressed, unless you turned that off.
It is just SSH that is always compressed?

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ It's impossible for everything to be true. +

On 13/08/2010 10:43 PM, Craig Ringer wrote:

> OpenSSL does provide some transparent crypto support. See:
> http://www.openssl.org/docs/ssl/SSL_COMP_add_compression_method.html

Some more info on this:

Apache mod_ssl docs mention that SSLv3 handshake is required to
negotiate compression in SSL. That shouldn't be a big issue for
postgresql, and at worst would just require a flag to switch back to
SSLv2 for incredibly ancient servers (if there are even any out there).

http://httpd.apache.org/docs/2.0/ssl/ssl_faq.html#comp

It also shows that OpenSSL's docs are out of date, because there's a
standard way to negotiate DEFLATE compression (zlib/gzip). Not that Pg
would need that, but it's nice to have a standard name to use and it
might help with non-OpenSSL implementations like JSSE/JCA for Java/JDBC.

See: http://www.ietf.org/rfc/rfc3749.txt

Unfortunately there's no reference to DEFLATE or to compression in the
JSSE standard docs, and the latest information I can find (admittedly
from 2007) says "Compression is not supported by Sun's JSSE provider. ".

Sigh. No compression for PgJDBC SSL. There's a 3rd pty JSSE provider
called "Jessie" that claims to support SSL compression, but it looks
pretty dead and I don't see people wanting compression that badly anyway.

So: It looks like SSL/TLS level compression is supported by OpenSSL, but
I don't see much indication that it's widely adopted. I can't find any
evidence Microsoft support it in their SSL/TLS implementation either.
Win2k3's explicitly lacks it, so even if it's been added since it won't
be widely deployed.

Overall: it sounds to me like SSL/TLS level compression would only be
useful for native libpq-to-postgresql connections, and probably wouldn't
be usable for non-libpq based database access drivers. It'd only work if
SSL was configured, which is a limitation it'd be nice to avoid. Plus,
it doesn't look like it's a clear-cut obviously backward compatible
change - there's the need to use the SSLv3 handshake, issues with late
standardization of compression protocol names, the fact that
distributions of OpenSSL might not be built with compression support, etc.

Personally, after looking into it a little, I'm not even close to
interested enough to do the required patching and compatibility testing.

Adding compression to the postgresql protocol would be much harder, but
looks like it'd be much more worthwhile. It's not something I personally
care about or want to look into though.

--
Craig Ringer

On 13/08/2010 10:50 PM, Bruce Momjian wrote:

> I thought all SSL traffic was compressed, unless you turned that off.
> It is just SSH that is always compressed?

Frankly, I thought all SSL traffic was compressed too, but the reading
I've just been doing suggests otherwise. It looks like compression *is*
done as part of cyphering and hashing and is used during key exchange
etc, but the actual application data being transported isn't compressed.
At least, that's how it seems to me from the digging I've just been
doing, though I want to look into it more.

AFAIK SSH isn't always compressed. It's only compressed if you pass the
"-C" flag, set -o Compression=yes, or add Compression=yes to .ssh/config
or /etc/ssh/ssh_config .

I'll admit being surprised, as it's widely stated that crypto is much
stronger if the data to be protected is compressed first. I guess the
CPU costs are significant enough that it's not widely done for bulk
data, though, only for the critical parts like negotiating and
exchanging the session key.

--
Craig Ringer

Craig Ringer <craig(at)postnewspapers(dot)com(dot)au> writes:
> Overall: it sounds to me like SSL/TLS level compression would only be
> useful for native libpq-to-postgresql connections, and probably wouldn't
> be usable for non-libpq based database access drivers. It'd only work if
> SSL was configured, which is a limitation it'd be nice to avoid.

I don't really see that. The case where it's sensible to use
compression on the connection is where you're pushing data across
a WAN. That's also pretty much exactly the situation where it's
sensible to use encryption. I guess there'd be a use case for
compression-without-encryption if you had a trustworthy private WAN,
but who's got one of those?

So I'm much more in favor of doing something like this than doing it
directly in our own protocol.

regards, tom lane

On Fri, 2010-08-13 at 11:38 -0400, Tom Lane wrote:
> Craig Ringer <craig(at)postnewspapers(dot)com(dot)au> writes:
> > Overall: it sounds to me like SSL/TLS level compression would only be
> > useful for native libpq-to-postgresql connections, and probably wouldn't
> > be usable for non-libpq based database access drivers. It'd only work if
> > SSL was configured, which is a limitation it'd be nice to avoid.
>
> I don't really see that. The case where it's sensible to use
> compression on the connection is where you're pushing data across
> a WAN. That's also pretty much exactly the situation where it's
> sensible to use encryption. I guess there'd be a use case for
> compression-without-encryption if you had a trustworthy private WAN,
> but who's got one of those?
>
> So I'm much more in favor of doing something like this than doing it
> directly in our own protocol.

Well as a company who implemented compression for postgresql on the wire
long ago :D. I can say it is actually useful as a whole. Even on private
networks.

It reduces congestion on networks, yes even 100 and 1000 networks.

It provides data to the client faster because there is reduced data to
push.

Compression is cheap even way back then because generally the data being
compressed is comparatively small. (Imagine how fast we compress with
CPUs that were just two years older)

Not really arguing with Tom's point as much as reminding people this has
been tested and proven as a benefit.

Also, let's not forget Mobile apps :D

http://archives.postgresql.org/pgsql-hackers/2002-12/msg00598.php

I say, let's DO IT! :)

Sincerely,

Joshua D. Drake

--
PostgreSQL.org Major Contributor
Command Prompt, Inc: http://www.commandprompt.com/ - 509.416.6579
Consulting, Training, Support, Custom Development, Engineering
http://twitter.com/cmdpromptinc | http://identi.ca/commandprompt

Bruce Momjian wrote:
> Craig Ringer wrote:
>
>> On 13/08/2010 9:31 PM, Bruce Momjian wrote:
>>
>>> Karl Denninger wrote:
>>>
>>>> I may be blind - I don't see a way to enable this. OpenSSL "kinda"
>>>> supports this - does Postgres' SSL connectivity allow it to be
>>>> supported/enabled?
>>>>
>>> What are you asking, exactly?
>>>
>> As far as I can tell they're asking for transport-level compression,
>> using gzip or similar, in much the same way as SSL/TLS currently
>> provides transport-level encryption. Compression at the postgresql
>> protocol level or above, so it's invisible at the level of the libpq
>> APIs for executing statements and processing results, and doesn't change
>> SQL processing.
>>
>> Since remote access is often combined with SSL, which is already
>> supported by libpq, using SSL-integrated compression seems pretty
>> promising if it's viable in practice. It'd avoid the pain of having to
>> add compression to the Pg protocol by putting it "outside" the current
>> protocol, in the SSL layer. Even better, compressing results before
>> encrypting them makes the encrypted traffic *much* stronger against
>> known-plaintext and pattern-based attacks. And, of course, compressing
>> the content costs CPU time but reduces the amount of data that must then
>> be compressed.
>>
>> OpenSSL does provide some transparent crypto support. See:
>> http://www.openssl.org/docs/ssl/SSL_COMP_add_compression_method.html
>>
>
> I thought all SSL traffic was compressed, unless you turned that off.
> It is just SSH that is always compressed?
>
SSL is NOT always compressed at the data level. SSH is if you ask for
it, but by default SSL is NOT.

This is a common (and false) belief.

-- Karl

On Fri, Aug 13, 2010 at 10:56 AM, Joshua D. Drake <jd(at)commandprompt(dot)com> wrote:
> On Fri, 2010-08-13 at 11:38 -0400, Tom Lane wrote:
>>
>> I don't really see that.  The case where it's sensible to use
>> compression on the connection is where you're pushing data across
>> a WAN.  That's also pretty much exactly the situation where it's
>> sensible to use encryption.  I guess there'd be a use case for
>> compression-without-encryption if you had a trustworthy private WAN,
>> but who's got one of those?
>>
>> So I'm much more in favor of doing something like this than doing it
>> directly in our own protocol.
>
> Well as a company who implemented compression for postgresql on the wire
> long ago :D. I can say it is actually useful as a whole. Even on private
> networks.

We use a SSH tunnel with compression over a 2MB/s link (IPSEC VPN)
from a production site to a DR site. We run slony over it and it
works quite well. From what I can tell, average about 95%
compression. This effectively gives us a 40MB/s link for replication.

We created a Solaris SMF service to maintain the SSH tunnel and a
restricted shell on the remote end. SSH connects with a private key.
SMF for slony has required dependencies on the local postgresql and
the SSH tunnel, so it all starts up and shutdowns down in the proper
sequence.

The SSH client created one forward port tunnel and one reverse port
tunnel, both on localhost. To abuse the tunnel one would have to
already be on the database server(s), so it seems (famous last words?)
fairly secure.

I think that adding compression to the SSL layer in Postgresql's wire
protocol would be great. We could eliminate our kludgey tunnel (aka,
yet another failure point).

Just my $0.02...