pgcrypto & strong ciphers limitation

Stefan reported me that prcrypto regression test fails on solaris 10
with openssl support. I investigated this problem and the result is that
Solaris 10 delivers only support for short keys up to 128. Strong crypto
(SUNWcry and SUNWcryr packages) is available on web download pages. (It
is result of US crypto export policy.)

However, on default installation (which is commonly used) it is a
problem. Regression test cannot be fixed because it tests strong
ciphers, but there two very strange issue:

1) First issue is blowfish cipher. Because pgcrypto uses old interface
instead new "evp" it calls bf_set_key function which does not return any
output and cut key if it is too long. See
http://src.opensolaris.org/source/xref/onnv/onnv-gate/usr/src/common/openssl/crypto/bf/bf_skey.c
line 84.

If user installs strong crypto he will not be able decrypt data which
has been encrypted before.

The fix of this issue is ugly, because there is not way how to verify
supported key length with old openssl API and only new API return err if
length is not supported.

2) AES ciphere crashes when key is longer. It happens because return
value from AES_set_encrypt_key is ignored and AES_encrypt is called with
uninitialized structure.

I attach patch which fix both issues, but main problem is there that old
openssl API is used and supported key lengths are hardcoded. I think we
can add to TODO list rewrite pgcrypto to use evp openssl interface.

Any comments?

Zdenek

On 7/24/07, Zdenek Kotala <Zdenek(dot)Kotala(at)sun(dot)com> wrote:
> Stefan reported me that prcrypto regression test fails on solaris 10
> with openssl support. I investigated this problem and the result is that
> Solaris 10 delivers only support for short keys up to 128. Strong crypto
> (SUNWcry and SUNWcryr packages) is available on web download pages. (It
> is result of US crypto export policy.)

Ugh, deliberately broken OpenSSL...

> However, on default installation (which is commonly used) it is a
> problem. Regression test cannot be fixed because it tests strong
> ciphers, but there two very strange issue:
>
> 1) First issue is blowfish cipher. Because pgcrypto uses old interface
> instead new "evp" it calls bf_set_key function which does not return any
> output and cut key if it is too long. See
> http://src.opensolaris.org/source/xref/onnv/onnv-gate/usr/src/common/openssl/crypto/bf/bf_skey.c
> line 84.
>
> If user installs strong crypto he will not be able decrypt data which
> has been encrypted before.
>
> The fix of this issue is ugly, because there is not way how to verify
> supported key length with old openssl API and only new API return err if
> length is not supported.

NAK. The fix is broken because it uses EVP interface. EVP is not
a general-purpose interface because not all valid keys for cipher
pass thru it. Only key-lengths used in SSL will work...

Could you rework the fix that it uses the BF_* interface,
does a test-encoding with full-length key and compares it to
expected result. And does it just once, not on each call.

That should be put into separate function probably.

> 2) AES ciphere crashes when key is longer. It happens because return
> value from AES_set_encrypt_key is ignored and AES_encrypt is called with
> uninitialized structure.

ACK, error checking is good. But please return PXE_KEY_TOO_BIG
directly from ossl_aes_key_init.

I must admit the internal API for ciphers is clumsy and could
need rework to something saner. This shows here.

> I attach patch which fix both issues, but main problem is there that old
> openssl API is used and supported key lengths are hardcoded. I think we
> can add to TODO list rewrite pgcrypto to use evp openssl interface.

pgcrypto _was_ written using EVP, but I needed to rewrite it
when I found out EVP supports only key lengths used in SSL.

--
marko

Marko Kreen wrote:
> On 7/24/07, Zdenek Kotala <Zdenek(dot)Kotala(at)sun(dot)com> wrote:

>> However, on default installation (which is commonly used) it is a
>> problem. Regression test cannot be fixed because it tests strong
>> ciphers, but there two very strange issue:
>>
>> 1) First issue is blowfish cipher. Because pgcrypto uses old interface
>> instead new "evp" it calls bf_set_key function which does not return any
>> output and cut key if it is too long. See
>> http://src.opensolaris.org/source/xref/onnv/onnv-gate/usr/src/common/openssl/crypto/bf/bf_skey.c
>>
>> line 84.
>>
>> If user installs strong crypto he will not be able decrypt data which
>> has been encrypted before.
>>
>> The fix of this issue is ugly, because there is not way how to verify
>> supported key length with old openssl API and only new API return err if
>> length is not supported.
>
> NAK. The fix is broken because it uses EVP interface. EVP is not
> a general-purpose interface because not all valid keys for cipher
> pass thru it. Only key-lengths used in SSL will work...

I'm not openssl expert, but if you look how to EVP call for setkey is
implemented you can see that finally is call BF_set_key. Only there is
one extra layer see
http://src.opensolaris.org/source/xref/onnv/onnv-gate/usr/src/common/openssl/crypto/evp/e_bf.c

> Could you rework the fix that it uses the BF_* interface,
> does a test-encoding with full-length key and compares it to
> expected result. And does it just once, not on each call.

OK. I can do, but it is not general solution. Because it will work only
in our case, because we know 128 is a restricted limit.

> That should be put into separate function probably.

yes

>> 2) AES ciphere crashes when key is longer. It happens because return
>> value from AES_set_encrypt_key is ignored and AES_encrypt is called with
>> uninitialized structure.
>
> ACK, error checking is good. But please return PXE_KEY_TOO_BIG
> directly from ossl_aes_key_init.

OK.

> I must admit the internal API for ciphers is clumsy and could
> need rework to something saner. This shows here.
>
>> I attach patch which fix both issues, but main problem is there that old
>> openssl API is used and supported key lengths are hardcoded. I think we
>> can add to TODO list rewrite pgcrypto to use evp openssl interface.
>
> pgcrypto _was_ written using EVP, but I needed to rewrite it
> when I found out EVP supports only key lengths used in SSL.

Is it still correct? It seems that blowfish accepts all key range, but
How I mention I'm not openssl guru and documentation is very bad :(.

Zdenek

On 7/24/07, Zdenek Kotala <Zdenek(dot)Kotala(at)sun(dot)com> wrote:
> Marko Kreen wrote:
> > NAK. The fix is broken because it uses EVP interface. EVP is not
> > a general-purpose interface because not all valid keys for cipher
> > pass thru it. Only key-lengths used in SSL will work...
>
> I'm not openssl expert, but if you look how to EVP call for setkey is
> implemented you can see that finally is call BF_set_key. Only there is
> one extra layer see
> http://src.opensolaris.org/source/xref/onnv/onnv-gate/usr/src/common/openssl/crypto/evp/e_bf.c

I glanced into evp.h for 0.9.7 and 0.9.6j and remembered that
there were 2 things EVP forced - key length and padding.

When I replied to you I remembered things bit wrong, there are
indeed way for changing key size even in 0.9.6, but not for
padding. EVP_CIPHER_CTX_set_padding() appers in only in 0.9.7.

I suspect as I could not work around forced padding I did not
research key size issue very deeply.

So we can revisit the issue when we are ready to drop
support for 0.9.6x.

> > Could you rework the fix that it uses the BF_* interface,
> > does a test-encoding with full-length key and compares it to
> > expected result. And does it just once, not on each call.
>
> OK. I can do, but it is not general solution. Because it will work only
> in our case, because we know 128 is a restricted limit.

It _is_ a general solution if you test with a 448 bit key.

Using BF_ API but testing via EVP_ API is unobvious first,
in addition leaving the user depending whether the molesters
got all the details right.

When everything uses EVP then indeed, we can test via EVP.

> > I must admit the internal API for ciphers is clumsy and could
> > need rework to something saner. This shows here.
> >
> >> I attach patch which fix both issues, but main problem is there that old
> >> openssl API is used and supported key lengths are hardcoded. I think we
> >> can add to TODO list rewrite pgcrypto to use evp openssl interface.
> >
> > pgcrypto _was_ written using EVP, but I needed to rewrite it
> > when I found out EVP supports only key lengths used in SSL.
>
> Is it still correct? It seems that blowfish accepts all key range, but

Yes, seems since 0.9.7 we could work with EVP.

> How I mention I'm not openssl guru and documentation is very bad :(.

It's somewhat lacking, yes. User is forced to read their source
which isn't very nice either...

--
marko

Marko Kreen wrote:
> On 7/24/07, Zdenek Kotala <Zdenek(dot)Kotala(at)sun(dot)com> wrote:
>> Marko Kreen wrote:
>> > NAK. The fix is broken because it uses EVP interface. EVP is not
>> > a general-purpose interface because not all valid keys for cipher
>> > pass thru it. Only key-lengths used in SSL will work...
>>
>> I'm not openssl expert, but if you look how to EVP call for setkey is
>> implemented you can see that finally is call BF_set_key. Only there is
>> one extra layer see
>> http://src.opensolaris.org/source/xref/onnv/onnv-gate/usr/src/common/openssl/crypto/evp/e_bf.c
>>
>
> I glanced into evp.h for 0.9.7 and 0.9.6j and remembered that
> there were 2 things EVP forced - key length and padding.
>
> When I replied to you I remembered things bit wrong, there are
> indeed way for changing key size even in 0.9.6, but not for
> padding. EVP_CIPHER_CTX_set_padding() appers in only in 0.9.7.
>
> I suspect as I could not work around forced padding I did not
> research key size issue very deeply.
>
> So we can revisit the issue when we are ready to drop
> support for 0.9.6x.

the last openssl 0.9.6 release was in march 2004 and 0.9.7 is available
since early 2003 - I don't think dropping support for it in 8.3+ would
be unreasonable at all ...

Stefan

On 7/24/07, Stefan Kaltenbrunner <stefan(at)kaltenbrunner(dot)cc> wrote:
> Marko Kreen wrote:
> > So we can revisit the issue when we are ready to drop
> > support for 0.9.6x.
>
> the last openssl 0.9.6 release was in march 2004 and 0.9.7 is available
> since early 2003 - I don't think dropping support for it in 8.3+ would
> be unreasonable at all ...

Now that I think about it, then yes, dropping 0.9.6 from 8.4
onwards should be no problem. Considering the code could need
good cleanup anyway, that may be a good moment for it.

--
marko

Stefan Kaltenbrunner <stefan(at)kaltenbrunner(dot)cc> writes:
> Marko Kreen wrote:
>> So we can revisit the issue when we are ready to drop
>> support for 0.9.6x.

> the last openssl 0.9.6 release was in march 2004 and 0.9.7 is available
> since early 2003 - I don't think dropping support for it in 8.3+ would
> be unreasonable at all ...

Any major rewrite of pgcrypto would be for 8.4 (or later) at this point.
I tend to agree that we could drop 0.9.6x support in that timeframe.

regards, tom lane

Just confirming, this should be applied to 8.3, right?

---------------------------------------------------------------------------

Zdenek Kotala wrote:
> Stefan reported me that prcrypto regression test fails on solaris 10
> with openssl support. I investigated this problem and the result is that
> Solaris 10 delivers only support for short keys up to 128. Strong crypto
> (SUNWcry and SUNWcryr packages) is available on web download pages. (It
> is result of US crypto export policy.)
>
> However, on default installation (which is commonly used) it is a
> problem. Regression test cannot be fixed because it tests strong
> ciphers, but there two very strange issue:
>
> 1) First issue is blowfish cipher. Because pgcrypto uses old interface
> instead new "evp" it calls bf_set_key function which does not return any
> output and cut key if it is too long. See
> http://src.opensolaris.org/source/xref/onnv/onnv-gate/usr/src/common/openssl/crypto/bf/bf_skey.c
> line 84.
>
> If user installs strong crypto he will not be able decrypt data which
> has been encrypted before.
>
> The fix of this issue is ugly, because there is not way how to verify
> supported key length with old openssl API and only new API return err if
> length is not supported.
>
>
> 2) AES ciphere crashes when key is longer. It happens because return
> value from AES_set_encrypt_key is ignored and AES_encrypt is called with
> uninitialized structure.
>
>
> I attach patch which fix both issues, but main problem is there that old
> openssl API is used and supported key lengths are hardcoded. I think we
> can add to TODO list rewrite pgcrypto to use evp openssl interface.
>
>
> Any comments?
>
> Zdenek
>

>
> ---------------------------(end of broadcast)---------------------------
> TIP 6: explain analyze is your friend

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://www.enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

Bruce Momjian wrote:
> Just confirming, this should be applied to 8.3, right?

I think marko is working on an updated patch for this:

http://archives.postgresql.org/pgsql-hackers/2007-09/msg00386.php

without that the backend will coredump if ones uses string ciphers with
pgcrypto on a default solaris install so it seems like a thing we should
fix for 8.3.

Stefan

Stefan Kaltenbrunner wrote:
> Bruce Momjian wrote:
>> Just confirming, this should be applied to 8.3, right?
>
> I think marko is working on an updated patch for this:
>
> http://archives.postgresql.org/pgsql-hackers/2007-09/msg00386.php
>
> without that the backend will coredump if ones uses string ciphers with
> pgcrypto on a default solaris install so it seems like a thing we should
> fix for 8.3.

Yes, I also would like to have backport for 8.2 and 8.1. Because this
branch are also affected. (I think backport is easy there are no much
change between 8.1 and 8.3)

Zdenek