Re: [HACKERS] Re: Updated libpq5 packages cause connection errors on postgresql 9.2

One of our servers is currently running on postgres 9.2 using the 9.2.9-1.pgdg70+1 packages from pgdg.

After an apt update this morning which brought in the libpq5 package version 9.4.0-1.pgdg70+1, connections to the database started failing with SSL errors logged on the server:

[unknown] [unknown] LOG: could not accept SSL connection: digest too big for rsa key

Rolling back the server and client to libpq5 version 9.3.5-2.pgdg70+1 fixed it.

This is running on an otherwise up-to-date Debian Wheezy. The SSL certificate is locally issued using an internal CA which has been added to the local trust store. SSL-related config options are left set to the defaults.

--
Chris Butler
Zedcore Systems Ltd

Telephone: 0114 303 0666
Direct dial: 0114 303 0572

Re: Chris Butler 2014-12-19 <1155204201(dot)65430(dot)1418975376728(dot)JavaMail(dot)zimbra(at)zedcore(dot)com>
> One of our servers is currently running on postgres 9.2 using the 9.2.9-1.pgdg70+1 packages from pgdg.
>
> After an apt update this morning which brought in the libpq5 package version 9.4.0-1.pgdg70+1, connections to the database started failing with SSL errors logged on the server:
>
> [unknown] [unknown] LOG: could not accept SSL connection: digest too big for rsa key
>
> Rolling back the server and client to libpq5 version 9.3.5-2.pgdg70+1 fixed it.
>
> This is running on an otherwise up-to-date Debian Wheezy. The SSL certificate is locally issued using an internal CA which has been added to the local trust store. SSL-related config options are left set to the defaults.

Hi Chris,

thanks for the report.

Googling for "digest too big for rsa key" seems to indicate that this
problem occurs when you are using (client?) certificates with short
RSA keys. 512 bits is most often cited in the problem reports,
something like 768 is around the minimum size that works, and of
course, anything smaller than 1024 or really 1536 (or 2048) bits is
too small for today's crypto standards.

So the question here is if this is also the problem you saw - are you
using client or server certificates with short keys?

What this explanation doesn't explain is why the problem occurs with
9.4's libpq5 while it works with 9.3's. The libssl version used for
building these packages should really be the same, 9.3.5-2.pgdg70+1
was built just two days ago as well.

I'm CCing -hackers, maybe someone there has an idea.

Christoph
--
cb(at)df7cb(dot)de | http://www.df7cb.de/

Hi Christoph,

----- Original Message -----
> From: "Christoph Berg" <cb(at)df7cb(dot)de>
> To: "Chris Butler" <cbutler(at)zedcore(dot)com>
>
> Googling for "digest too big for rsa key" seems to indicate that this
> problem occurs when you are using (client?) certificates with short
> RSA keys. 512 bits is most often cited in the problem reports,
> something like 768 is around the minimum size that works, and of
> course, anything smaller than 1024 or really 1536 (or 2048) bits is
> too small for today's crypto standards.
>
> So the question here is if this is also the problem you saw - are you
> using client or server certificates with short keys?

Yes, that would appear to be the case - the key we're using is only 512 bits. I'll make sure we replace the certificate before re-applying the update (which will probably be after the holidays now).

> What this explanation doesn't explain is why the problem occurs with
> 9.4's libpq5 while it works with 9.3's. The libssl version used for
> building these packages should really be the same, 9.3.5-2.pgdg70+1
> was built just two days ago as well.

For info, I can confirm that both libraries are loading the same libssl:

zedcore(at)web2:/tmp/usr/lib/x86_64-linux-gnu$ ldd /usr/lib/x86_64-linux-gnu/libpq.so.5 | grep libssl
libssl.so.1.0.0 => /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0 (0x00007f3e8d898000)
zedcore(at)web2:/tmp/usr/lib/x86_64-linux-gnu$ ldd ./libpq.so.5 | grep libssl
libssl.so.1.0.0 => /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0 (0x00007f5d76176000)

I can see a few changes are listed in the 9.4 changelog relating to SSL, so my guess would be one of those changes has altered the behaviour of libssl when presented with a small key.

--
Chris Butler
Zedcore Systems Ltd

Telephone: 0114 303 0666
Direct dial: 0114 303 0572

On Fri, Dec 19, 2014 at 11:52 AM, Christoph Berg <cb(at)df7cb(dot)de> wrote:
>
> Re: Chris Butler 2014-12-19 <
> 1155204201(dot)65430(dot)1418975376728(dot)JavaMail(dot)zimbra(at)zedcore(dot)com>
> > One of our servers is currently running on postgres 9.2 using the
> 9.2.9-1.pgdg70+1 packages from pgdg.
> >
> > After an apt update this morning which brought in the libpq5 package
> version 9.4.0-1.pgdg70+1, connections to the database started failing with
> SSL errors logged on the server:
> >
> > [unknown] [unknown] LOG: could not accept SSL connection: digest too
> big for rsa key
> >
> > Rolling back the server and client to libpq5 version 9.3.5-2.pgdg70+1
> fixed it.
> >
> > This is running on an otherwise up-to-date Debian Wheezy. The SSL
> certificate is locally issued using an internal CA which has been added to
> the local trust store. SSL-related config options are left set to the
> defaults.
>
> Hi Chris,
>
> thanks for the report.
>
> Googling for "digest too big for rsa key" seems to indicate that this
> problem occurs when you are using (client?) certificates with short
> RSA keys. 512 bits is most often cited in the problem reports,
> something like 768 is around the minimum size that works, and of
> course, anything smaller than 1024 or really 1536 (or 2048) bits is
> too small for today's crypto standards.
>
> So the question here is if this is also the problem you saw - are you
> using client or server certificates with short keys?
>
> What this explanation doesn't explain is why the problem occurs with
> 9.4's libpq5 while it works with 9.3's. The libssl version used for
> building these packages should really be the same, 9.3.5-2.pgdg70+1
> was built just two days ago as well.
>
> I'm CCing -hackers, maybe someone there has an idea.
>

Some googling shows that this could be because it's negotiating TLS 1.2
which the key is just too small for. And we did change that in 9.4 - commit
326e1d73c476a0b5061ef00134bdf57aed70d5e7 disabled SSL in favor of always
using TLS for security reasons.

--
Magnus Hagander
Me: http://www.hagander.net/
Work: http://www.redpill-linpro.com/

Magnus Hagander <magnus(at)hagander(dot)net> writes:
> On Fri, Dec 19, 2014 at 11:52 AM, Christoph Berg <cb(at)df7cb(dot)de> wrote:
>> Googling for "digest too big for rsa key" seems to indicate that this
>> problem occurs when you are using (client?) certificates with short
>> RSA keys. 512 bits is most often cited in the problem reports,
>> something like 768 is around the minimum size that works, and of
>> course, anything smaller than 1024 or really 1536 (or 2048) bits is
>> too small for today's crypto standards.
>>
>> So the question here is if this is also the problem you saw - are you
>> using client or server certificates with short keys?
>>
>> What this explanation doesn't explain is why the problem occurs with
>> 9.4's libpq5 while it works with 9.3's. The libssl version used for
>> building these packages should really be the same, 9.3.5-2.pgdg70+1
>> was built just two days ago as well.

> Some googling shows that this could be because it's negotiating TLS 1.2
> which the key is just too small for. And we did change that in 9.4 - commit
> 326e1d73c476a0b5061ef00134bdf57aed70d5e7 disabled SSL in favor of always
> using TLS for security reasons.

Hm ... the 9.4 release notes fail to describe that change adequately, and
certainly don't mention that it would have any compatibility implications.
Guess that needs to be fixed. Does anyone know offhand what the change in
the minimum key length is across SSL/TLS versions, exactly?

regards, tom lane

On Fri, Dec 19, 2014 at 3:57 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
>
> Magnus Hagander <magnus(at)hagander(dot)net> writes:
> > On Fri, Dec 19, 2014 at 11:52 AM, Christoph Berg <cb(at)df7cb(dot)de> wrote:
> >> Googling for "digest too big for rsa key" seems to indicate that this
> >> problem occurs when you are using (client?) certificates with short
> >> RSA keys. 512 bits is most often cited in the problem reports,
> >> something like 768 is around the minimum size that works, and of
> >> course, anything smaller than 1024 or really 1536 (or 2048) bits is
> >> too small for today's crypto standards.
> >>
> >> So the question here is if this is also the problem you saw - are you
> >> using client or server certificates with short keys?
> >>
> >> What this explanation doesn't explain is why the problem occurs with
> >> 9.4's libpq5 while it works with 9.3's. The libssl version used for
> >> building these packages should really be the same, 9.3.5-2.pgdg70+1
> >> was built just two days ago as well.
>
> > Some googling shows that this could be because it's negotiating TLS 1.2
> > which the key is just too small for. And we did change that in 9.4 -
> commit
> > 326e1d73c476a0b5061ef00134bdf57aed70d5e7 disabled SSL in favor of always
> > using TLS for security reasons.
>
> Hm ... the 9.4 release notes fail to describe that change adequately, and
> certainly don't mention that it would have any compatibility implications.
> Guess that needs to be fixed. Does anyone know offhand what the change in
> the minimum key length is across SSL/TLS versions, exactly?
>

I haven't seen a specific number, it might depend on exactly which cipher
is negotiated. See for example
http://openssl.6102.n7.nabble.com/What-is-the-reason-for-error-quot-SSL-negotiation-failed-error-04075070-rsa-routines-RSA-sign-digest-td43953.html

All references I have foud say at least 2014 is safe and 512 is broken, but
there are points in betwee nthat apparently works in some cases only.

I think if we say "use 1024 bits or more" we err on the safe side.

--
Magnus Hagander
Me: http://www.hagander.net/
Work: http://www.redpill-linpro.com/

On Sat, Dec 20, 2014 at 12:27:05PM +0100, Magnus Hagander wrote:
> I haven't seen a specific number, it might depend on exactly which cipher is
> negotiated. See for example http://openssl.6102.n7.nabble.com/
> What-is-the-reason-for-error-quot-SSL-negotiation-failed-error-04075070-rsa-routines-RSA-sign-digest-td43953.html
>
> All references I have foud say at least 2014 is safe and 512 is broken, but
> there are points in betwee nthat apparently works in some cases only.
>
> I think if we say "use 1024 bits or more" we err on the safe side. 

Did we ever decide on this?

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ Everyone has their own god. +

Re: Bruce Momjian 2015-04-01 <20150401160907(dot)GJ4466(at)momjian(dot)us>
> On Sat, Dec 20, 2014 at 12:27:05PM +0100, Magnus Hagander wrote:
> > I haven't seen a specific number, it might depend on exactly which cipher is
> > negotiated. See for example http://openssl.6102.n7.nabble.com/
> > What-is-the-reason-for-error-quot-SSL-negotiation-failed-error-04075070-rsa-routines-RSA-sign-digest-td43953.html
> >
> > All references I have foud say at least 2014 is safe and 512 is broken, but
> > there are points in betwee nthat apparently works in some cases only.
> >
> > I think if we say "use 1024 bits or more" we err on the safe side. 
>
> Did we ever decide on this?

The question seems to be if we want to recommend "1024 or more" or
something more sophisticated like "use something between 512 and 1024
but ymmv .... 1024 should work in any case". Given that more bits
should be more secure, and 1024 shouldn't pose a performance problem
for anyone, going for the short version shouldn't do any harm.

Christoph
--
cb(at)df7cb(dot)de | http://www.df7cb.de/