Re: BUG #5468: Pg doesn't send accepted root CA list to client during SSL client cert request

The following bug has been logged online:

Bug reference: 5468
Logged by: Craig Ringer
Email address: craig(at)postnewspapers(dot)com(dot)au
PostgreSQL version: 8.4
Operating system: Ubuntu 10.04, but affects all
Description: Pg doesn't send accepted root CA list to client during
SSL client cert request
Details:

When configured to request a client certificate by the installation of the
'root.crt' file in the data dir, PostgreSQL will instruct OpenSSL to send a
CertificateRequest message during the SSL handshake. This asks the client to
send a certificate.

However, Pg doesn't tell OpenSSL to present the list of accepted signing
roots to the client, so the client has no way of knowing what client
certificate to present.

Existing clients (such as psql) generally only have one certificate/key
pair, and will blindly present it without checking what the server supports.
So it works fine.

If a client has a selection of keypairs, however, it will be unable to
negotiate with the server as it has no way to know which keypair to offer.
It can brute-force this with multiple connection attempts, but that's more
than little ugly. It may also try to guess the right client cert to send
based on the cert the server presented, but that'll only work if the server
cert happens to be signed by the same CA as the client certs, which is
frequently NOT the case.

Pg needs to tell OpenSSL to present the accepted root certificate(s) to the
client during negotiation, so the client can tell what to do. Adding a
suitable call to SSL_CTX_set_client_CA_list(...) in
src/backend/libpq/be-secure.c will do the trick, though it'll require
explicit loading of the CA cert list rather than the current approach of
just telling OpenSSL the file name.

This change will fix Java/JDBC clients trying to negotiate with a Pg server.
At present, a Java client using the Sun built-in X509KeyManager
implementation fails to negotiate because it doesn't know what cert to
present to the server. The user may provide a custom X509KeyManager, but in
doing so makes it difficult for the user to use PKCS#11 hardware tokens,
system-wide single sign-on, kerberos key storage, or other mechanisms.

This issue will also affect other clients using key stores capable of
holding multiple keys, like Mozilla's NSS and any PKCS#11 hardware token
provider, so it's not just Java specific.

On Fri, May 21, 2010 at 7:52 AM, Craig Ringer
<craig(at)postnewspapers(dot)com(dot)au> wrote:
>
> The following bug has been logged online:
>
> Bug reference:      5468
> Logged by:          Craig Ringer
> Email address:      craig(at)postnewspapers(dot)com(dot)au
> PostgreSQL version: 8.4
> Operating system:   Ubuntu 10.04, but affects all
> Description:        Pg doesn't send accepted root CA list to client during
> SSL client cert request
> Details:

This is actually not a bug, but a documented way how it's done. It's
actually even on the TODO to get it fixed, referencing bug #5245 -
from what I can tell that's what you're talking about, except we need
to send it in both directions?

> When configured to request a client certificate by the installation of the
> 'root.crt' file in the data dir, PostgreSQL will instruct OpenSSL to send a
> CertificateRequest message during the SSL handshake. This asks the client to
> send a certificate.
>
> However, Pg doesn't tell OpenSSL to present the list of accepted signing
> roots to the client, so the client has no way of knowing what client
> certificate to present.
>
> Existing clients (such as psql) generally only have one certificate/key
> pair, and will blindly present it without checking what the server supports.
> So it works fine.
>
> If a client has a selection of keypairs, however, it will be unable to
> negotiate with the server as it has no way to know which keypair to offer.
> It can brute-force this with multiple connection attempts, but that's more
> than little ugly. It may also try to guess the right client cert to send
> based on the cert the server presented, but that'll only work if the server
> cert happens to be signed by the same CA as the client certs, which is
> frequently NOT the case.
>
> Pg needs to tell OpenSSL to present the accepted root certificate(s) to the
> client during negotiation, so the client can tell what to do. Adding a
> suitable call to SSL_CTX_set_client_CA_list(...) in
> src/backend/libpq/be-secure.c will do the trick, though it'll require
> explicit loading of the CA cert list rather than the current approach of
> just telling OpenSSL the file name.
>
> This change will fix Java/JDBC clients trying to negotiate with a Pg server.
> At present, a Java client using the Sun built-in X509KeyManager
> implementation fails to negotiate because it doesn't know what cert to
> present to the server. The user may provide a custom X509KeyManager, but in
> doing so makes it difficult for the user to use PKCS#11 hardware tokens,
> system-wide single sign-on, kerberos key storage, or other mechanisms.
>
> This issue will also affect other clients using key stores capable of
> holding multiple keys, like Mozilla's NSS and any PKCS#11 hardware token
> provider, so it's not just Java specific.

Yeah, all our (at least my) testing is done on OpenSSL - I had no idea
of this behaviour of the java layer really.

Unfortunately, I don't think this is something we can fix as a bugfix
- it'll be a pretty clear change of behaviour, so I think it's
something we will need to push back for a full release, which would
mean 9.1. What would be good is if we can collect a list of interop
issues and collect test-cases for them, because given the state of the
openssl documentation it really comes down to having fairly detailed
test-cases...

--
Magnus Hagander
Me: http://www.hagander.net/
Work: http://www.redpill-linpro.com/

On 21/05/2010 10:35 PM, Magnus Hagander wrote:
> On Fri, May 21, 2010 at 7:52 AM, Craig Ringer
> <craig(at)postnewspapers(dot)com(dot)au> wrote:
>>
>> The following bug has been logged online:
>>
>> Bug reference: 5468
>> Logged by: Craig Ringer
>> Email address: craig(at)postnewspapers(dot)com(dot)au
>> PostgreSQL version: 8.4
>> Operating system: Ubuntu 10.04, but affects all
>> Description: Pg doesn't send accepted root CA list to client during
>> SSL client cert request
>> Details:
>
> This is actually not a bug, but a documented way how it's done. It's
> actually even on the TODO to get it fixed, referencing bug #5245 -
> from what I can tell that's what you're talking about, except we need
> to send it in both directions?

Bug 5245 is not the same issue. They're talking about the server not
sending the full certificate chain for the cert that identifies the
server (server.crt). It's nothing to do with client certificates.
Without the full chain, the client can't verify the server unless it
happens to already have the intermediate certs between the server's cert
and the trusted root that signed it installed locally. I haven't
encountered #5245 myself, but will test it shortly to verify. It'd
certainly count as a significant bug, as it would make it impossible to
use indirect trust to verify a server (as is the case when a corporate
CA signed by a "big name" CA is in use).

What I'm talking about is the server not sending the set of certificates
trusted as client certificate signers (root.crt) to the client when it
sends a CertificateRequest to demand a client certificate from the
client. Without that, the client doesn't know which client certificate
to present to the server in response to the CertificateRequest.

They're somewhat related, but a fix to one won't affect a fix to the other.

> Yeah, all our (at least my) testing is done on OpenSSL - I had no idea
> of this behaviour of the java layer really.

You would have the same issue with a client using the Mozilla NSS
libraries - but I'm not aware of any that do. Essentially, the only
reason OpenSSL works is because typical setups only have one client
certificate availible so they'll blindly present it without knowing or
caring what the server accepts.

> Unfortunately, I don't think this is something we can fix as a bugfix
> - it'll be a pretty clear change of behaviour, so I think it's
> something we will need to push back for a full release, which would
> mean 9.1.

Urk. I think you're right, but I don't like it :S

I'm working on testing a patch that gets Pg to present all trusted roots
to the client now, and will send it in as a starting point once I've
tested it with the clients I have on hand - PgJDBC, libpq, etc.

--
Craig Ringer

On 22/05/10 12:01, Craig Ringer wrote:
> On 21/05/2010 10:35 PM, Magnus Hagander wrote:
>>
>> Unfortunately, I don't think this is something we can fix as a bugfix
>> - it'll be a pretty clear change of behaviour, so I think it's
>> something we will need to push back for a full release, which would
>> mean 9.1.
>
> Urk. I think you're right, but I don't like it :S
>
> I'm working on testing a patch that gets Pg to present all trusted roots
> to the client now, and will send it in as a starting point once I've
> tested it with the clients I have on hand - PgJDBC, libpq, etc.

I've attached the patch. As you can see it's trivial, but this being
OpenSSL a small change doesn't necessarily have a small effect. While it
was produced against 8.4, it should apply fine against HEAD. I'm pulling
a shallow clone now just to make sure.

In this case though, it's a call that has _absolutely_ no effect if
'root.crt' isn't present in the data dir to tell the server to request a
client certificate during SSL negotiations. It's never run.

If 'root.crt' _is_ present, there are no new steps in the negotiation
behaviour, nor any new requirements of the client or server. All that
happens is that the client is given some more information which it may
choose to use to select an appropriate client certificate to present.

Clients that already worked didn't use this information - after all, it
was absent from the CertificateRequest sent by the server so they
couldn't use it if they wanted to. This includes libpq+openssl clients
(psql, PgAdmin III, etc). It also includes Java/JDBC clients with a
custom SSLSocketFactory that uses a custom X509KeyManager that
unconditionally presents a single client certificate - as this was the
only way to make client certificate authentication work in Java/PgJDBC
without this change.

I've verified that psql (8.4.3) and PgAdmin III (1.10.2) using libpq
8.4.3 are happy with this change, and behave no differently. See below.

I've also tested and made sure that a Java app with a
X509KeyManager/SSLSocketFactory that unconditionally present a client
certificate work fine (and unchanged) before and after the change.
Though after the change the app doesn't need the custom SSLSocketFactory
it works fine with it. I'll post an executable .jar and source .zip once
I tidy it up a little, as it'll serve as a useful example for how to use
libpq with client certificates - and how you *don't* need a custom
SSLSocketFactory if Pg sends the cert list.

psqlODBC uses psql with OpenSSL so it should behave just like psql, but
I can't find any documentation on how to use client certificates with it
anyway.

There are a lot of other interfaces here:

http://www.postgresql.org/download/products/2

... most of which I should be able to test given time. The problem is
that half the time they fail to document whether they support client
certificates at all, or how to use them, suggesting that it's likely
nobody uses that feature with them anyway. Furthermore, many client
interfaces are based on libpq and will inherit its client certificate
support.

For libpq-based clients:

==== libpq NO CLIENT CERT PRESENT ====
For these, nothing is in .postgresql/ so the clients can't present a
cert when requested.

- If no client cert is present locally but the server demands one (
root.crt is present and clientcert=1 is present on the matching line of
pg_hba.conf), they successfully negotiate an SSL connection without
presenting a client cert and are then disconnected by the server with a
message saying a client cert is required - as expected. No change.

- If the server requests a client cert but doesn't require it ( root.crt
is present, but clientcert=1 is not present on the matching line of
pg_hba.conf ) they successfully negotiate SSL without presenting a
client certificate and establish a successful connection. No change.

==== libpq CORRECT CLIENT CERT PRESENT ====
For these, a cert and key signed by a trusted root in root.crt is
present on the client at .postgresql/postgresql.{crt,key}

- If the server demands a client cert (root.crt is present and
clientcert=1) they present that client cert to the server, are verified,
and connect successfully. No change.

- If the server demands a client cert (root.crt is present and
clientcert=1) they present that client cert to the server, are verified,
and connect successfully. No change.

==== libpq WRONG CLIENT CERT PRESENT ====
For these, a client cert and key are present in
.postgresql/postgresql.{crt,key} but they're signed by a CA not in the
server's trusted CA list (root.crt) so they can't be verified.

- If the server demands a client cert, negotiation fails with "psql: SSL
error: tlsv1 alert unknown ca". No change.

- If the server requests but does not demand a client cert, negotiation
fails with "psql: SSL error: tlsv1 alert unknown ca". No change.

Once I have the Java test app and have tried the clients I can from the
client list, I'll send this patch in properly. I'd really like it to be
considered for 9.0, but I can see it can't really be included in 8.4
(helpful as that'd be).

--
Craig Ringer

Craig Ringer <craig(at)postnewspapers(dot)com(dot)au> writes:
> + SSL_CTX_set_client_CA_list( SSL_context, SSL_load_client_CA_file(ROOT_CERT_FILE) );

Hmm, what about failures? If we're loading the root cert file a second
time, it's possible that the user just changed it and the load now fails
for some reason.

regards, tom lane

On 24/05/10 00:58, Tom Lane wrote:
> Craig Ringer<craig(at)postnewspapers(dot)com(dot)au> writes:
>> + SSL_CTX_set_client_CA_list( SSL_context, SSL_load_client_CA_file(ROOT_CERT_FILE) );
>
> Hmm, what about failures? If we're loading the root cert file a second
> time, it's possible that the user just changed it and the load now fails
> for some reason.

True I guess. It's a pretty narrow window for failure given that we
*just* loaded it milliseconds ago, but not impossible.

As it is, SSL_ctx_set_client_CA_list handles null input gracefully, and
the client_CA member of the SSL_context starts out null, so there'll be
no harm in calling SSL_CTX_set_client_CA_list with a null arg when
SSL_load_client_CA_file fails. It just assigns null to the the
already-null client_CA member of the context. OpenSSL checks for null in
client_CA before doing anything with it.

If the user does somehow manage to trigger this case, the worst that'll
happen is that the list of trusted certificates will not be sent to the
client - exactly as currently happens. This may cause some clients to
fail to negotiate, but cannot grant access that would otherwise not be
granted. Triggering it requires access to the datadir (or where a
symlink from the datadir points) and has such a narrow time window I'm
not sure it's worth worrying about.

I *could* avoid the second file load by supplying the list of loaded
certificates from SSL_load_client_CA_file to OpenSSL as trusted
certificates for verifying clients. I'd really rather not mess with the
logic around verifying client certificates, though, especially given the
state of the OpenSSL documentation. What I've done is the approach
recommended in the documentation (see the man page for
SSL_CTX_set_client_CA_list) and seems safest.

Still, I'm happy to move some code around so that either load of
ROOT_CERT_FILE will report an error. The reason I did it this way is
that it's a small, clear change that only affects users of client
certificates. I've attached a revision that catches both failures, using
the existing error message. Hope that suits.

--
Craig Ringer

Craig Ringer <craig(at)postnewspapers(dot)com(dot)au> writes:
> Bug 5245 is not the same issue. They're talking about the server not
> sending the full certificate chain for the cert that identifies the
> server (server.crt). It's nothing to do with client certificates.
> Without the full chain, the client can't verify the server unless it
> happens to already have the intermediate certs between the server's cert
> and the trusted root that signed it installed locally. I haven't
> encountered #5245 myself, but will test it shortly to verify. It'd
> certainly count as a significant bug, as it would make it impossible to
> use indirect trust to verify a server (as is the case when a corporate
> CA signed by a "big name" CA is in use).

BTW, does anyone know exactly how to fix that? I'm looking at a related
request internal to Red Hat right now.

regards, tom lane

On Tue, May 25, 2010 at 17:48, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> Craig Ringer <craig(at)postnewspapers(dot)com(dot)au> writes:
>> Bug 5245 is not the same issue. They're talking about the server not
>> sending the full certificate chain for the cert that identifies the
>> server (server.crt). It's nothing to do with client certificates.
>> Without the full chain, the client can't verify the server unless it
>> happens to already have the intermediate certs between the server's cert
>> and the trusted root that signed it installed locally. I haven't
>> encountered #5245 myself, but will test it shortly to verify. It'd
>> certainly count as a significant bug, as it would make it impossible to
>> use indirect trust to verify a server (as is the case when a corporate
>> CA signed by a "big name" CA is in use).
>
> BTW, does anyone know exactly how to fix that?  I'm looking at a related
> request internal to Red Hat right now.

I have it on my TODO to figure it out, but from what I can tell it's
very close to being undocumented, like most of OpenSSL. So it will
need research of how others do it, etc. Unless someone can figure out
how to do it, and I can stick to juts reviewing it, there is pretty
much zero chance that I can get that done for 9.0 (even if we call it
a bugfix) due to lack of time over the next couple of weeks.

(and yes, I intend to get back in on the rest of this thread as well
once I've cleared my pgcon-induced backlog)

--
Magnus Hagander
Me: http://www.hagander.net/
Work: http://www.redpill-linpro.com/

Magnus Hagander <magnus(at)hagander(dot)net> writes:
> On Tue, May 25, 2010 at 17:48, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
>> Craig Ringer <craig(at)postnewspapers(dot)com(dot)au> writes:
>>> Bug 5245 is not the same issue.
>>
>> BTW, does anyone know exactly how to fix that? I'm looking at a related
>> request internal to Red Hat right now.

> I have it on my TODO to figure it out, but from what I can tell it's
> very close to being undocumented, like most of OpenSSL.

I've been googling this a bit, and the theory I have at the moment is
that Craig's proposed fix for #5468 will *also* fix #5245. However,
I'm not too adept at using openssl so actually testing that will
probably take a bit of time. Are you in a position to check it quickly?

regards, tom lane

On Tue, May 25, 2010 at 18:44, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> Magnus Hagander <magnus(at)hagander(dot)net> writes:
>> On Tue, May 25, 2010 at 17:48, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
>>> Craig Ringer <craig(at)postnewspapers(dot)com(dot)au> writes:
>>>> Bug 5245 is not the same issue.
>>>
>>> BTW, does anyone know exactly how to fix that?  I'm looking at a related
>>> request internal to Red Hat right now.
>
>> I have it on my TODO to figure it out, but from what I can tell it's
>> very close to being undocumented, like most of OpenSSL.
>
> I've been googling this a bit, and the theory I have at the moment is
> that Craig's proposed fix for #5468 will *also* fix #5245.  However,
> I'm not too adept at using openssl so actually testing that will
> probably take a bit of time.  Are you in a position to check it quickly?

'fraid not. I'm doing intensive training this week, so the first
chance I'll have to give a proper check of it will be next week.
sorry.

--
Magnus Hagander
Me: http://www.hagander.net/
Work: http://www.redpill-linpro.com/

Magnus Hagander <magnus(at)hagander(dot)net> writes:
> On Tue, May 25, 2010 at 17:48, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
>> Craig Ringer <craig(at)postnewspapers(dot)com(dot)au> writes:
>>> Bug 5245 is not the same issue. They're talking about the server not
>>> sending the full certificate chain for the cert that identifies the
>>> server (server.crt). It's nothing to do with client certificates.
>>> Without the full chain, the client can't verify the server unless it
>>> happens to already have the intermediate certs between the server's cert
>>> and the trusted root that signed it installed locally. I haven't
>>> encountered #5245 myself, but will test it shortly to verify. It'd
>>> certainly count as a significant bug, as it would make it impossible to
>>> use indirect trust to verify a server (as is the case when a corporate
>>> CA signed by a "big name" CA is in use).
>>
>> BTW, does anyone know exactly how to fix that? I'm looking at a related
>> request internal to Red Hat right now.

> I have it on my TODO to figure it out, but from what I can tell it's
> very close to being undocumented, like most of OpenSSL.

I spent some time experimenting with this. I set up a self-signed root
CA, then an intermediate CA signed by the root, and then created server
and client certs signed by the intermediate CA. I find that everything
appears to work just fine if (1) the server's root.crt contains both the
intermediate and root certs, and (2) the client's root.crt contains the
root cert. (Most other combinations don't work, in particular if I give
the client only the intermediate CA's cert, it rejects connections.)

This leaves me a bit mystified as to what all the squawking is about.
IIUC, putting the full chain of CA certs into server.crt is the standard
way to set up a server that is signed by a non-root CA. Maybe we just
need to document that? It certainly appears to me that the server sends
all those certs to the client, because the client wouldn't be able to
verify the server's cert without the intermediate CA cert, which it
hasn't got on its end.

(Either that, or there's something seriously broken in libpq's checking
of server certs :-()

I notice that the complaint in bug #5245, like the one in #5468,
refers specifically to Java client behavior. I'm suspicious that
what we're dealing with is a Java requirement that openssl itself
does not have.

regards, tom lane

On 25/05/10 23:48, Tom Lane wrote:
> Craig Ringer<craig(at)postnewspapers(dot)com(dot)au> writes:
>> Bug 5245 is not the same issue. They're talking about the server not
>> sending the full certificate chain for the cert that identifies the
>> server (server.crt). It's nothing to do with client certificates.
>> Without the full chain, the client can't verify the server unless it
>> happens to already have the intermediate certs between the server's cert
>> and the trusted root that signed it installed locally. I haven't
>> encountered #5245 myself, but will test it shortly to verify. It'd
>> certainly count as a significant bug, as it would make it impossible to
>> use indirect trust to verify a server (as is the case when a corporate
>> CA signed by a "big name" CA is in use).
>
> BTW, does anyone know exactly how to fix that? I'm looking at a related
> request internal to Red Hat right now.

The first thing to test is whether concatenating the root cert onto the
server cert in 'server.crt' does the trick. Though, really, OpenSSL
should do the right thing automatically so long as it has the CA
certificate loaded.

Certainly my (patched) server is doing the right thing and sending the
certificate. I'm 99% sure it did so before patching, though, just from
having root.crt installed. However, this only works because the CA I
want to validate clients against happens to be the same CA that signed
my server's certificate, which is frequently *not* the case.

I do *not* have the CA cert concatenated onto server.crt. I'll have to
see if that works, because that's how it's usually done with OpenSSL.

BTW, the little Java app I posted for client certifiate testing will let
you get detailed tracing of a Pg SSL connection. Just run it with the
default sslsocketfactory and no client cert:

java -jar PgClientCertDemo.jar default '' '' '' \

jdbc:postgresql://YOURSERVER/YOURDATABASE?ssl=true&user=blah&password=blah

and you'll get detailed trace information (possibly followed by an
exception if it couldn't negotiate for some reason). Search for
'ServerHello' to find the start of the area of interest in the
negotiation. Search for 'chain [' to find the server certificate chain
entries.

--
Craig Ringer

Craig Ringer <craig(at)postnewspapers(dot)com(dot)au> writes:
> I do *not* have the CA cert concatenated onto server.crt. I'll have to
> see if that works, because that's how it's usually done with OpenSSL.

Hmm. That case doesn't work for me; what does work is including the
intermediate cert in the server's root.crt.

regards, tom lane

On 26/05/10 07:37, Tom Lane wrote:
> Craig Ringer<craig(at)postnewspapers(dot)com(dot)au> writes:
>> I do *not* have the CA cert concatenated onto server.crt. I'll have to
>> see if that works, because that's how it's usually done with OpenSSL.
>
> Hmm. That case doesn't work for me; what does work is including the
> intermediate cert in the server's root.crt.

Sorry, that was my poor choice of words.

s/the CA cert/the full certificate chain/g

It is the intermediate certs that the client may not have that are the
important ones. 'the CA' I was referring to was the _intermediate_ CA,
eg the company sub-CA; I just needed to be (a lot) clearer about it.

--
Craig Ringer

Craig Ringer <craig(at)postnewspapers(dot)com(dot)au> writes:
> On 26/05/10 07:37, Tom Lane wrote:
>> Craig Ringer<craig(at)postnewspapers(dot)com(dot)au> writes:
>>> I do *not* have the CA cert concatenated onto server.crt. I'll have to
>>> see if that works, because that's how it's usually done with OpenSSL.
>>
>> Hmm. That case doesn't work for me; what does work is including the
>> intermediate cert in the server's root.crt.

> Sorry, that was my poor choice of words.

> s/the CA cert/the full certificate chain/g

What I meant to question is *which* file the intermediate CA certs
go into. It doesn't seem tremendously sensible to me to put them into
the server.crt file, since that's intended to define exactly one cert,
namely the one identifying the server. On the other hand, putting them
into the root.crt file implies that the intermediate certs are as good
as the real root CA for trust purposes, which might not quite be the
right thing either.

regards, tom lane

Craig Ringer <craig(at)postnewspapers(dot)com(dot)au> writes:
> BTW, the little Java app I posted for client certifiate testing will let
> you get detailed tracing of a Pg SSL connection.

Where did you post that exactly? I'm sure not finding it in this thread.

regards, tom lane

* Tom Lane (tgl(at)sss(dot)pgh(dot)pa(dot)us) wrote:
> What I meant to question is *which* file the intermediate CA certs
> go into. It doesn't seem tremendously sensible to me to put them into
> the server.crt file, since that's intended to define exactly one cert,
> namely the one identifying the server. On the other hand, putting them
> into the root.crt file implies that the intermediate certs are as good
> as the real root CA for trust purposes, which might not quite be the
> right thing either.

root CA's are self-signed. intermediate CAs are not. They typically
both go into directories/files like 'cacerts' (eg: Strongswan expects
them in the cacerts directory). Most systems (uh, all?) will validate
all the way up to a self-signed cert- intermediate CAs are only used as
a mechanism to get to the root CA. I don't believe there's any
confusion about intermediate CAs being accepted as root CAs just because
they're in the same file or directory.

All that being said- I don't think anyone would really complain if
intermediate CAs and root CAs were stored in different
directories/files. That's how Windows has certificates separated out.

Thanks,

Stephen

I wrote:
> What I meant to question is *which* file the intermediate CA certs
> go into.

I did some more experimentation using "openssl s_client" to look at the
cert exchange (after lobotomizing the postmaster so that it'd start a
TLS exchange immediately upon connection, without our normal startup
protocol). It appears to be the case that there are two configurations
that work for CA chains:

1. server.crt can include just the server's own cert, if all
intermediate certs up to and including the root are listed in root.crt.

2. server.crt can include the server's own cert plus all intermediate
certs *up to and including the root*. In this case just the root need
be listed in root.crt.

One would think that the root cert could be omitted from server.crt if
it's given in root.crt, but apparently this is not so, except for the
special case where the server cert is directly signed by a root CA.

I am now of the opinion that bug #5245 is in fact an exact dup of
bug #5468. The previous reporter was jumping to conclusions about what
his problem was: it was not that the server didn't send the full cert
chain, but that Java couldn't do the right thing without having the list
of cert names.

It's possible, depending on exactly what Java does in these cases, that
it has to have cert names given to it for the intermediate certs as well
as the root. This would imply that only configuration #1 will work with
Java clients, even after we add the SSL_CTX_set_client_CA_list call.
That would be too bad, since configuration #2 is probably a bit
preferable from the point of view of considering fewer certs to be
fully trusted.

In any case I'm thinking that we need to document how to set up
configurations with chains of CA certs.

regards, tom lane

On 26/05/10 09:35, Tom Lane wrote:

> I am now of the opinion that bug #5245 is in fact an exact dup of
> bug #5468. The previous reporter was jumping to conclusions about what
> his problem was: it was not that the server didn't send the full cert
> chain, but that Java couldn't do the right thing without having the list
> of cert names.

No, they ARE NOT the same thing.

#5468 is about *CLIENT* *CERTIFICATE* *AUTHENTICATION* where the
*SERVER* VALIDATES THE *CLIENT* after the server sends a
ServerHello.

#5245 is about *CLIENT* *VALIDATION* *OF* *THE* *SERVER*, where the
*CLIENT* VALIDATES THE *SERVER* after the server sends a
CertificateRequest.

You are confusing these two unrelated phases of SSL negotiation.

For the complaint in #5245 to be addressed, the server must send the
full certificate chain for the certificate the server is using to
identify its self as pgserver.domain.com to the client during the
ServerHello phase of SSL negotiation. If correctly configured, the
server already does this, and #5245 really just needs some documentation
improvements.

For #5468 to be addressed, the server must send the CA certificates (not
necessarily the full chain) of any CAs it trusts to identify clients to
the client during the optional CertificateRequest phase of SSL
negotiaton. This is only important if clientcert=1 is specified in
pg_hba.conf .

Sorry to continue "squarking" about this, but I just don't know how to
explain it any better without stepping through the full SSL negotiation.
It's frustrating that you're focusing on #5245 (which I didn't bring up
and isn't the same thing at all) and disregarding the issue I'm trying
to get fixed because #5245 is related to server misconfiguration.

Hang on, I'll send side-by-side comparisons of patched and unnpatched
server chat along with why they differ and why it's important in a min.

> It's possible, depending on exactly what Java does in these cases, that
> it has to have cert names given to it for the intermediate certs as well
> as the root.

(this relates to #5245 **NOT** #5468)

Java, exactly like OpenSSL, needs *SOME* way to obtain any certificates
between the a CA the client trusts and the server's certificate. If the
client has the required intermidiate certs pre-installed, the server
doesn't have to send them. If the client doesn't have them
pre-installed, the server must send them or the server has no way to
verify the chain of trust.

This is bog-standard SSL stuff.

> In any case I'm thinking that we need to document how to set up
> configurations with chains of CA certs.

Yes, and patch the server to send the list of trusted CAs to the client
during client certificate negotiaton to fix #5468 .

--
Craig Ringer

Tech-related writing: http://soapyfrogs.blogspot.com/

On 26/05/10 09:59, Craig Ringer wrote:
> On 26/05/10 09:35, Tom Lane wrote:
>
>> I am now of the opinion that bug #5245 is in fact an exact dup of
>> bug #5468. The previous reporter was jumping to conclusions about what
>> his problem was: it was not that the server didn't send the full cert
>> chain, but that Java couldn't do the right thing without having the list
>> of cert names.
>
> No, they ARE NOT the same thing.
>
> #5468 is about *CLIENT* *CERTIFICATE* *AUTHENTICATION* where the
> *SERVER* VALIDATES THE *CLIENT* after the server sends a
> ServerHello.
>
> #5245 is about *CLIENT* *VALIDATION* *OF* *THE* *SERVER*, where the
> *CLIENT* VALIDATES THE *SERVER* after the server sends a
> CertificateRequest.

Argh, now I'm getting MYSELF backwards. Correction:

#5468 is about *CLIENT* *CERTIFICATE* *AUTHENTICATION* where the
*SERVER* VALIDATES THE *CLIENT* after the server sends a
*CertificateRequest*. <-- Was reversed above

#5245 is about *CLIENT* *VALIDATION* *OF* *THE* *SERVER*, where the
*CLIENT* VALIDATES THE *SERVER* after the server sends a
*ServerHello*. <-- Was reversed above

--
Craig Ringer

Tech-related writing: http://soapyfrogs.blogspot.com/

Craig Ringer <craig(at)postnewspapers(dot)com(dot)au> writes:
> You are confusing these two unrelated phases of SSL negotiation.

No, I don't think so.

> For the complaint in #5245 to be addressed, the server must send the
> full certificate chain for the certificate the server is using to
> identify its self as pgserver.domain.com to the client during the
> ServerHello phase of SSL negotiation. If correctly configured, the
> server already does this, and #5245 really just needs some documentation
> improvements.

As best I can tell, the server already does that, if correctly
configured, and the configuration described in #5245 is correct.
Therefore, it's failing because of something else. What the reporter
of #5245 *says* the bug is is not necessarily what it *actually* is.
What I believe his *actual* problem is is that Java is unable to verify
the cert chain without a name for (at least) the root cert. That makes
it the same as #5468, or at least it has the same fix.

I have found an additional bug here, but it's in libpq not the server,
and thus not responsible for either your bug report or his. I'll start
a new thread about that in a minute.

regards, tom lane

On 26/05/10 10:16, Tom Lane wrote:
> Craig Ringer <craig(at)postnewspapers(dot)com(dot)au> writes:
>> You are confusing these two unrelated phases of SSL negotiation.
>
> No, I don't think so.

http://www.cgisecurity.com/owasp/html/ch07s04.html

See in the second part, the new entry #5 "client request"
("CertificateRequest") ? That's the big Pg gets wrong at the moment.

It's not the same as #2 in that diagram, which is what #5245 talks about.

I'm going to send you a canned configuration to demonstrate this, along
with network traces from wireshark and a session log from the test app.
Give me an hour or so to put it together.

--
Craig Ringer

Tech-related writing: http://soapyfrogs.blogspot.com/

All,

Sorry, I havn't really been following this thread.

* Craig Ringer (craig(at)postnewspapers(dot)com(dot)au) wrote:
> #5245 is about *CLIENT* *VALIDATION* *OF* *THE* *SERVER*, where the
> *CLIENT* VALIDATES THE *SERVER* after the server sends a
> CertificateRequest.
>
> For #5468 to be addressed, the server must send the CA certificates (not
> necessarily the full chain) of any CAs it trusts to identify clients to
> the client during the optional CertificateRequest phase of SSL
> negotiaton. This is only important if clientcert=1 is specified in
> pg_hba.conf .

Yeah, this is right. I've had similar issues in the past, and what's
better, some systems are too stupid to realize that they need to ignore
CAs which aren't ones that they trust (hello Winbloze IPSEC). Being
able to specificially say what CA cert should be sent to a given client
in pg_hba.conf would certainly be nice. I don't know if any of the
systems which talk to PG will have this problem (hopefully they'll all
use libpq...), but it's definitely something I've run into in the past.

> Java, exactly like OpenSSL, needs *SOME* way to obtain any certificates
> between the a CA the client trusts and the server's certificate. If the
> client has the required intermidiate certs pre-installed, the server
> doesn't have to send them. If the client doesn't have them
> pre-installed, the server must send them or the server has no way to
> verify the chain of trust.
>
> This is bog-standard SSL stuff.

Right, this should be supported and handled correctly (though, to be
fair, alot of times people just configure the clients with the
intermediate certs needed.. I realize that's not ideal though, since
they can change over time and make things more difficult.).

> > In any case I'm thinking that we need to document how to set up
> > configurations with chains of CA certs.
>
> Yes, and patch the server to send the list of trusted CAs to the client
> during client certificate negotiaton to fix #5468 .

Agreed.

Thanks,

Stephen

On 26/05/10 10:25, Stephen Frost wrote:

>>> In any case I'm thinking that we need to document how to set up
>>> configurations with chains of CA certs.
>>
>> Yes, and patch the server to send the list of trusted CAs to the client
>> during client certificate negotiaton to fix #5468 .
>
> Agreed.

Yeah, I'd really love to focus on the issue I reported (#5468) not an
earlier issue that was bought up during the conversation...

I'm putting together a completely self-contained test case ( database,
home-made CA, client and server SSL certs, pg_hba.conf, client
application, etc ) to demonstrate this at the moment, as I haven't been
successful in explaining it despite my best efforts.

Meanwhile, the mailing list seems to be silently eating my test program.
So: you can download it from:

executable jar with built-in usage/help:

http://www.postnewspapers.com.au/~craig/PgClientCertDemo.jar

sources and README:

http://www.postnewspapers.com.au/~craig/PgClientCertDemo.zip

Run the jar as:

java -jar PgClientCertDemo.jar

for help.

--
Craig Ringer

Tech-related writing: http://soapyfrogs.blogspot.com/

On 26/05/10 10:25, Stephen Frost wrote:

>>> In any case I'm thinking that we need to document how to set up
>>> configurations with chains of CA certs.
>>
>> Yes, and patch the server to send the list of trusted CAs to the client
>> during client certificate negotiaton to fix #5468 .
>
> Agreed.

A quick update on my own testing:

I've found that the Sun PKCS#12 keystore provider behaves just like
OpenSSL. It unconditionally sends the one and only client cert it has to
the server - after all, there's only one to choose from. This is a royal
pain to use, though, and requires the app's security to be configured
from the command line at each launch, or the app to override all user
settings and thus disable use of PKCS#11 hardware keys, etc.

The issue only arises if there is a keystore in use where the client may
have more than one client certificate/key availible to it and must pick
which one to send to the server. This is true of the default Sun JKS
keystore format, and for PKCS#11 stores like hardware crypto keys.

My self-contained test case will demonstrate both PKCS#12 file and JKS
keystore cases. Give me a bit to put it all together and you'll have
something you can play with, watch chat on the network, etc.

--
Craig Ringer

Tech-related writing: http://soapyfrogs.blogspot.com/

Stephen Frost <sfrost(at)snowman(dot)net> writes:
> * Tom Lane (tgl(at)sss(dot)pgh(dot)pa(dot)us) wrote:
>> What I meant to question is *which* file the intermediate CA certs
>> go into. It doesn't seem tremendously sensible to me to put them into
>> the server.crt file, since that's intended to define exactly one cert,

> root CA's are self-signed. intermediate CAs are not. They typically
> both go into directories/files like 'cacerts' (eg: Strongswan expects
> them in the cacerts directory). Most systems (uh, all?) will validate
> all the way up to a self-signed cert- intermediate CAs are only used as
> a mechanism to get to the root CA. I don't believe there's any
> confusion about intermediate CAs being accepted as root CAs just because
> they're in the same file or directory.

> All that being said- I don't think anyone would really complain if
> intermediate CAs and root CAs were stored in different
> directories/files. That's how Windows has certificates separated out.

After some more testing I think I've wrapped my head around how this
really ought to be documented, and it goes like this:

* server.crt is about proving the server's identity to the clients.
It must contain a cert or cert chain that leads to a CA that is trusted
by the clients (ie, is listed in the client-side root.crt files).

* root.crt is about proving the clients' identities to the server.
It lists the root CAs that the server trusts for that purpose.

* Similarly, the client-side postgresql.crt contains a cert (or,
if we fix libpq's bugs, a cert chain) that leads to a CA trusted
by the server, while the client-side root.crt lists the root CAs
that the client trusts for verifying the server.

In principle, you could have the server and clients using totally
nonoverlapping sets of trusted CAs (nonoverlapping root.crt lists),
as long as each can chain its identity up to a CA the other trusts.
So it's all nice and symmetrical.

The experimentation I did earlier indicates that there's some leakage
whereby openssl will pick up intermediate certs that are listed in
root.crt and use them to form a chain supporting the server.crt
certificate, but that doesn't fit nicely into the mental model.
I'm now of the opinion that we shouldn't recommend or even document
that way of doing things.

regards, tom lane

On 26/05/10 11:01, Tom Lane wrote:
> In principle, you could have the server and clients using totally
> nonoverlapping sets of trusted CAs (nonoverlapping root.crt lists),
> as long as each can chain its identity up to a CA the other trusts.
> So it's all nice and symmetrical.

... and it's exactly this cases that confuses keystore based clients
that may have multiple certs installed.

See the self-contained test case here:

http://www.postnewspapers.com.au/~craig/testcase.zip

... which includes a Pg datadir and configuration, the certificate
authority, the certificates, a detailed log of test case setup, the test
programs, logs of test output along with explanation of those logs, etc.

--
Craig Ringer

Tech-related writing: http://soapyfrogs.blogspot.com/

Craig Ringer <craig(at)postnewspapers(dot)com(dot)au> writes:
> See the self-contained test case here:
> http://www.postnewspapers.com.au/~craig/testcase.zip

Thanks for posting that; it makes it a lot easier to experiment with the
behavior of the Java software stack.

I've applied your patch along with some hacking on libpq. As far as
I can tell, things now work nicely with chained certificates on either
end, but it could definitely do with more testing if you have time to
poke at CVS HEAD.

I'm still a bit mystified about bug #5245 though. I can see two
possible explanations for that one:

1. The reporter was wrong about which server version he was using;
pre-8.4 servers would in fact not send the whole cert chain, cf
http://archives.postgresql.org/pgsql-committers/2009-05/msg00195.php

2. The reporter was wrong about the actual cause of his problem, and
despite his description, the true reason his Java client was failing
was the lack of SSL_CTX_set_client_CA_list().

Anyway, as far as I can tell the case described there works now.

regards, tom lane

On 27/05/10 10:05, Tom Lane wrote:
> Craig Ringer<craig(at)postnewspapers(dot)com(dot)au> writes:
>> See the self-contained test case here:
>> http://www.postnewspapers.com.au/~craig/testcase.zip
>
> Thanks for posting that; it makes it a lot easier to experiment with the
> behavior of the Java software stack.
>
> I've applied your patch along with some hacking on libpq. As far as
> I can tell, things now work nicely with chained certificates on either
> end, but it could definitely do with more testing if you have time to
> poke at CVS HEAD.

Thanks for that. I'll pull git, rebuild and re-test.

Handily this has got me running on HEAD builds for my testing/dev, so
I'll be able to set up with SR/HS and integrate that into my regular app
testing and development.

Sorry for getting grumpy, BTW. I was having a hard time explaining what
I was on about.

--
Craig Ringer