Re: SSL regression test suite

While working on the SSL refactoring patch, it struck me that we don't
have any regression tests for SSL support. A suite to test all the
different sslmodes etc. is essential before we can start implementing
alternatives to OpenSSL.

Now that we use TAP for testing client tools, I think we can use that to
test various SSL options too. I came up with the attached. Comments?

It currently assumes that the client's and the server's hostnames are
"postgres-client.test" and "postgres-server.test", respectively. That
makes it a bit tricky to run on a single systme. The README includes
instructions; basically you need to set up an additional loopback
device, and add entries to /etc/hosts for that.

It would make sense to separate the client and server portions of the
test so that you could run the server on one host and the client on
another. That's a TODO; I'm not sure how to do that.

- Heikki

On Mon, Aug 4, 2014 at 10:38 AM, Heikki Linnakangas
<hlinnakangas(at)vmware(dot)com> wrote:
> Now that we use TAP for testing client tools, I think we can use that to
> test various SSL options too. I came up with the attached. Comments?
>
> It currently assumes that the client's and the server's hostnames are
> "postgres-client.test" and "postgres-server.test", respectively. That makes
> it a bit tricky to run on a single systme. The README includes instructions;
> basically you need to set up an additional loopback device, and add entries
> to /etc/hosts for that.

That seems so onerous that I think few people will do it, and not
regularly, resulting in the tests breaking and nobody noticing.
Reconfiguring the loopback interface like that requires root
privilege, and won't survive a reboot, and doing it in the system
configuration will require hackery specific to the particular flavor
of Linux you're running, and probably other hackery on non-Linux
systems (never mind Windows). Why can't you make it work over
127.0.0.1?

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On 08/05/2014 10:46 PM, Robert Haas wrote:
> On Mon, Aug 4, 2014 at 10:38 AM, Heikki Linnakangas
> <hlinnakangas(at)vmware(dot)com> wrote:
>> Now that we use TAP for testing client tools, I think we can use that to
>> test various SSL options too. I came up with the attached. Comments?
>>
>> It currently assumes that the client's and the server's hostnames are
>> "postgres-client.test" and "postgres-server.test", respectively. That makes
>> it a bit tricky to run on a single systme. The README includes instructions;
>> basically you need to set up an additional loopback device, and add entries
>> to /etc/hosts for that.
>
> That seems so onerous that I think few people will do it, and not
> regularly, resulting in the tests breaking and nobody noticing.
> Reconfiguring the loopback interface like that requires root
> privilege, and won't survive a reboot, and doing it in the system
> configuration will require hackery specific to the particular flavor
> of Linux you're running, and probably other hackery on non-Linux
> systems (never mind Windows).

Yeah, you're probably right.

> Why can't you make it work over 127.0.0.1?

I wanted it to be easy to run the client and the server on different
hosts. As soon as we have more than one SSL implementation, it would be
really nice to do interoperability testing between a client and a server
using different implementations.

Also, to test sslmode=verify-full, where the client checks that the
server certificate's hostname matches the hostname that it connected to,
you need to have two aliases for the same server, one that matches the
certificate and one that doesn't. But I think I found a way around that
part; if the certificate is set up for "localhost", and connect to
"127.0.0.1", you get a mismatch.

So, I got rid of the DNS setup, it only depends localhost/127.0.0.1 now.
Patch attached. That means that it's not easy to run the client and the
server on different hosts, but we can improve that later.

- Heikki

On 2014-08-12 14:01:18 +0300, Heikki Linnakangas wrote:
> On 08/05/2014 10:46 PM, Robert Haas wrote:
> >Why can't you make it work over 127.0.0.1?
>
> I wanted it to be easy to run the client and the server on different hosts.
> As soon as we have more than one SSL implementation, it would be really nice
> to do interoperability testing between a client and a server using different
> implementations.
>
> Also, to test sslmode=verify-full, where the client checks that the server
> certificate's hostname matches the hostname that it connected to, you need
> to have two aliases for the same server, one that matches the certificate
> and one that doesn't. But I think I found a way around that part; if the
> certificate is set up for "localhost", and connect to "127.0.0.1", you get a
> mismatch.

Alternatively, and to e.g. test wildcard certs and such, I think you can
specify both host and hostaddr to connect to connect without actually
doing a dns lookup.

Greetings,

Andres Freund

--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

On 08/12/2014 02:28 PM, Andres Freund wrote:
> On 2014-08-12 14:01:18 +0300, Heikki Linnakangas wrote:
>> Also, to test sslmode=verify-full, where the client checks that the server
>> certificate's hostname matches the hostname that it connected to, you need
>> to have two aliases for the same server, one that matches the certificate
>> and one that doesn't. But I think I found a way around that part; if the
>> certificate is set up for "localhost", and connect to "127.0.0.1", you get a
>> mismatch.
>
> Alternatively, and to e.g. test wildcard certs and such, I think you can
> specify both host and hostaddr to connect to connect without actually
> doing a dns lookup.

Oh, I didn't know that's possible! Yeah, that's a good solution.

- Heikki

On 08/12/2014 03:53 PM, Heikki Linnakangas wrote:
> On 08/12/2014 02:28 PM, Andres Freund wrote:
>> On 2014-08-12 14:01:18 +0300, Heikki Linnakangas wrote:
>>> Also, to test sslmode=verify-full, where the client checks that the server
>>> certificate's hostname matches the hostname that it connected to, you need
>>> to have two aliases for the same server, one that matches the certificate
>>> and one that doesn't. But I think I found a way around that part; if the
>>> certificate is set up for "localhost", and connect to "127.0.0.1", you get a
>>> mismatch.
>>
>> Alternatively, and to e.g. test wildcard certs and such, I think you can
>> specify both host and hostaddr to connect to connect without actually
>> doing a dns lookup.
>
> Oh, I didn't know that's possible! Yeah, that's a good solution.

Here's a new version of the SSL regression suite I wrote earlier. It now
specifies both host and hostaddr in the connection string as Andres
suggested, so it no longer requires changes to network configuration. I
added a bunch of tests for the SAN feature that Alexey Klyukin wrote and
was committed earlier. Plus a lot of miscellaneous cleanup.

This probably needs some further cleanup before it's ready for
committing. One issues is that it creates a temporary cluster that
listens for TCP connections on localhost, which isn't safe on a
multi-user system.

- Heikki

On 10/06/2014 04:21 PM, Heikki Linnakangas wrote:
> Here's a new version of the SSL regression suite I wrote earlier. It now
> specifies both host and hostaddr in the connection string as Andres
> suggested, so it no longer requires changes to network configuration. I
> added a bunch of tests for the SAN feature that Alexey Klyukin wrote and
> was committed earlier. Plus a lot of miscellaneous cleanup.

And here's another version. It now includes tests for CRLs, and uses a
root CA that's used to sign the server and client CA's certificates, to
test that using intermediary CAs work.

> This probably needs some further cleanup before it's ready for
> committing. One issues is that it creates a temporary cluster that
> listens for TCP connections on localhost, which isn't safe on a
> multi-user system.

This issue remains. There isn't much we can do about it; SSL doesn't
work over Unix domain sockets. We could make it work, but that's a whole
different feature.

How do people feel about including this test suite in the source tree?
It's probably not suitable for running as part of "make check-world",
but it's extremely handy if you're working on a patch related to SSL.
I'd like to commit this, even if it has some rough edges. That way we
can improve it later, rather than have it fall into oblivion. Any
objections?

- Heikki

On Thu, Dec 04, 2014 at 02:42:41PM +0200, Heikki Linnakangas wrote:
> On 10/06/2014 04:21 PM, Heikki Linnakangas wrote:
> >This probably needs some further cleanup before it's ready for
> >committing. One issues is that it creates a temporary cluster that
> >listens for TCP connections on localhost, which isn't safe on a
> >multi-user system.
>
> This issue remains. There isn't much we can do about it; SSL doesn't work
> over Unix domain sockets. We could make it work, but that's a whole
> different feature.
>
> How do people feel about including this test suite in the source tree? It's
> probably not suitable for running as part of "make check-world",

What makes it unsuitable?

> but it's extremely handy if you're working on a patch related to
> SSL. I'd like to commit this, even if it has some rough edges. That
> way we can improve it later, rather than have it fall into oblivion.
> Any objections?

Not from me :)

Cheers,
David.
--
David Fetter <david(at)fetter(dot)org> http://fetter.org/
Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter
Skype: davidfetter XMPP: david(dot)fetter(at)gmail(dot)com

Remember to vote!
Consider donating to Postgres: http://www.postgresql.org/about/donate

Heikki Linnakangas <hlinnakangas(at)vmware(dot)com> writes:
> On 10/06/2014 04:21 PM, Heikki Linnakangas wrote:
>> This probably needs some further cleanup before it's ready for
>> committing. One issues is that it creates a temporary cluster that
>> listens for TCP connections on localhost, which isn't safe on a
>> multi-user system.

> This issue remains. There isn't much we can do about it; SSL doesn't
> work over Unix domain sockets. We could make it work, but that's a whole
> different feature.

> How do people feel about including this test suite in the source tree?
> It's probably not suitable for running as part of "make check-world",
> but it's extremely handy if you're working on a patch related to SSL.
> I'd like to commit this, even if it has some rough edges. That way we
> can improve it later, rather than have it fall into oblivion. Any
> objections?

As long as it's not run by any standard target, and there's some
documentation explaining why not, I see no reason it can't be in the
tree.

regards, tom lane

Heikki Linnakangas wrote:

> How do people feel about including this test suite in the source tree?

+1

> It's probably not suitable for running as part of "make check-world",
> but it's extremely handy if you're working on a patch related to SSL.
> I'd like to commit this, even if it has some rough edges. That way we
> can improve it later, rather than have it fall into oblivion. Any
> objections?

To prevent it from breaking, one idea is to have one or more buildfarm
animals that run this test as a separate module.

--
Álvaro Herrera http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

On Thu, Dec 04, 2014 at 02:42:41PM +0200, Heikki Linnakangas wrote:
> On 10/06/2014 04:21 PM, Heikki Linnakangas wrote:
> >This probably needs some further cleanup before it's ready for
> >committing. One issues is that it creates a temporary cluster that
> >listens for TCP connections on localhost, which isn't safe on a
> >multi-user system.
>
> This issue remains. There isn't much we can do about it; SSL doesn't work
> over Unix domain sockets. We could make it work, but that's a whole
> different feature.

A large subset of the test suite could be made secure. Omit or lock down
"trustdb", and skip affected tests. (Perhaps have an --unsafe-tests option to
reactivate them.) Instead of distributing frozen keys, generate all keys
on-demand. Ensure that key files have secure file modes from the start.
Having said that, ...

> How do people feel about including this test suite in the source tree? It's
> probably not suitable for running as part of "make check-world", but it's
> extremely handy if you're working on a patch related to SSL. I'd like to
> commit this, even if it has some rough edges. That way we can improve it
> later, rather than have it fall into oblivion. Any objections?

... +1 for having this suite in the tree, even if check-world ignores it.
Echoing Tom's comment, the README should mention its security weakness.

Thanks,
nm