Re: Plans for 8.4

I'm looking at shortlisting things that I'll be working on for 8.4. I've
thrown out a few small patches I had, but now am looking at these
projects:

* Hot Standby
* Logical replication via transaction log
* Integration with synchronous replication

* Join removal
* stats hooks
* DDL locking reductions

Initial design work on them is mostly done, just need to start posting
designs and patches. I've got a early days prototype of Hot Standby that
allows connection, plus months of thinking on the other topics, so I'm
further ahead than it may previously have appeared. All of this is
dependent upon sponsorship... THANK YOU to current sponsors.

I've had to break off work on MERGE now. For lack of time, plus the
presence of some unresolved details in the design that make it
questionable whether it would be fully accepted.

I've also punted on a few other items that look like more time than
value to the project, in my opinion. But it might be possible to pick up
later, depending what happens in next few months.

--
Simon Riggs www.2ndQuadrant.com
PostgreSQL Training, Services and Support

Hi,

Simon Riggs wrote:
> * Logical replication via transaction log
> * Integration with synchronous replication

I'm curious on what you mean by these two points. AFAIK you are speaking
of "logical" replication for methods which don't rely on the internal
storage format of the database, but instead replicate closer to the SQL
level. But the transaction log (the XLOG for Postres, REDO log for
Oracle, etc..) is certainly bound to the storage format, so I'd classify
the log shipping approaches as "physical" replication. What is it you
are talking about?

What do you plan to integrate with synchronous replication?

Did you consider having a look at the Postgres-R project?

Regards

Markus Wanner

On Jul 22, 2008, at 08:35, Simon Riggs wrote:

> * Hot Standby
> * Logical replication via transaction log
> * Integration with synchronous replication

Getting these in 8.4 would so rock. Thank you!

David

On Tue, 2008-07-22 at 18:37 +0200, Markus Wanner wrote:

> Simon Riggs wrote:
> > * Logical replication via transaction log
> > * Integration with synchronous replication
>
> I'm curious on what you mean by these two points. AFAIK you are speaking
> of "logical" replication for methods which don't rely on the internal
> storage format of the database, but instead replicate closer to the SQL
> level. But the transaction log (the XLOG for Postres, REDO log for
> Oracle, etc..) is certainly bound to the storage format, so I'd classify
> the log shipping approaches as "physical" replication. What is it you
> are talking about?

Reconstructing SQL from WAL, allowing logical apply. So yes, you're
right, the actual replication of the data from one node to another is
physical - its the final apply of the changes that is logical. So this
fits neatly with our commitment to put synchronous replication into
server.

It allows WAL to be used as the replication transport, which reduces the
impact of tracking changes via triggers. The presence of commit records
in the sequence can be used to parallelize the apply phase if required.

I'm proposing to put the mechanisms in place to allow WAL to be used by
other replication projects. Which turn out to be relatively small
changes to the main server.

--
Simon Riggs www.2ndQuadrant.com
PostgreSQL Training, Services and Support

On Tue, 2008-07-22 at 10:23 -0700, David E. Wheeler wrote:
> On Jul 22, 2008, at 08:35, Simon Riggs wrote:
>
> > * Hot Standby
> > * Logical replication via transaction log
> > * Integration with synchronous replication
>
> Getting these in 8.4 would so rock.

Agreed. Thank me when its done, 'cos it ain't done yet by a long way!

--
Simon Riggs www.2ndQuadrant.com
PostgreSQL Training, Services and Support

Simon Riggs <simon(at)2ndquadrant(dot)com> writes:
> On Tue, 2008-07-22 at 18:37 +0200, Markus Wanner wrote:
>> What is it you
>> are talking about?

> Reconstructing SQL from WAL, allowing logical apply.
> ...
> I'm proposing to put the mechanisms in place to allow WAL to be used by
> other replication projects. Which turn out to be relatively small
> changes to the main server.

[ retrieves eyebrows from ceiling... ] I doubt that's even possible,
let alone a "relatively small change".

regards, tom lane

Hi,

Simon Riggs wrote:
> Reconstructing SQL from WAL, allowing logical apply. So yes, you're
> right, the actual replication of the data from one node to another is
> physical - its the final apply of the changes that is logical. So this
> fits neatly with our commitment to put synchronous replication into
> server.

Aha, thanks for your explanation.

> It allows WAL to be used as the replication transport, which reduces the
> impact of tracking changes via triggers. The presence of commit records
> in the sequence can be used to parallelize the apply phase if required.

I note you are comparing against trigger based replication solutions.

> I'm proposing to put the mechanisms in place to allow WAL to be used by
> other replication projects. Which turn out to be relatively small
> changes to the main server.

Comparing with the Postgres-R way to do it, there's a difference in the
serialized change set format. Where Postgres-R uses it's a specialized
format, you are proposing to use the existing WAL format to transfer
change set data, which is architecture dependent and quite tightly bound
to the physical format on disc. It cannot be changed to also carry other
information required for replication, without having an impact on WAL
for single node installations. I think the CPU load to serialize and
deserialize is pretty similar and most probably negligible.

Regards

Markus

On Tue, 2008-07-22 at 16:54 -0400, Tom Lane wrote:
> Simon Riggs <simon(at)2ndquadrant(dot)com> writes:
> > On Tue, 2008-07-22 at 18:37 +0200, Markus Wanner wrote:
> >> What is it you
> >> are talking about?
>
> > Reconstructing SQL from WAL, allowing logical apply.
> > ...
> > I'm proposing to put the mechanisms in place to allow WAL to be used by
> > other replication projects. Which turn out to be relatively small
> > changes to the main server.
>
> [ retrieves eyebrows from ceiling... ] I doubt that's even possible,
> let alone a "relatively small change".

I'm talking about building the transport layer into Postgres core only.
SMOP outside of core, with the right changes.

There's always potential blockers to any problem. We'll see when I
publish the design.

--
Simon Riggs www.2ndQuadrant.com
PostgreSQL Training, Services and Support

What's the time frame for 8.4?

I'm making no promises, but what would people think of a hostgss hba
option?

Using it would imply the gssapi/sspi authentication option. It would
be mutually exclusive of the ssl link-encryption option. It would
support strong encryption of the whole connection without the need to
get X509 certs deployed (which would be a big win if you're using
gssapi/sspi authentication anyway).

The thing that prevented me from including it in the gssapi patches I
did for 8.3 was that I couldn't disentangle the program logic to the
point of inserting the gssapi security layer code above the SSL code
and below everything else. I'm thinking that doing both is pretty
much an edge case, so I propose to do gssapi security layers instead
of SSL. The mods are a lot more obvious.

I'm *NOT* proposing to make build support of gssapi security layers
exclusive of SSL. You might, for example, configure a server to
support username/password over SSL for intra-net addresses, but
support gssapi for Internet addresses.

------------------------------------------------------
The opinions expressed in this message are mine,
not those of Caltech, JPL, NASA, or the US Government.
Henry(dot)B(dot)Hotz(at)jpl(dot)nasa(dot)gov, or hbhotz(at)oxy(dot)edu

* Henry B. Hotz (hbhotz(at)oxy(dot)edu) wrote:
> I'm making no promises, but what would people think of a hostgss hba
> option?

As described, sounds like a win to me. It'd be very nice to be able to
just use GSSAPI encryption on the link. That, combined w/ Magnus' work
on username/princ mappings, would really bring PostgreSQL up to date wrt
GSSAPI support.

It'd really be great to have this support in the ODBC and JDBC drivers
too.. I think in JDBC it might 'just work', I'm less sure about ODBC.

As a practical question- would you really need a seperate explicit
pg_hba option for it? It'd be nice to be able to require it, if
desired, but that strikes me as more sensible as an option to the 'gss'
auth mechanism?

Thanks!

Stephen

"Henry B. Hotz" <hbhotz(at)oxy(dot)edu> writes:
> What's the time frame for 8.4?

http://wiki.postgresql.org/wiki/PostgreSQL_8.4_Development_Plan

Executive summary: new-feature patches had better be submitted before
1 November.

> I'm making no promises, but what would people think of a hostgss hba
> option?

Not qualified to comment on the usefulness of this ...

regards, tom lane

Stephen Frost wrote:
> * Henry B. Hotz (hbhotz(at)oxy(dot)edu) wrote:
>> I'm making no promises, but what would people think of a hostgss hba
>> option?
>
> As described, sounds like a win to me. It'd be very nice to be able to
> just use GSSAPI encryption on the link. That, combined w/ Magnus' work
> on username/princ mappings, would really bring PostgreSQL up to date wrt
> GSSAPI support.

Yeah, +1 on this feature, it would be quite useful.

> It'd really be great to have this support in the ODBC and JDBC drivers
> too.. I think in JDBC it might 'just work', I'm less sure about ODBC.

ODBC will need hackery I think. They use libpq for authentication only,
but have their own SSL code and such. I do think ODBC would be a fairly
major point to it being a success, though, so it'd be good if a plan
could be secured for it. But it's not a showstopper, of course.

> As a practical question- would you really need a seperate explicit
> pg_hba option for it? It'd be nice to be able to require it, if
> desired, but that strikes me as more sensible as an option to the 'gss'
> auth mechanism?

Yeah, if we can get rid of that, that'd be good. The stuff I'm working
on will allow us to have multiple parameters for each row in name/value
pairs, so if we could use that, it'd be better. (I've been considering
changing how host/hostssl work that way as well - by having a parameter
similar to what we have on the client side with sslmode=...)

A thought that I came across - is it even possible to use GSSAPI
encryption *without* using GSSAPI authentication? If not, it really
seems like it should belong more in the parameter part of the field.
Since in that case it is also not possible to enable encryption *before*
authentication, or is it?

//Magnus

On Jul 31, 2008, at 7:58 AM, Magnus Hagander wrote:

> Stephen Frost wrote:
>> * Henry B. Hotz (hbhotz(at)oxy(dot)edu) wrote:
>>> I'm making no promises, but what would people think of a hostgss hba
>>> option?
>>
>> As described, sounds like a win to me. It'd be very nice to be
>> able to
>> just use GSSAPI encryption on the link. That, combined w/ Magnus'
>> work
>> on username/princ mappings, would really bring PostgreSQL up to
>> date wrt
>> GSSAPI support.
>
> Yeah, +1 on this feature, it would be quite useful.
>
>
>> It'd really be great to have this support in the ODBC and JDBC
>> drivers
>> too.. I think in JDBC it might 'just work', I'm less sure about
>> ODBC.
>
> ODBC will need hackery I think. They use libpq for authentication
> only,
> but have their own SSL code and such. I do think ODBC would be a
> fairly
> major point to it being a success, though, so it'd be good if a plan
> could be secured for it. But it's not a showstopper, of course.

I don't know enough about ODBC. If ODBC does SSL independently of PG
then it requires thought by someone who understands ODBC.

>
>> As a practical question- would you really need a seperate explicit
>> pg_hba option for it? It'd be nice to be able to require it, if
>> desired, but that strikes me as more sensible as an option to the
>> 'gss'
>> auth mechanism?
>
> Yeah, if we can get rid of that, that'd be good. The stuff I'm working
> on will allow us to have multiple parameters for each row in name/
> value
> pairs, so if we could use that, it'd be better. (I've been considering
> changing how host/hostssl work that way as well - by having a
> parameter
> similar to what we have on the client side with sslmode=...)
>
> A thought that I came across - is it even possible to use GSSAPI
> encryption *without* using GSSAPI authentication? If not, it really
> seems like it should belong more in the parameter part of the field.
> Since in that case it is also not possible to enable encryption
> *before*
> authentication, or is it?

You're on the right track. My problem isn't the hba file parsing at
all.

My problem is the interaction between the buffering logic and the
encrypted I/O routines. The technical issue is that to make a GSSAPI
security layer independent of SSL you need to invent a whole new
buffering layer. That's a lot of work, and it only buys you the
ability to do both SSL and GSSAPI at the same time. That doesn't seem
worth it.

The code being affected is what's currently configured in column 1 of
hba. The ability to use the new capability requires that SSL *NOT* be
configured in column 1 for the relevant client addresses. In short,
no, it doesn't make sense to make it an option to the gss
authentication method, even though it requires it. If we make it an
option to the gss authentication method it would still need to act
like it was specified in column 1, which would be confusing.

GSSAPI security layers are negotiated after the authentication (or at
least after the start of authentication). There are GSSAPI status
flags that indicate if the security layer is available yet. The
GSSAPI security layer code would check those flags and gss_wrap() or
not accordingly. (-: There's a flush() or two from my original patch
that will need to be added back in, otherwise we'll encrypt a message
that tells the other end how to decrypt messages. Not a big deal. ;-)

------------------------------------------------------
The opinions expressed in this message are mine,
not those of Caltech, JPL, NASA, or the US Government.
Henry(dot)B(dot)Hotz(at)jpl(dot)nasa(dot)gov, or hbhotz(at)oxy(dot)edu

Henry B. Hotz wrote:
>
> On Jul 31, 2008, at 7:58 AM, Magnus Hagander wrote:
>
>> Stephen Frost wrote:
>>> * Henry B. Hotz (hbhotz(at)oxy(dot)edu) wrote:
>>>> I'm making no promises, but what would people think of a hostgss hba
>>>> option?
>>>
>>> As described, sounds like a win to me. It'd be very nice to be able to
>>> just use GSSAPI encryption on the link. That, combined w/ Magnus' work
>>> on username/princ mappings, would really bring PostgreSQL up to date wrt
>>> GSSAPI support.
>>
>> Yeah, +1 on this feature, it would be quite useful.
>>
>>
>>> It'd really be great to have this support in the ODBC and JDBC drivers
>>> too.. I think in JDBC it might 'just work', I'm less sure about ODBC.
>>
>> ODBC will need hackery I think. They use libpq for authentication only,
>> but have their own SSL code and such. I do think ODBC would be a fairly
>> major point to it being a success, though, so it'd be good if a plan
>> could be secured for it. But it's not a showstopper, of course.
>
> I don't know enough about ODBC. If ODBC does SSL independently of PG
> then it requires thought by someone who understands ODBC.

I just meant please consider coordinating with the ODBC folks to make
sure it gets in there as well - and in time for the same release.

>>> As a practical question- would you really need a seperate explicit
>>> pg_hba option for it? It'd be nice to be able to require it, if
>>> desired, but that strikes me as more sensible as an option to the 'gss'
>>> auth mechanism?
>>
>> Yeah, if we can get rid of that, that'd be good. The stuff I'm working
>> on will allow us to have multiple parameters for each row in name/value
>> pairs, so if we could use that, it'd be better. (I've been considering
>> changing how host/hostssl work that way as well - by having a parameter
>> similar to what we have on the client side with sslmode=...)
>>
>> A thought that I came across - is it even possible to use GSSAPI
>> encryption *without* using GSSAPI authentication? If not, it really
>> seems like it should belong more in the parameter part of the field.
>> Since in that case it is also not possible to enable encryption *before*
>> authentication, or is it?
>
> You're on the right track. My problem isn't the hba file parsing at all.
>
> My problem is the interaction between the buffering logic and the
> encrypted I/O routines. The technical issue is that to make a GSSAPI
> security layer independent of SSL you need to invent a whole new
> buffering layer. That's a lot of work, and it only buys you the ability
> to do both SSL and GSSAPI at the same time. That doesn't seem worth it.

Yeah, there seems to be no general point in that.

However, implementing a layer there might have other benefits. Such as
being able to use other SSL implementations (right now we only do
OpenSSL. There has been talk about GnuTLS, and it would be good to be
able to do schannel)

> The code being affected is what's currently configured in column 1 of
> hba. The ability to use the new capability requires that SSL *NOT* be
> configured in column 1 for the relevant client addresses. In short, no,
> it doesn't make sense to make it an option to the gss authentication
> method, even though it requires it. If we make it an option to the gss
> authentication method it would still need to act like it was specified
> in column 1, which would be confusing.

Does this hold even if we move the "hostssl" stuff into a parameter "at
the end"? I was thinking maybe something like:
host all all 0.0.0.0/0 md5 ssl=require
host all all 0.0.0.0/0 gss ssl=forbid gssencrypt=require

(you get the idea)

> GSSAPI security layers are negotiated after the authentication (or at
> least after the start of authentication). There are GSSAPI status flags
> that indicate if the security layer is available yet. The GSSAPI
> security layer code would check those flags and gss_wrap() or not
> accordingly. (-: There's a flush() or two from my original patch that
> will need to be added back in, otherwise we'll encrypt a message that
> tells the other end how to decrypt messages. Not a big deal. ;-)

Ok, makes sense.

//Magnus

Added to TODO:

* Add 'hostgss' pg_hba.conf option to allow GSS link-level encryption

http://archives.postgresql.org/pgsql-hackers/2008-07/msg01454.php

---------------------------------------------------------------------------

Henry B. Hotz wrote:
> What's the time frame for 8.4?
>
> I'm making no promises, but what would people think of a hostgss hba
> option?
>
> Using it would imply the gssapi/sspi authentication option. It would
> be mutually exclusive of the ssl link-encryption option. It would
> support strong encryption of the whole connection without the need to
> get X509 certs deployed (which would be a big win if you're using
> gssapi/sspi authentication anyway).
>
> The thing that prevented me from including it in the gssapi patches I
> did for 8.3 was that I couldn't disentangle the program logic to the
> point of inserting the gssapi security layer code above the SSL code
> and below everything else. I'm thinking that doing both is pretty
> much an edge case, so I propose to do gssapi security layers instead
> of SSL. The mods are a lot more obvious.
>
> I'm *NOT* proposing to make build support of gssapi security layers
> exclusive of SSL. You might, for example, configure a server to
> support username/password over SSL for intra-net addresses, but
> support gssapi for Internet addresses.
>
> ------------------------------------------------------
> The opinions expressed in this message are mine,
> not those of Caltech, JPL, NASA, or the US Government.
> Henry(dot)B(dot)Hotz(at)jpl(dot)nasa(dot)gov, or hbhotz(at)oxy(dot)edu
>
>
>
>
> --
> Sent via pgsql-hackers mailing list (pgsql-hackers(at)postgresql(dot)org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-hackers

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +