Re: Planning incompatibilities for Postgres 10.0

There are a number of changes we'd probably like to make to the way
things work in Postgres. This thread is not about discussing what
those are, just to say that requirements exist and have been discussed
in various threads over time.

The constraint on such changes is that we've decided that we must have
an upgrade path from release to release.

So I'd like to make a formal suggestion of a plan for how we cope with this:

1. Implement online upgrade in 9.4 via the various facilities we have
in-progress. That looks completely possible.

2. Name the next release after that 10.0 (would have been 9.5). We
declare now that
a) 10.0 will support on-line upgrade from 9.4 (only)
b) various major incompatibilities will be introduced in 10.0 - the
change in release number will indicate to everybody that is the case
c) agree that there will be no pg_upgrade patch from 9.4 to 10.0, so
that we will not be constrained by that

This plan doesn't presume any particular change. Each change would
need to be discussed on a separate thread, with a separate case for
each. All I'm suggesting is that we have a coherent plan for the
timing of such changes, so we can bundle them together into one
release.

By doing this now we give ourselves lots of time to plan changes that
will see us good for another decade. If we don't do this, then we
simply risk losing the iniative by continuing to support legacy
formats and approaches.

--
Simon Riggs http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

On Sat, May 25, 2013 at 4:39 AM, Simon Riggs <simon(at)2ndquadrant(dot)com> wrote:
> There are a number of changes we'd probably like to make to the way
> things work in Postgres. This thread is not about discussing what
> those are, just to say that requirements exist and have been discussed
> in various threads over time.
>
> The constraint on such changes is that we've decided that we must have
> an upgrade path from release to release.
>
> So I'd like to make a formal suggestion of a plan for how we cope with this:
>
> 1. Implement online upgrade in 9.4 via the various facilities we have
> in-progress. That looks completely possible.
>
> 2. Name the next release after that 10.0 (would have been 9.5). We
> declare now that
> a) 10.0 will support on-line upgrade from 9.4 (only)
> b) various major incompatibilities will be introduced in 10.0 - the
> change in release number will indicate to everybody that is the case
> c) agree that there will be no pg_upgrade patch from 9.4 to 10.0, so
> that we will not be constrained by that
>
> This plan doesn't presume any particular change. Each change would
> need to be discussed on a separate thread, with a separate case for
> each. All I'm suggesting is that we have a coherent plan for the
> timing of such changes, so we can bundle them together into one
> release.
>
> By doing this now we give ourselves lots of time to plan changes that
> will see us good for another decade. If we don't do this, then we
> simply risk losing the iniative by continuing to support legacy
> formats and approaches.

Huh. I don't think that bumping the version number to 10.0 vs 9.5 is
justification to introduce breaking changes. In fact, I would rather
see 10.0 be the version where we formally stop doing that. I
understand that some stuff needs to be improved but it often doesn't
seem to be worth the cost in the long run.

merlin

On Sat, 2013-05-25 at 10:39 +0100, Simon Riggs wrote:
> The constraint on such changes is that we've decided that we must have
> an upgrade path from release to release.

Is this proposal only relaxing the binary upgrade requirement, or would
it also relax other compatibility requirements, such as language and API
compatibility?

We need a couple major drivers of the incompatibility that really show
users some value for going through the upgrade pain. Preferably, at
least one would be a serious performance boost, because the users that
encounter the most logical upgrade pain are also the ones that need a
performance boost the most.

Before we set a specific schedule, I think it would be a good idea to
start prototyping some performance improvements that involve breaking
the data format. Then, depending on how achievable it is, we can plan
for however many more 9.X releases we think we need. That being said, I
agree with you that planning in advance is important here, so that
everyone knows when they need to get format-breaking changes in by.

Regards,
Jeff Davis

On 25 May 2013 18:13, Jeff Davis <pgsql(at)j-davis(dot)com> wrote:
> On Sat, 2013-05-25 at 10:39 +0100, Simon Riggs wrote:
>> The constraint on such changes is that we've decided that we must have
>> an upgrade path from release to release.
>
> Is this proposal only relaxing the binary upgrade requirement, or would
> it also relax other compatibility requirements, such as language and API
> compatibility?

I'm suggesting that as many as possible changes we would like to make
can happen in one release. This is for the benefit of users, so we
dont make every release a source of incompatibilities.

And that release should be the first one where we have online upgrade
possible, which is one after next.

--
Simon Riggs http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

On Sat, May 25, 2013 at 10:39:30AM +0100, Simon Riggs wrote:
> There are a number of changes we'd probably like to make to the way
> things work in Postgres. This thread is not about discussing what
> those are, just to say that requirements exist and have been discussed
> in various threads over time.
>
> The constraint on such changes is that we've decided that we must have
> an upgrade path from release to release.
>
> So I'd like to make a formal suggestion of a plan for how we cope with this:
>
> 1. Implement online upgrade in 9.4 via the various facilities we have
> in-progress. That looks completely possible.
>
> 2. Name the next release after that 10.0 (would have been 9.5). We
> declare now that
> a) 10.0 will support on-line upgrade from 9.4 (only)
> b) various major incompatibilities will be introduced in 10.0 - the
> change in release number will indicate to everybody that is the case
> c) agree that there will be no pg_upgrade patch from 9.4 to 10.0, so
> that we will not be constrained by that

Assuming online upgrade is going to require logical replication, you are
also assuming 2x storage as you need to have a second cluster to perform
the upgrade. pg_upgrade would still be needed to upgrade a cluster
in-place.

This sounds like, "I created a new tool which does some of what the old
tool does. Let's break the old tool to allow some unspecified changes I
might want to make." I consider this thread to be not thought-through,
obviously.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ It's impossible for everything to be true. +

On 25 May 2013 21:44, Bruce Momjian <bruce(at)momjian(dot)us> wrote:
> On Sat, May 25, 2013 at 10:39:30AM +0100, Simon Riggs wrote:
>> There are a number of changes we'd probably like to make to the way
>> things work in Postgres. This thread is not about discussing what
>> those are, just to say that requirements exist and have been discussed
>> in various threads over time.
>>
>> The constraint on such changes is that we've decided that we must have
>> an upgrade path from release to release.
>>
>> So I'd like to make a formal suggestion of a plan for how we cope with this:
>>
>> 1. Implement online upgrade in 9.4 via the various facilities we have
>> in-progress. That looks completely possible.
>>
>> 2. Name the next release after that 10.0 (would have been 9.5). We
>> declare now that
>> a) 10.0 will support on-line upgrade from 9.4 (only)
>> b) various major incompatibilities will be introduced in 10.0 - the
>> change in release number will indicate to everybody that is the case
>> c) agree that there will be no pg_upgrade patch from 9.4 to 10.0, so
>> that we will not be constrained by that
>
> Assuming online upgrade is going to require logical replication, you are
> also assuming 2x storage as you need to have a second cluster to perform
> the upgrade.

The people that want online upgrade already have 1+ other systems to
do this with.

> pg_upgrade would still be needed to upgrade a cluster
> in-place.
> This sounds like, "I created a new tool which does some of what the old
> tool does. Let's break the old tool to allow some unspecified changes I
> might want to make."

I haven't argued against pg_upgrade in general, nor said anything
about breaking it. I proposed that we don't support a pg_upgrade path
between two near-future releases, as a way of introducing
incompatibilities. After that, we would continue to use pg_upgrade for
later releases.

Logical replication is being developed, which gives us a complete code
path for doing what we'd need to do. The most important thing is we
wouldn't need to develop any other code that exists just for upgrade.

Writing special code just for pg_upgrade will take a lot of work.
Running that code would mean pg_upgrade would touch the actual
database, which would be down for a long time while it runs. And if it
hits a bug during or after, you're hosed. So you'd need to take a full
backup before you started the process, probably storing it on disk
somewhere and so you would need x2 disk space with this route also.
Specialised code is less well tested, which means bugs are more likely
to occur and tends to perform more poorly. Not only that, but the
first person to want an incompatibility gets to write all the code
needed and take responsibility for the bugs. I can't comment for
others, but I can say I would not personally choose that route - it
looks both expensive and risky.

> I consider this thread to be not thought-through, obviously.

My proposal has had lots of serious consideration, but that is not the
topic of this thread.

The title of the thread is a general one, with a clear objective.

I'm looking for a way forwards that allows us to introduce the changes
that many have proposed and which regrettably result in
incompatibilities. If we have no plan I think its likely it will never
happen and it is currently blocking useful change.

Please explain what you consider to be a better plan, so we can judge
all proposals together.

--
Simon Riggs http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

On Sun, May 26, 2013 at 10:53:37AM +0100, Simon Riggs wrote:
> > I consider this thread to be not thought-through, obviously.
>
> My proposal has had lots of serious consideration, but that is not the
> topic of this thread.
>
> The title of the thread is a general one, with a clear objective.
>
> I'm looking for a way forwards that allows us to introduce the changes
> that many have proposed and which regrettably result in
> incompatibilities. If we have no plan I think its likely it will never
> happen and it is currently blocking useful change.
>
> Please explain what you consider to be a better plan, so we can judge
> all proposals together.

I agree with the idea of using logical replication as a way to do
pg_upgrade version-breaking releases. What I don't know is what
incompatible changes are pending that would require this.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ It's impossible for everything to be true. +

On Sun, May 26, 2013 at 09:18:11AM -0400, Bruce Momjian wrote:
> On Sun, May 26, 2013 at 10:53:37AM +0100, Simon Riggs wrote:
> > > I consider this thread to be not thought-through, obviously.
> >
> > My proposal has had lots of serious consideration, but that is not the
> > topic of this thread.
> >
> > The title of the thread is a general one, with a clear objective.
> >
> > I'm looking for a way forwards that allows us to introduce the changes
> > that many have proposed and which regrettably result in
> > incompatibilities. If we have no plan I think its likely it will never
> > happen and it is currently blocking useful change.
> >
> > Please explain what you consider to be a better plan, so we can judge
> > all proposals together.
>
> I agree with the idea of using logical replication as a way to do
> pg_upgrade version-breaking releases. What I don't know is what
> incompatible changes are pending that would require this.

Sorry I was unclear. When I said "not thought-through", I meant, you
need to start with the _reason_ we need to break pg_upgrade in an
upcoming version, then we can start to plan how to do it. The logical
replication idea is a good one for getting us through pg_upgrade
version-breaking releases.

I am fine with breaking pg_upgrade, but I just don't see the pending
reason at this point.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ It's impossible for everything to be true. +

On 05/26/2013 04:22 PM, Bruce Momjian wrote:
> On Sun, May 26, 2013 at 09:18:11AM -0400, Bruce Momjian wrote:
>> On Sun, May 26, 2013 at 10:53:37AM +0100, Simon Riggs wrote:
>>>> I consider this thread to be not thought-through, obviously.
>>> My proposal has had lots of serious consideration, but that is not the
>>> topic of this thread.
>>>
>>> The title of the thread is a general one, with a clear objective.
>>>
>>> I'm looking for a way forwards that allows us to introduce the changes
>>> that many have proposed and which regrettably result in
>>> incompatibilities. If we have no plan I think its likely it will never
>>> happen and it is currently blocking useful change.
>>>
>>> Please explain what you consider to be a better plan, so we can judge
>>> all proposals together.
>> I agree with the idea of using logical replication as a way to do
>> pg_upgrade version-breaking releases. What I don't know is what
>> incompatible changes are pending that would require this.
> Sorry I was unclear. When I said "not thought-through", I meant, you
> need to start with the _reason_ we need to break pg_upgrade in an
> upcoming version, then we can start to plan how to do it. The logical
> replication idea is a good one for getting us through pg_upgrade
> version-breaking releases.
>
> I am fine with breaking pg_upgrade, but I just don't see the pending
> reason at this point.
Not sure which ones Simon meant, but at least any new/better
storage manager would seem to me to be requiring
a non-pg_upgrade upgrade path unless we require the storage manager
to also include a parallel implementation of pg_upgrade.

The family of possible storage magers here would include column stores,
distributed / partitioned / replicated memory-only / index-structured / ...
storages which all could have advantages in certain situations and whic
all
need an upgrade path.

While you could do this using sequance of first pg_upgrading and then doing
some internal data migration to new storage manager doing this in one go
would be much smoother.

--
Hannu Krosing
PostgreSQL Consultant
Performance, Scalability and High Availability
2ndQuadrant Nordic OÜ

> Not sure which ones Simon meant, but at least any new/better
> storage manager would seem to me to be requiring
> a non-pg_upgrade upgrade path unless we require the storage manager
> to also include a parallel implementation of pg_upgrade.

Isn't this a bit of horse-cart inversion here? We just hashed out a
tentative, incomplete pseudo-spec for storage managers *yesterday*. We
don't have a complete spec at this point, let alone a development plan,
and it's entirely possible that we'll be able to implement SMs without
breaking pgupgrade.

It's also not at all clear that we can develop SMs in less than 2 years.
I tend to think it unlikely.

First, let's have a few features for which breaking binary compatibility
is a necessity or a clear benefit. Then we'll schedule when to break them.

--
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

* Josh Berkus (josh(at)agliodbs(dot)com) wrote:
> and it's entirely possible that we'll be able to implement SMs without
> breaking pgupgrade.

I'd certainly hope so.. It's certainly not obvious, to me at least,
why a new SM or supporting any of those features would require
breaking pg_upgrade. Perhaps there's something I'm not seeing there,
but it had better be a *really* good reason..

btw, has anyone posted the SM API proposal..? Unfortunately, I think I
had to leave before that was hashed out..

> First, let's have a few features for which breaking binary compatibility
> is a necessity or a clear benefit. Then we'll schedule when to break them.

Agreed.

Thanks,

Stephen

Stephen Frost <sfrost(at)snowman(dot)net> writes:
> btw, has anyone posted the SM API proposal..? Unfortunately, I think I
> had to leave before that was hashed out..

There isn't one yet. We think we understand where the pain points are,
but there's still a long way to go to have a proposal.

regards, tom lane

The assumption that we ought to plan expressly for an incompatibility that
essentially discards pg_upgrade seems premature, particularly in advance of
would-be solutions that, in some cases, mightn't actually work.

If pg_upgrade doesn't work, then, at present, the plausible solutions are
to either dump and restore, which might take way too long, or use one of
the logical replication systems (e.g. - Slony, Londiste, or similar, in the
absence of the would-be built-in logical replication).

Unfortunately, there are significant scenarios where none of these work,
particularly for data warehouse-like systems where the database size is so
large that the users cannot afford the disk space to construct a replica.
It sure seems premature to intentionally leave that set of users out in the
cold.

On 05/25/2013 05:39 PM, Simon Riggs wrote:
> 2. Name the next release after that 10.0 (would have been 9.5). We
> declare now that
> a) 10.0 will support on-line upgrade from 9.4 (only)
> b) various major incompatibilities will be introduced in 10.0 - the
> change in release number will indicate to everybody that is the case
> c) agree that there will be no pg_upgrade patch from 9.4 to 10.0, so
> that we will not be constrained by that
While we're talking about changing things, what about:

- Switching to single-major-version release numbering. The number of
people who say "PostgreSQL 9.x" is amazing; even *packagers* get this
wrong and produce "postgresql-9" packages. Witness Amazon Linux's awful
PostgreSQL packages for example. Going to PostgreSQL 10.0, 11.0, 12.0,
etc with a typical major/minor scheme might be worth considering.

- s/cluster/server/g . Just because "cluster" is historical usage
doesn't make it any less confusing for users.

*dives for asbestos fire suit*

--
Craig Ringer http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

On Mon, May 27, 2013 at 2:01 PM, Craig Ringer <craig(at)2ndquadrant(dot)com> wrote:

> On 05/25/2013 05:39 PM, Simon Riggs wrote:
> - Switching to single-major-version release numbering. The number of
> people who say "PostgreSQL 9.x" is amazing; even *packagers* get this
> wrong and produce "postgresql-9" packages. Witness Amazon Linux's awful
> PostgreSQL packages for example. Going to PostgreSQL 10.0, 11.0, 12.0,
> etc with a typical major/minor scheme might be worth considering.
>
In this case you don't even need the 2nd digit...
Btw, -1 for the idea, as it would remove the possibility to tell that a new
major release incrementing the 1st digit of version number brings more
enhancement than normal major releases incrementing the 1st digit. This was
the case for 9.0, helping people in remembering that streaming replication
has been introduced from 9.x series.
--
Michael

On Sun, May 26, 2013 at 09:18:41PM -0400, Stephen Frost wrote:
> * Josh Berkus (josh(at)agliodbs(dot)com) wrote:
> > and it's entirely possible that we'll be able to implement SMs without
> > breaking pgupgrade.
>
> I'd certainly hope so.. It's certainly not obvious, to me at least,
> why a new SM or supporting any of those features would require
> breaking pg_upgrade. Perhaps there's something I'm not seeing there,
> but it had better be a *really* good reason..

If I had to _guess_, I would say users who are using the default storage
manager would still be able to use pg_upgrade, and those using
non-default storage managers perhaps can't.

But, again, this is all so hypothetical that it doesn't seem worth
talking about. My big point is that someone came to me at PGCon asking
if I knew anything about why Simon thought we needed to break pg_upgrade
in <2 years, and I said no, so I had go digging into my email to find
out what was going on. Simon has a very visible position in the
community, so when he suggests something, people take it seriously,
which means I have to address it. I would prefer if there was more
thought put into the ideas before they are posted.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ It's impossible for everything to be true. +

* Bruce Momjian (bruce(at)momjian(dot)us) wrote:
> If I had to _guess_, I would say users who are using the default storage
> manager would still be able to use pg_upgrade, and those using
> non-default storage managers perhaps can't.

That would make sense.

> But, again, this is all so hypothetical that it doesn't seem worth
> talking about.

Having a specific list of "these are the things we want to change, and
why, and here is why pg_upgrade can't support it" would be much more
useful to work from, I agree.

That said, many discussions and ideas do get shut down, perhaps too
early, because of pg_upgrade considerations. If we had a plan to have
an incompatible release in the future, those ideas and discussions might
be able to progress to a point where we determine it's worth it to take
the pain of a non-pg_upgrade-supported release. That's a bit of a
stretch, in my view, but I suppose it's possible. Even so though, I
would suggest that we put together a wiki page to list out those items
and encourage people to add to such a list; perhaps having an item on
that list would make discussion about it progress beyond "it breaks
pg_upgrade".

Thanks,

Stephen

On Mon, May 27, 2013 at 08:26:48AM -0400, Stephen Frost wrote:
> * Bruce Momjian (bruce(at)momjian(dot)us) wrote:
> > If I had to _guess_, I would say users who are using the default storage
> > manager would still be able to use pg_upgrade, and those using
> > non-default storage managers perhaps can't.
>
> That would make sense.
>
> > But, again, this is all so hypothetical that it doesn't seem worth
> > talking about.
>
> Having a specific list of "these are the things we want to change, and
> why, and here is why pg_upgrade can't support it" would be much more
> useful to work from, I agree.
>
> That said, many discussions and ideas do get shut down, perhaps too
> early, because of pg_upgrade considerations. If we had a plan to have
> an incompatible release in the future, those ideas and discussions might
> be able to progress to a point where we determine it's worth it to take
> the pain of a non-pg_upgrade-supported release. That's a bit of a
> stretch, in my view, but I suppose it's possible. Even so though, I
> would suggest that we put together a wiki page to list out those items
> and encourage people to add to such a list; perhaps having an item on
> that list would make discussion about it progress beyond "it breaks
> pg_upgrade".

Yes, we should be collecting things we want to do for a pg_upgrade break
so we can see the list all in one place.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ It's impossible for everything to be true. +

On 05/26/2013 06:18 PM, Josh Berkus wrote:
>> Not sure which ones Simon meant, but at least any new/better
>> storage manager would seem to me to be requiring
>> a non-pg_upgrade upgrade path unless we require the storage manager
>> to also include a parallel implementation of pg_upgrade.
> Isn't this a bit of horse-cart inversion here? We just hashed out a
> tentative, incomplete pseudo-spec for storage managers *yesterday*.
Many people have been *thinking* about pluggable storage /
storage managers for much longer time.
> We
> don't have a complete spec at this point, let alone a development plan,
I think we will have a development plan *before* complete spec
anyway :)
> and it's entirely possible that we'll be able to implement SMs without
> breaking pgupgrade.
My point was exactly to not spend majority of new storage manager
discussion on "does it break pg_upgrade", "maybe we can find a way
to do it without breaking pg_upgrade", etc...
> It's also not at all clear that we can develop SMs in less than 2 years.
> I tend to think it unlikely.
I think the important part of Simons message was not "two years"
> First, let's have a few features for which breaking binary compatibility
> is a necessity or a clear benefit. Then we'll schedule when to break them.
But rather than "it breaks pg_upgrade" not being a complete stopper for
proposed useful features that might break it.

--
Hannu Krosing
PostgreSQL Consultant
Performance, Scalability and High Availability
2ndQuadrant Nordic OÜ

Bruce Momjian <bruce(at)momjian(dot)us> writes:
> On Mon, May 27, 2013 at 08:26:48AM -0400, Stephen Frost wrote:
>> That said, many discussions and ideas do get shut down, perhaps too
>> early, because of pg_upgrade considerations. If we had a plan to have
>> an incompatible release in the future, those ideas and discussions might
>> be able to progress to a point where we determine it's worth it to take
>> the pain of a non-pg_upgrade-supported release. That's a bit of a
>> stretch, in my view, but I suppose it's possible. Even so though, I
>> would suggest that we put together a wiki page to list out those items
>> and encourage people to add to such a list; perhaps having an item on
>> that list would make discussion about it progress beyond "it breaks
>> pg_upgrade".

> Yes, we should be collecting things we want to do for a pg_upgrade break
> so we can see the list all in one place.

Precisely. We've said right along that we reserve the right to have a
non-upgradable disk format change whenever sufficiently many reasons
accumulate to do that. The way to go about that is to collect projects
that need to be kept on hold for such a release --- not to say we're
going to have such a release and then look for reasons.

regards, tom lane

Michael Paquier escribió:
> On Mon, May 27, 2013 at 2:01 PM, Craig Ringer <craig(at)2ndquadrant(dot)com> wrote:
>
> > On 05/25/2013 05:39 PM, Simon Riggs wrote:
> > - Switching to single-major-version release numbering. The number of
> > people who say "PostgreSQL 9.x" is amazing; even *packagers* get this
> > wrong and produce "postgresql-9" packages. Witness Amazon Linux's awful
> > PostgreSQL packages for example. Going to PostgreSQL 10.0, 11.0, 12.0,
> > etc with a typical major/minor scheme might be worth considering.
> >
> In this case you don't even need the 2nd digit...

You do -- they are used for minor releases, i.e. 10.1 would be a bugfix
release for 10.0. If we continue using the current numbering scheme,
10.1 would be the major version after 10.0.

> Btw, -1 for the idea, as it would remove the possibility to tell that a new
> major release incrementing the 1st digit of version number brings more
> enhancement than normal major releases incrementing the 1st digit. This was
> the case for 9.0, helping people in remembering that streaming replication
> has been introduced from 9.x series.

All major releases bring lots of enhancements. Streaming replication
might be great for some people, but I'm sure there are things in 8.4 and
9.1 that are equally great for some other people.

--
Álvaro Herrera http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

Tom Lane wrote:
> Bruce Momjian <bruce(at)momjian(dot)us> writes:

> > Yes, we should be collecting things we want to do for a pg_upgrade break
> > so we can see the list all in one place.
>
> Precisely. We've said right along that we reserve the right to have a
> non-upgradable disk format change whenever sufficiently many reasons
> accumulate to do that.

Do we have a wiki page about this?

--
Álvaro Herrera http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

On Mon, May 27, 2013 at 09:17:50AM -0400, Bruce Momjian wrote:
> > That said, many discussions and ideas do get shut down, perhaps too
> > early, because of pg_upgrade considerations. If we had a plan to have
> > an incompatible release in the future, those ideas and discussions might
> > be able to progress to a point where we determine it's worth it to take
> > the pain of a non-pg_upgrade-supported release. That's a bit of a
> > stretch, in my view, but I suppose it's possible. Even so though, I
> > would suggest that we put together a wiki page to list out those items
> > and encourage people to add to such a list; perhaps having an item on
> > that list would make discussion about it progress beyond "it breaks
> > pg_upgrade".
>
> Yes, we should be collecting things we want to do for a pg_upgrade break
> so we can see the list all in one place.

OK, I have added a section to the TODO list for this:

Desired changes that would prevent upgrades with pg_upgrade

32-bit page checksums

Are there any others?

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ It's impossible for everything to be true. +

On 27 May 2013 15:36, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> Bruce Momjian <bruce(at)momjian(dot)us> writes:
>> On Mon, May 27, 2013 at 08:26:48AM -0400, Stephen Frost wrote:
>>> That said, many discussions and ideas do get shut down, perhaps too
>>> early, because of pg_upgrade considerations. If we had a plan to have
>>> an incompatible release in the future, those ideas and discussions might
>>> be able to progress to a point where we determine it's worth it to take
>>> the pain of a non-pg_upgrade-supported release. That's a bit of a
>>> stretch, in my view, but I suppose it's possible. Even so though, I
>>> would suggest that we put together a wiki page to list out those items
>>> and encourage people to add to such a list; perhaps having an item on
>>> that list would make discussion about it progress beyond "it breaks
>>> pg_upgrade".
>
>> Yes, we should be collecting things we want to do for a pg_upgrade break
>> so we can see the list all in one place.
>
> Precisely. We've said right along that we reserve the right to have a
> non-upgradable disk format change whenever sufficiently many reasons
> accumulate to do that.

I'm happy with that.

I was also thinking about collecting changes not related just to disk
format, if any exist.

> The way to go about that is to collect projects
> that need to be kept on hold for such a release --- not to say we're
> going to have such a release and then look for reasons.

Agreed.

I was trying to establish a realistic timeline for such events, so
that the planning was able to be taken seriously. Yes, it wass a "work
backwards" or "what if" type of planning. But now we have a rough plan
of how it might look, collecting ideas can begin.

--
Simon Riggs http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

Bruce Momjian <bruce(at)momjian(dot)us> writes:
> On Mon, May 27, 2013 at 09:17:50AM -0400, Bruce Momjian wrote:
>> Yes, we should be collecting things we want to do for a pg_upgrade break
>> so we can see the list all in one place.

> OK, I have added a section to the TODO list for this:

> Desired changes that would prevent upgrades with pg_upgrade
> 32-bit page checksums

> Are there any others?

GiST indexes really oughta have a metapage so there can be a version
number in them.

Also, if we are going to unify hstore and json, it'd be nice if we could
change the existing binary representation of hstore (per discussions at
Oleg and Teodor's talk --- this will be moot if we invent a new core
type, but it'd be better not to have to).

There are probably some other data-type-specific cleanups we could
make, but I have to go get on an airplane so no time to think about it.

regards, tom lane

Bruce Momjian wrote:

> OK, I have added a section to the TODO list for this:
>
> Desired changes that would prevent upgrades with pg_upgrade
>
> 32-bit page checksums
>
> Are there any others?

I would have each data segment be self-identifying, i.e. have a magic
number at the beginning of the file and the relation OID, some fork
identification and the segment number somewhere -- probably the special
space of the first page.

--
Álvaro Herrera http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

On Tue, May 28, 2013 at 12:36 AM, Alvaro Herrera
<alvherre(at)2ndquadrant(dot)com>wrote:

> Michael Paquier escribió:
> > On Mon, May 27, 2013 at 2:01 PM, Craig Ringer <craig(at)2ndquadrant(dot)com>
> wrote:
> >
> > > On 05/25/2013 05:39 PM, Simon Riggs wrote:
> > > - Switching to single-major-version release numbering. The number of
> > > people who say "PostgreSQL 9.x" is amazing; even *packagers* get this
> > > wrong and produce "postgresql-9" packages. Witness Amazon Linux's awful
> > > PostgreSQL packages for example. Going to PostgreSQL 10.0, 11.0, 12.0,
> > > etc with a typical major/minor scheme might be worth considering.
> > >
> > In this case you don't even need the 2nd digit...
>
> You do -- they are used for minor releases, i.e. 10.1 would be a bugfix
> release for 10.0. If we continue using the current numbering scheme,
> 10.1 would be the major version after 10.0.
>
Sorry for the confusion. I meant that the 2nd digit would not be necessary
when identifying a given major release, so I just didn't get the meaning of
what Craig said. As you say, you would still need the 2nd digit for minor
releases.
--
Michael

On Tue, May 28, 2013 at 07:39:35AM +0900, Michael Paquier wrote:
> On Tue, May 28, 2013 at 12:36 AM, Alvaro Herrera
> <alvherre(at)2ndquadrant(dot)com>wrote:
>
> > Michael Paquier escribió:
> > > On Mon, May 27, 2013 at 2:01 PM, Craig Ringer <craig(at)2ndquadrant(dot)com>
> > wrote:
> > >
> > > > On 05/25/2013 05:39 PM, Simon Riggs wrote:
> > > > - Switching to single-major-version release numbering. The number of
> > > > people who say "PostgreSQL 9.x" is amazing; even *packagers* get this
> > > > wrong and produce "postgresql-9" packages. Witness Amazon Linux's awful
> > > > PostgreSQL packages for example. Going to PostgreSQL 10.0, 11.0, 12.0,
> > > > etc with a typical major/minor scheme might be worth considering.
> > > >
> > > In this case you don't even need the 2nd digit...
> >
> > You do -- they are used for minor releases, i.e. 10.1 would be a bugfix
> > release for 10.0. If we continue using the current numbering scheme,
> > 10.1 would be the major version after 10.0.
> >
> Sorry for the confusion. I meant that the 2nd digit would not be necessary
> when identifying a given major release, so I just didn't get the meaning of
> what Craig said. As you say, you would still need the 2nd digit for minor
> releases.

What's been proposed before that wouldn't break previous applications
is a numbering system like this:

10.0.0
10.0.1
10.0.2
10.0.3
...
11.0.0
11.0.1

i.e. only change the "most-major" version number and always leave the
"less-major" number as zero.

Cheers,
David.
--
David Fetter <david(at)fetter(dot)org> http://fetter.org/
Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter
Skype: davidfetter XMPP: david(dot)fetter(at)gmail(dot)com
iCal: webcal://www.tripit.com/feed/ical/people/david74/tripit.ics

Remember to vote!
Consider donating to Postgres: http://www.postgresql.org/about/donate

Michael Paquier escribió:
> On Tue, May 28, 2013 at 12:36 AM, Alvaro Herrera
> <alvherre(at)2ndquadrant(dot)com>wrote:

> > You do -- they are used for minor releases, i.e. 10.1 would be a bugfix
> > release for 10.0. If we continue using the current numbering scheme,
> > 10.1 would be the major version after 10.0.
> >
> Sorry for the confusion. I meant that the 2nd digit would not be necessary
> when identifying a given major release, so I just didn't get the meaning of
> what Craig said. As you say, you would still need the 2nd digit for minor
> releases.

Well, that seems okay to me. We used to see a lot of people talking
about "Postgres 8.x" when they meant, say, 8.3; and now we have people
talking about "Postgres 9" when in reality they mean 9.1 or some other
specific major version. Having the second digit be part of the major
version number is a difficult idea to convey to external people.

--
Álvaro Herrera http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

On Tue, May 28, 2013 at 7:52 AM, David Fetter <david(at)fetter(dot)org> wrote:

> What's been proposed before that wouldn't break previous applications
> is a numbering system like this:
>
> 10.0.0
> 10.0.1
> 10.0.2
> 10.0.3
> ...
> 11.0.0
> 11.0.1
>
> i.e. only change the "most-major" version number and always leave the
> "less-major" number as zero.
>
Thanks for the clarification. Firefox did exactly the same from 4.0.
--
Michael

On 05/27/2013 05:45 PM, Michael Paquier wrote:
> On Mon, May 27, 2013 at 2:01 PM, Craig Ringer <craig(at)2ndquadrant(dot)com> wrote:
>
>> On 05/25/2013 05:39 PM, Simon Riggs wrote:
>> - Switching to single-major-version release numbering. The number of
>> people who say "PostgreSQL 9.x" is amazing; even *packagers* get this
>> wrong and produce "postgresql-9" packages. Witness Amazon Linux's awful
>> PostgreSQL packages for example. Going to PostgreSQL 10.0, 11.0, 12.0,
>> etc with a typical major/minor scheme might be worth considering.
>>
> In this case you don't even need the 2nd digit...
> Btw, -1 for the idea, as it would remove the possibility to tell that a new
> major release incrementing the 1st digit of version number brings more
> enhancement than normal major releases incrementing the 1st digit. This was
> the case for 9.0, helping people in remembering that streaming replication
> has been introduced from 9.x series.
I don't find bumping the major to be particularly helpful. Every
release brings major features - and some introduce major incompatibilities.

8.4 introduced CTEs.
8.3 broke tons of client code with the removal of implicit casts to text.

It really depends on what features you consider more major/significant.
Personally I don't think it makes sense to try to say "this release is
bigger" in Pg - at least not in terms of enhancement. I can see value in
using this-release-is-bigger for "this brings more breakage" - but would
strongly prefer a smooth and continuous release numbering that doesn't
confuse the heck out of users.

I'm extremely tired of being told "I'm running PostgreSQL 8.x" or "I'm
running PostgreSQL 9.x" and having to point out the version policy, the
fact that there are four years and huge fixes/enhancements between 8.0
and 8.4, etc.

The version policy makes _no distinction_ between which digit changes in
a major release:

"PostgreSQL major releases include new features and occur roughly once
every year. A major release is numbered by increasing either the first
or second part of the version number, e.g. 8.2 to 8.3.

"Major releases usually change the internal format of system tables and
data files. These changes are often complex, so we do not maintain
backward compatibility of all stored data. A dump/reload of the database
or use of the pg_upgrade module is required for major upgrades."

and I strongly believe that we should drop the notion entirely.

...

--
Craig Ringer http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

On 05/28/2013 07:22 AM, Michael Paquier wrote:
> On Tue, May 28, 2013 at 7:52 AM, David Fetter <david(at)fetter(dot)org> wrote:
>
>> What's been proposed before that wouldn't break previous applications
>> is a numbering system like this:
>>
>> 10.0.0
>> 10.0.1
>> 10.0.2
>> 10.0.3
>> ...
>> 11.0.0
>> 11.0.1
>>
>> i.e. only change the "most-major" version number and always leave the
>> "less-major" number as zero.
>>
> Thanks for the clarification. Firefox did exactly the same from 4.0.
Yeah... I was more meaning 10.0, 10.1, 10.2 etc for minor releases, but
I can imagine people coding logic to check "major version" using the
first two digits, so you're quite right that it'd need to be
grandfathered into 10.0.1, 10.0.2, etc. Sigh.

The upside of that is that it'd reinforce the idea that we sometimes
struggle to get across to people - that minor patch releases are *minor*
and *safe* to just upgrade to without jumping through change-approval
hoops, vendor approval for updates, two-year-long QA and all the other
baggage many IT departments seem to have.

--
Craig Ringer http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

On 05/28/2013 12:41 AM, Simon Riggs wrote:
> I'm happy with that.
>
> I was also thinking about collecting changes not related just to disk
> format, if any exist.
Any wire protocol or syntax changes?

I can't seem to find a "things we want to do in wire protocol v4" doc in
the wiki but I know I've seen occasional discussion of things that can't
be done without protocol changes. Anyone with a better memory than me
able to pitch in?

What'd be required to support in-band query cancellation? Sending
per-statement GUCs (to allow true statement timeout)?

I can't think of any major syntax warts and grandfathered quirks that'd
be really great to get rid of if we had the freedom to break things.

--
Craig Ringer http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

On 05/27/2013 04:58 PM, Craig Ringer wrote:
>
> On 05/28/2013 12:41 AM, Simon Riggs wrote:
>> I'm happy with that.
>>
>> I was also thinking about collecting changes not related just to disk
>> format, if any exist.
> Any wire protocol or syntax changes?
>
> I can't seem to find a "things we want to do in wire protocol v4" doc in
> the wiki but I know I've seen occasional discussion of things that can't
> be done without protocol changes. Anyone with a better memory than me
> able to pitch in?
>
> What'd be required to support in-band query cancellation? Sending
> per-statement GUCs (to allow true statement timeout)?
>

I would like to see the ability to define if a query is read only at the
protocol level, so that load balances that speak libpq can know what to
do with the query without parsing it.

JD

--
Command Prompt, Inc. - http://www.commandprompt.com/ 509-416-6579
PostgreSQL Support, Training, Professional Services and Development
High Availability, Oracle Conversion, Postgres-XC, @cmdpromptinc
For my dreams of your image that blossoms
a rose in the deeps of my heart. - W.B. Yeats

On 28/05/13 11:48, Craig Ringer wrote:
> On 05/27/2013 05:45 PM, Michael Paquier wrote:
>> On Mon, May 27, 2013 at 2:01 PM, Craig Ringer<craig(at)2ndquadrant(dot)com> wrote:
>>
>>> On 05/25/2013 05:39 PM, Simon Riggs wrote:
>>> - Switching to single-major-version release numbering. The number of
>>> people who say "PostgreSQL 9.x" is amazing; even *packagers* get this
>>> wrong and produce "postgresql-9" packages. Witness Amazon Linux's awful
>>> PostgreSQL packages for example. Going to PostgreSQL 10.0, 11.0, 12.0,
>>> etc with a typical major/minor scheme might be worth considering.
>>>
>> In this case you don't even need the 2nd digit...
>> Btw, -1 for the idea, as it would remove the possibility to tell that a new
>> major release incrementing the 1st digit of version number brings more
>> enhancement than normal major releases incrementing the 1st digit. This was
>> the case for 9.0, helping people in remembering that streaming replication
>> has been introduced from 9.x series.
> I don't find bumping the major to be particularly helpful. Every
> release brings major features - and some introduce major
> incompatibilities.
>
> 8.4 introduced CTEs.
> 8.3 broke tons of client code with the removal of implicit casts to text.
>
> It really depends on what features you consider more
> major/significant. Personally I don't think it makes sense to try to
> say "this release is bigger" in Pg - at least not in terms of
> enhancement. I can see value in using this-release-is-bigger for "this
> brings more breakage" - but would strongly prefer a smooth and
> continuous release numbering that doesn't confuse the heck out of users.
>
> I'm extremely tired of being told "I'm running PostgreSQL 8.x" or "I'm
> running PostgreSQL 9.x" and having to point out the version policy,
> the fact that there are four years and huge fixes/enhancements between
> 8.0 and 8.4, etc.
>
> The version policy makes _no distinction_ between which digit changes
> in a major release:
>
> "PostgreSQL major releases include new features and occur roughly once
> every year. A major release is numbered by increasing either the first
> or second part of the version number, e.g. 8.2 to 8.3.
>
> "Major releases usually change the internal format of system tables
> and data files. These changes are often complex, so we do not maintain
> backward compatibility of all stored data. A dump/reload of the
> database or use of the pg_upgrade module is required for major upgrades."
>
> and I strongly believe that we should drop the notion entirely.
>
> ...
>
> --
> Craig Ringerhttp://www.2ndQuadrant.com/
> PostgreSQL Development, 24x7 Support, Training & Services
Yes, I hate the Firefox style number inflation.

Cheers,
Gavin

On 05/28/2013 09:39 AM, Gavin Flower wrote:
> Yes, I hate the Firefox style number inflation.
I was arguing *for* it ;-)

I don't like it much either, but (a) we do about one release a year, not
one every few weeks and (b) it's very clear from a quick look at Stack
Overflow or first-posts to pgsql-general how confusing two-part major
versions are to users. If it's a bit less aesthetically pleasing I'm OK
with that.

--
Craig Ringer http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

On 05/27/2013 06:53 PM, Craig Ringer wrote:
>
> On 05/28/2013 09:39 AM, Gavin Flower wrote:
>> Yes, I hate the Firefox style number inflation.
> I was arguing *for* it ;-)
>
> I don't like it much either, but (a) we do about one release a year, not
> one every few weeks and (b) it's very clear from a quick look at Stack
> Overflow or first-posts to pgsql-general how confusing two-part major
> versions are to users. If it's a bit less aesthetically pleasing I'm OK
> with that.
>

This argument comes up every couple of years and the people that are
trying to solve the problem by changing the versioning are ignoring the
fact that there is no problem to solve.

Consider the following exchange:

Client: I have X problem with PostgreSQL
CMD: What version?
Client: 9
CMD: Which version of 9?
Client: 9.0.2
CMD: You should be running 9.2.4 or at least 9.0.13

Now, if we change the version numbers:

Client: I have X problem with PostgreSQL
CMD: What version?
Client: 9
CMD: Which version of 9?
Client: 9.0.2
CMD: You should be running 10.0.5 or at least 9.0.13

The conversation does not change.

Further, we are not Firefox. We are not user software. We are developer
software.

Sincerely,

Joshua D. Drake

On Sat, May 25, 2013 at 11:27 AM, Merlin Moncure <mmoncure(at)gmail(dot)com> wrote:
> On Sat, May 25, 2013 at 4:39 AM, Simon Riggs <simon(at)2ndquadrant(dot)com> wrote:
>> There are a number of changes we'd probably like to make to the way
>> things work in Postgres. This thread is not about discussing what
>> those are, just to say that requirements exist and have been discussed
>> in various threads over time.
>>
>> The constraint on such changes is that we've decided that we must have
>> an upgrade path from release to release.
>>
>> So I'd like to make a formal suggestion of a plan for how we cope with this:
>>
>> 1. Implement online upgrade in 9.4 via the various facilities we have
>> in-progress. That looks completely possible.
>>
>> 2. Name the next release after that 10.0 (would have been 9.5). We
>> declare now that
>> a) 10.0 will support on-line upgrade from 9.4 (only)
>> b) various major incompatibilities will be introduced in 10.0 - the
>> change in release number will indicate to everybody that is the case
>> c) agree that there will be no pg_upgrade patch from 9.4 to 10.0, so
>> that we will not be constrained by that
>>
>> This plan doesn't presume any particular change. Each change would
>> need to be discussed on a separate thread, with a separate case for
>> each. All I'm suggesting is that we have a coherent plan for the
>> timing of such changes, so we can bundle them together into one
>> release.
>>
>> By doing this now we give ourselves lots of time to plan changes that
>> will see us good for another decade. If we don't do this, then we
>> simply risk losing the iniative by continuing to support legacy
>> formats and approaches.
>
> Huh. I don't think that bumping the version number to 10.0 vs 9.5 is
> justification to introduce breaking changes. In fact, I would rather
> see 10.0 be the version where we formally stop doing that. I
> understand that some stuff needs to be improved but it often doesn't
> seem to be worth the cost in the long run.

Please disregard this comment -- I didn't realize the topic was
regarding on disk format -- I mistakenly though it was opening the
door for user level feature changes.

merlin

On 05/28/2013 06:13 AM, Joshua D. Drake wrote:
>
> On 05/27/2013 06:53 PM, Craig Ringer wrote:
>>
>> On 05/28/2013 09:39 AM, Gavin Flower wrote:
>>> Yes, I hate the Firefox style number inflation.
>> I was arguing *for* it ;-)
>>
>> I don't like it much either, but (a) we do about one release a year, not
>> one every few weeks and (b) it's very clear from a quick look at Stack
>> Overflow or first-posts to pgsql-general how confusing two-part major
>> versions are to users. If it's a bit less aesthetically pleasing I'm OK
>> with that.
>>
>
> This argument comes up every couple of years and the people that
> are trying to solve the problem by changing the versioning are
> ignoring the fact that there is no problem to solve.
>
> Consider the following exchange:
>
> Client: I have X problem with PostgreSQL
> CMD: What version?
> Client: 9
> CMD: Which version of 9?
> Client: 9.0.2
> CMD: You should be running 9.2.4 or at least 9.0.13
If the problem has the "at least" part, then the first part is superfluous.

If somebody wants to figure out how to run streaming CTE-s on
"postgresql 8" then you need to ask
for exact "major version which is two first digits" if they want to run
streaming replication there you
can skip on e-mail exchange and tell right away that SR was added in
version 9.0

> ...
>
> The conversation does not change.
>
> Further, we are not Firefox. We are not user software. We are
> developer software.
At least some of the real-world problems with PostgreSQL
comes from "We are developer software" mentality.

Yes, We are developer software, but we are also a
DBA/maintainer/infrastructure manager
software which needs to live a long time after the "development" is
finished.

--
Hannu Krosing
PostgreSQL Consultant
Performance, Scalability and High Availability
2ndQuadrant Nordic OÜ

>> This argument comes up every couple of years and the people that
>> are trying to solve the problem by changing the versioning are
>> ignoring the fact that there is no problem to solve.

We just had this discussion on -advocacy (where it belongs, frankly) a
couple months ago:

http://www.postgresql.org/message-id/512E8EF8.3000507@agliodbs.com

To sum up: the negatives of changing our version numbering scheme
outweighed the positives.

--
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

On Tue, May 28, 2013 at 11:56 AM, Josh Berkus <josh(at)agliodbs(dot)com> wrote:
>
>>> This argument comes up every couple of years and the people that
>>> are trying to solve the problem by changing the versioning are
>>> ignoring the fact that there is no problem to solve.
>
> We just had this discussion on -advocacy (where it belongs, frankly)

+1.

> a
> couple months ago:
>
> http://www.postgresql.org/message-id/512E8EF8.3000507@agliodbs.com
>
> To sum up: the negatives of changing our version numbering scheme
> outweighed the positives.

And +1 to that, too.

FWIW, I think we may want to consider retitling 9.4 as 10.0, not
because of any binary compatibility break (which, for the record, I
oppose) but because of features. It's a little early to make that
call just yet, of course, but I have a good feeling about this cycle.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On 05/28/2013 08:36 AM, Hannu Krosing wrote:

>> The conversation does not change.
>>
>> Further, we are not Firefox. We are not user software. We are
>> developer software.
> At least some of the real-world problems with PostgreSQL
> comes from "We are developer software" mentality.
>
> Yes, We are developer software, but we are also a
> DBA/maintainer/infrastructure manager

I would not hire any of those three that weren't smart enough to
understand our versioning scheme or had the wits to open a web browser
and google:

PostgreSQL versioning

The answer is link #1 on Google.

That said, I won't raise a stink. I am not really of a strong opinion
either way except to say we are not solving a problem. We are just
tickling each other's fancies.

JD

--
Command Prompt, Inc. - http://www.commandprompt.com/ 509-416-6579
PostgreSQL Support, Training, Professional Services and Development
High Availability, Oracle Conversion, Postgres-XC, @cmdpromptinc
For my dreams of your image that blossoms
a rose in the deeps of my heart. - W.B. Yeats

On Tue, May 28, 2013 at 07:58:33AM +0800, Craig Ringer wrote:
> On 05/28/2013 12:41 AM, Simon Riggs wrote:
> > I'm happy with that.
> >
> > I was also thinking about collecting changes not related just to disk
> > format, if any exist.
> Any wire protocol or syntax changes?
>
> I can't seem to find a "things we want to do in wire protocol v4" doc in
> the wiki but I know I've seen occasional discussion of things that can't
> be done without protocol changes. Anyone with a better memory than me
> able to pitch in?

Sure, it is on the TODO list:

https://wiki.postgresql.org/wiki/Todo#.2Fcontrib.2Fpg_upgrade

I can only get a link to pg_upgrade from there, so look two sections
below that for "Wire Protocol Changes".

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ It's impossible for everything to be true. +

On Mon, May 27, 2013 at 05:21:16PM -0700, Joshua D. Drake wrote:
>
> On 05/27/2013 04:58 PM, Craig Ringer wrote:
> >
> >On 05/28/2013 12:41 AM, Simon Riggs wrote:
> >>I'm happy with that.
> >>
> >>I was also thinking about collecting changes not related just to disk
> >>format, if any exist.
> >Any wire protocol or syntax changes?
> >
> >I can't seem to find a "things we want to do in wire protocol v4" doc in
> >the wiki but I know I've seen occasional discussion of things that can't
> >be done without protocol changes. Anyone with a better memory than me
> >able to pitch in?
> >
> >What'd be required to support in-band query cancellation? Sending
> >per-statement GUCs (to allow true statement timeout)?
> >
>
> I would like to see the ability to define if a query is read only at
> the protocol level, so that load balances that speak libpq can know
> what to do with the query without parsing it.

Sounds nice, but how would we do that? That would require libpq to know
it, right? Do we pass anything back after parsing but before execution?
Could it be optional? What about functions that modify the database
--- isn't that only known at execution time?

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ It's impossible for everything to be true. +

On Mon, May 27, 2013 at 02:09:05PM -0400, Tom Lane wrote:
> Bruce Momjian <bruce(at)momjian(dot)us> writes:
> > On Mon, May 27, 2013 at 09:17:50AM -0400, Bruce Momjian wrote:
> >> Yes, we should be collecting things we want to do for a pg_upgrade break
> >> so we can see the list all in one place.
>
> > OK, I have added a section to the TODO list for this:
>
> > Desired changes that would prevent upgrades with pg_upgrade
> > 32-bit page checksums
>
> > Are there any others?
>
> GiST indexes really oughta have a metapage so there can be a version
> number in them.
>
> Also, if we are going to unify hstore and json, it'd be nice if we could
> change the existing binary representation of hstore (per discussions at
> Oleg and Teodor's talk --- this will be moot if we invent a new core
> type, but it'd be better not to have to).
>
> There are probably some other data-type-specific cleanups we could
> make, but I have to go get on an airplane so no time to think about it.

OK, GiST and hstore added to TODO list.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ It's impossible for everything to be true. +

On 05/28/2013 02:18 PM, Bruce Momjian wrote:

>> I would like to see the ability to define if a query is read only at
>> the protocol level, so that load balances that speak libpq can know
>> what to do with the query without parsing it.
>
> Sounds nice, but how would we do that? That would require libpq to know
> it, right? Do we pass anything back after parsing but before execution?
> Could it be optional? What about functions that modify the database
> --- isn't that only known at execution time?

I can't speak to the actual C code that would be required but from a
user space, I could see something like this:

con = psycopg2.connect(database='testdb', user='test',
transaction-type='r')

Thus when the connection is made, before anything else is done, we know
it is a read only connection and therefore any load balancer speaking
libpq would also know it is a read only. The default of course would be
r/w and you would use a different connection handler for r/w or w queries.

The other option would be to do it on query execute but that doesn't
seem as efficient as it would have to be parsed each time. Although it
would still be better than reading the actual SQL.

Sincerely,

Joshua D. Drake

--
Command Prompt, Inc. - http://www.commandprompt.com/ 509-416-6579
PostgreSQL Support, Training, Professional Services and Development
High Availability, Oracle Conversion, Postgres-XC, @cmdpromptinc
For my dreams of your image that blossoms
a rose in the deeps of my heart. - W.B. Yeats

Bruce Momjian wrote:
> On Mon, May 27, 2013 at 05:21:16PM -0700, Joshua D. Drake wrote:

> > I would like to see the ability to define if a query is read only at
> > the protocol level, so that load balances that speak libpq can know
> > what to do with the query without parsing it.
>
> Sounds nice, but how would we do that? That would require libpq to know
> it, right? Do we pass anything back after parsing but before execution?
> Could it be optional? What about functions that modify the database
> --- isn't that only known at execution time?

Well, if you hit anything that tries to acquire an Xid, and you're in a
context that said only read-only was acceptable, just raise an error.

In a similar vein, I vaguely recall we discussed (after some security
vulnerability involving SQL injection) a mode where we only accept only
one command per PQexec() call, i.e. reject execution of commands that
contain multiple queries.

--
Álvaro Herrera http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

On Tue, May 28, 2013 at 02:26:06PM -0700, Joshua D. Drake wrote:
> >Sounds nice, but how would we do that? That would require libpq to know
> >it, right? Do we pass anything back after parsing but before execution?
> > Could it be optional? What about functions that modify the database
> >--- isn't that only known at execution time?
>
> I can't speak to the actual C code that would be required but from a
> user space, I could see something like this:
>
> con = psycopg2.connect(database='testdb', user='test',
> transaction-type='r')
>
> Thus when the connection is made, before anything else is done, we
> know it is a read only connection and therefore any load balancer
> speaking libpq would also know it is a read only. The default of
> course would be r/w and you would use a different connection handler
> for r/w or w queries.
>
> The other option would be to do it on query execute but that doesn't
> seem as efficient as it would have to be parsed each time. Although
> it would still be better than reading the actual SQL.

Well, you could do SET TRANSACTION READ ONLY, and that would prevent any
write transactions. You could assume it is a read query, and get the
error and resubmit on the master if that happens, but that sounds
inefficient. I thought you were asking for something where you could
submit a query and it would report back as read/write or read-only.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ It's impossible for everything to be true. +

On Mon, May 27, 2013 at 03:06:13PM -0400, Alvaro Herrera wrote:
> Bruce Momjian wrote:
>
> > OK, I have added a section to the TODO list for this:
> >
> > Desired changes that would prevent upgrades with pg_upgrade
> >
> > 32-bit page checksums
> >
> > Are there any others?
>
> I would have each data segment be self-identifying, i.e. have a magic
> number at the beginning of the file and the relation OID, some fork
> identification and the segment number somewhere -- probably the special
> space of the first page.

Is this something we want on the TODO? I was not clear how to do with
without making the first page format special or wasting space on all the
other pages.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ It's impossible for everything to be true. +

On 05/28/2013 03:36 PM, Bruce Momjian wrote:

>> The other option would be to do it on query execute but that doesn't
>> seem as efficient as it would have to be parsed each time. Although
>> it would still be better than reading the actual SQL.
>
> Well, you could do SET TRANSACTION READ ONLY, and that would prevent any
> write transactions. You could assume it is a read query, and get the
> error and resubmit on the master if that happens, but that sounds
> inefficient. I thought you were asking for something where you could
> submit a query and it would report back as read/write or read-only.

No I am suggesting something that before anything happens with the
parser, the protocol knows what is up. So things like pgpool-ii don't
even need a parser, it just knows it is a read only query because the
protocol says so.

JD

--
Command Prompt, Inc. - http://www.commandprompt.com/ 509-416-6579
PostgreSQL Support, Training, Professional Services and Development
High Availability, Oracle Conversion, Postgres-XC, @cmdpromptinc
For my dreams of your image that blossoms
a rose in the deeps of my heart. - W.B. Yeats

On Tue, May 28, 2013 at 03:39:10PM -0700, Joshua D. Drake wrote:
>
> On 05/28/2013 03:36 PM, Bruce Momjian wrote:
>
> >>The other option would be to do it on query execute but that doesn't
> >>seem as efficient as it would have to be parsed each time. Although
> >>it would still be better than reading the actual SQL.
> >
> >Well, you could do SET TRANSACTION READ ONLY, and that would prevent any
> >write transactions. You could assume it is a read query, and get the
> >error and resubmit on the master if that happens, but that sounds
> >inefficient. I thought you were asking for something where you could
> >submit a query and it would report back as read/write or read-only.
>
> No I am suggesting something that before anything happens with the
> parser, the protocol knows what is up. So things like pgpool-ii
> don't even need a parser, it just knows it is a read only query
> because the protocol says so.

Oh, that is an interesting idea. The application is indicating it is
read-only via the protocol, and poolers can optimize that. Don't we
have the ability to pass arbitrary GUC values back through the protocol,
e.g. transaction_read_only? If not, that might be a way to do this
cleanly.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ It's impossible for everything to be true. +

On 05/28/2013 04:05 PM, Bruce Momjian wrote:
>
> On Tue, May 28, 2013 at 03:39:10PM -0700, Joshua D. Drake wrote:
>>
>> On 05/28/2013 03:36 PM, Bruce Momjian wrote:
>>
>>>> The other option would be to do it on query execute but that doesn't
>>>> seem as efficient as it would have to be parsed each time. Although
>>>> it would still be better than reading the actual SQL.
>>>
>>> Well, you could do SET TRANSACTION READ ONLY, and that would prevent any
>>> write transactions. You could assume it is a read query, and get the
>>> error and resubmit on the master if that happens, but that sounds
>>> inefficient. I thought you were asking for something where you could
>>> submit a query and it would report back as read/write or read-only.
>>
>> No I am suggesting something that before anything happens with the
>> parser, the protocol knows what is up. So things like pgpool-ii
>> don't even need a parser, it just knows it is a read only query
>> because the protocol says so.
>
> Oh, that is an interesting idea. The application is indicating it is
> read-only via the protocol, and poolers can optimize that. Don't we
> have the ability to pass arbitrary GUC values back through the protocol,
> e.g. transaction_read_only? If not, that might be a way to do this
> cleanly.
>

I don't know but I don't think so. Anything that is calling SET is going
to run through the parser.

JD

--
Command Prompt, Inc. - http://www.commandprompt.com/ 509-416-6579
PostgreSQL Support, Training, Professional Services and Development
High Availability, Oracle Conversion, Postgres-XC, @cmdpromptinc
For my dreams of your image that blossoms
a rose in the deeps of my heart. - W.B. Yeats

On 05/29/2013 05:11 AM, Bruce Momjian wrote:

> Sure, it is on the TODO list:
>
> https://wiki.postgresql.org/wiki/Todo#.2Fcontrib.2Fpg_upgrade
>
> I can only get a link to pg_upgrade from there, so look two sections
> below that for "Wire Protocol Changes".
Thanks.

The direct link is
https://wiki.postgresql.org/wiki/Todo#Wire_Protocol_Changes for anyone
looking for it later.

--
Craig Ringer http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

On Tue, May 28, 2013 at 4:26 PM, Joshua D. Drake <jd(at)commandprompt(dot)com> wrote:
>
> On 05/28/2013 02:18 PM, Bruce Momjian wrote:
>
>>> I would like to see the ability to define if a query is read only at
>>> the protocol level, so that load balances that speak libpq can know
>>> what to do with the query without parsing it.
>>
>>
>> Sounds nice, but how would we do that? That would require libpq to know
>> it, right? Do we pass anything back after parsing but before execution?
>> Could it be optional? What about functions that modify the database
>> --- isn't that only known at execution time?
>
>
> I can't speak to the actual C code that would be required but from a user
> space, I could see something like this:
>
> con = psycopg2.connect(database='testdb', user='test', transaction-type='r')
>
> Thus when the connection is made, before anything else is done, we know it
> is a read only connection and therefore any load balancer speaking libpq
> would also know it is a read only. The default of course would be r/w and
> you would use a different connection handler for r/w or w queries.
>

you can do that today already, kind-of

create an entry in pgbouncer that connect to
host=read-only.servers.dns and make read-only.servers.dns to point to
more than 1 ip.
then when the application wants to do load balancing, just connect to
the entry that points to read-only.servers.dns and let the magic
happens

which would be great is this to happen transparently to the application

> The other option would be to do it on query execute but that doesn't seem as
> efficient as it would have to be parsed each time. Although it would still
> be better than reading the actual SQL.
>

another idea, as someone else mentioned, and i think has been
discussed bedore is a function that says if the query is r-o or not...
maybe even exporting the plan so we don't need to replan again...

Not sure if that is possible, just hand waving...

--
Jaime Casanova www.2ndQuadrant.com
Professional PostgreSQL: Soporte 24x7 y capacitación
Phone: +593 4 5107566 Cell: +593 987171157

> On 05/28/2013 04:05 PM, Bruce Momjian wrote:
>>
>> On Tue, May 28, 2013 at 03:39:10PM -0700, Joshua D. Drake wrote:
>>>
>>> On 05/28/2013 03:36 PM, Bruce Momjian wrote:
>>>
>>>>> The other option would be to do it on query execute but that doesn't
>>>>> seem as efficient as it would have to be parsed each time. Although
>>>>> it would still be better than reading the actual SQL.
>>>>
>>>> Well, you could do SET TRANSACTION READ ONLY, and that would prevent
>>>> any
>>>> write transactions. You could assume it is a read query, and get the
>>>> error and resubmit on the master if that happens, but that sounds
>>>> inefficient. I thought you were asking for something where you could
>>>> submit a query and it would report back as read/write or read-only.
>>>
>>> No I am suggesting something that before anything happens with the
>>> parser, the protocol knows what is up. So things like pgpool-ii
>>> don't even need a parser, it just knows it is a read only query
>>> because the protocol says so.
>>
>> Oh, that is an interesting idea. The application is indicating it is
>> read-only via the protocol, and poolers can optimize that. Don't we
>> have the ability to pass arbitrary GUC values back through the
>> protocol,
>> e.g. transaction_read_only? If not, that might be a way to do this
>> cleanly.
>>
>
> I don't know but I don't think so. Anything that is calling SET is
> going to run through the parser.

Right. SET command needs to be parsed by the parser. However, we
already have embedded parameters in the start up packet, which needs
to be recognized by pooler anyway. See "StartupMessage" section in:
http://www.postgresql.org/docs/9.3/static/protocol-message-formats.html

I am not sure backend currently permits to have
default_transaction_read_only = on in the startup packet or not
though.
--
Tatsuo Ishii
SRA OSS, Inc. Japan
English: http://www.sraoss.co.jp/index_en.php
Japanese: http://www.sraoss.co.jp

Bruce Momjian wrote:
> On Mon, May 27, 2013 at 03:06:13PM -0400, Alvaro Herrera wrote:
> > Bruce Momjian wrote:

> > I would have each data segment be self-identifying, i.e. have a magic
> > number at the beginning of the file and the relation OID, some fork
> > identification and the segment number somewhere -- probably the special
> > space of the first page.
>
> Is this something we want on the TODO? I was not clear how to do with
> without making the first page format special or wasting space on all the
> other pages.

I don't think the special space has to necessarily be identically sized
in all the pages -- I admit I haven't looked closely, but the special
size is part of the page header.

--
Álvaro Herrera http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

On Mon, May 27, 2013 at 9:41 AM, Simon Riggs <simon(at)2ndquadrant(dot)com> wrote:
> On 27 May 2013 15:36, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
>> Bruce Momjian <bruce(at)momjian(dot)us> writes:
>>> On Mon, May 27, 2013 at 08:26:48AM -0400, Stephen Frost wrote:
>>>> That said, many discussions and ideas do get shut down, perhaps too
>>>> early, because of pg_upgrade considerations. If we had a plan to have
>>>> an incompatible release in the future, those ideas and discussions might
>>>> be able to progress to a point where we determine it's worth it to take
>>>> the pain of a non-pg_upgrade-supported release. That's a bit of a
>>>> stretch, in my view, but I suppose it's possible. Even so though, I
>>>> would suggest that we put together a wiki page to list out those items
>>>> and encourage people to add to such a list; perhaps having an item on
>>>> that list would make discussion about it progress beyond "it breaks
>>>> pg_upgrade".
>>
>>> Yes, we should be collecting things we want to do for a pg_upgrade break
>>> so we can see the list all in one place.
>>
>> Precisely. We've said right along that we reserve the right to have a
>> non-upgradable disk format change whenever sufficiently many reasons
>> accumulate to do that.

Here's one that's come up a few times: being able to tweak the
out-of-line storage strategy, e.g. change the compression format used.
I think some folks were lamenting the lack of a convenient byte in
the right place for that one.