Re: Two-phase commit for 8.1

Hi,

Now that we got 8.0 out of the door, I'm submitting my two-phase commit patch
again for discussion.

http://www.hut.fi/~hlinnaka/pgsql/

Do we want it in 8.1, if we want a short development cycle? It needs a new
pg_twophase subdirectory, and it introduces a new system view, so I guess it
requires an initdb (or pg_upgrade).

Any comments on the implementation or the new commands?

I would appreciate help testing the JDBC driver with an application server that
does XA recovery properly.

- Heikki

If the patch is ready to be committed early in the cycle, I'd say most
definitely ... just depends on how late in the cycle its ready ...

I *believe* that 8.1, we're looking at a 2mo cycle before beta, so figure
beta for ~April 1st (no april fools jokes, eh?) ...

On Wed, 19 Jan 2005, Heikki Linnakangas wrote:

> Hi,
>
> Now that we got 8.0 out of the door, I'm submitting my two-phase commit patch
> again for discussion.
>
> http://www.hut.fi/~hlinnaka/pgsql/
>
> Do we want it in 8.1, if we want a short development cycle? It needs a new
> pg_twophase subdirectory, and it introduces a new system view, so I guess it
> requires an initdb (or pg_upgrade).
>
> Any comments on the implementation or the new commands?
>
> I would appreciate help testing the JDBC driver with an application server
> that does XA recovery properly.
>
> - Heikki
>
> ---------------------------(end of broadcast)---------------------------
> TIP 2: you can get off all lists at once with the unregister command
> (send "unregister YourEmailAddressHere" to majordomo(at)postgresql(dot)org)
>

----
Marc G. Fournier Hub.Org Networking Services (http://www.hub.org)
Email: scrappy(at)hub(dot)org Yahoo!: yscrappy ICQ: 7615664

"Marc G. Fournier" <scrappy(at)postgresql(dot)org> writes:
> If the patch is ready to be committed early in the cycle, I'd say most
> definitely ... just depends on how late in the cycle its ready ...

My recollection is that it's quite far from being complete. I had hoped
to spend some time during the 8.1 cycle helping Heikki finish it up,
but if we stick to the 2-month-dev-cycle idea I'm afraid there's no way
it'll be done in time. I thought that "some time" would probably amount
to a solid man-month or so, and there's no way I can spend half my time
on just one feature for this cycle.

If Heikki wants this in for 8.1, the right thing to do is vote against
the short-dev-cycle idea. But we need a plausible answer about what to
do about ARC to make that credible...

regards, tom lane

> If the patch is ready to be committed early in the cycle, I'd say most
> definitely ... just depends on how late in the cycle its ready ...
>
> I *believe* that 8.1, we're looking at a 2mo cycle before beta, so
> figure beta for ~April 1st (no april fools jokes, eh?) ...

You guys are crazy :) We haven't had a release cycle less than a year
in many, many releases :)

Chris

On Thursday 20 January 2005 04:16, Christopher Kings-Lynne wrote:
> > If the patch is ready to be committed early in the cycle, I'd say most
> > definitely ... just depends on how late in the cycle its ready ...
> >
> > I *believe* that 8.1, we're looking at a 2mo cycle before beta, so
> > figure beta for ~April 1st (no april fools jokes, eh?) ...
>
> You guys are crazy :) We haven't had a release cycle less than a year
> in many, many releases :)
>

If ARC is deemed a serious enough problem that we need to address it *now*,
then I think we should do a 2 month cycle where core focus will be on putting
in an ARC replacement and allowing only changes that do not require an
initdb. If we stick to that we can do a new release in probably 3-4 months
and I think that will be acceptable as long as dump/reload is not required
for upgrade. I still think it might be worth contacting IBM and asking them
if thier intention is to enforce the ARC patent if it is granted before
pushing forward with that plan, but it seems like a good course of action to
follow just in case. If others felt strongly, we could probably start an 8.2
branch right now as well and put someone like Neil in charge of keeping 8.2
up to date with 8.1 changes while proceeding forward with other new whizbang
functionality that requires initdb (I pick Neil cause iirc he has some initdb
requiring changes planned for development, but it could be someone else).

--
Robert Treat
Build A Brighter Lamp :: Linux Apache {middleware} PostgreSQL

On Wed, Jan 19, 2005 at 07:42:03PM -0500, Tom Lane wrote:
> "Marc G. Fournier" <scrappy(at)postgresql(dot)org> writes:
> > If the patch is ready to be committed early in the cycle, I'd say most
> > definitely ... just depends on how late in the cycle its ready ...
>
> My recollection is that it's quite far from being complete. I had hoped
> to spend some time during the 8.1 cycle helping Heikki finish it up,
> but if we stick to the 2-month-dev-cycle idea I'm afraid there's no way
> it'll be done in time. I thought that "some time" would probably amount
> to a solid man-month or so, and there's no way I can spend half my time
> on just one feature for this cycle.
>
> If Heikki wants this in for 8.1, the right thing to do is vote against
> the short-dev-cycle idea. But we need a plausible answer about what to
> do about ARC to make that credible...
>

I think the idea of making a buffer management algorithm API and then
preparing a simple LRU algorithm to have ready to plug in if the ARC
patent is granted would be doable in a short development cycle. Then
we can take advantage of as well as test other algorithms more easily.

Ken

On Wed, 19 Jan 2005, Tom Lane wrote:

> "Marc G. Fournier" <scrappy(at)postgresql(dot)org> writes:
>> If the patch is ready to be committed early in the cycle, I'd say most
>> definitely ... just depends on how late in the cycle its ready ...
>
> My recollection is that it's quite far from being complete. I had hoped
> to spend some time during the 8.1 cycle helping Heikki finish it up,
> but if we stick to the 2-month-dev-cycle idea I'm afraid there's no way
> it'll be done in time. I thought that "some time" would probably amount
> to a solid man-month or so, and there's no way I can spend half my time
> on just one feature for this cycle.
>
> If Heikki wants this in for 8.1, the right thing to do is vote against
> the short-dev-cycle idea. But we need a plausible answer about what to
> do about ARC to make that credible...

I'm not sure what I want.

If the 8.1 cycle really is a short one, say 3 months, then I have no
problem waiting for 8.2. But we have a very bad track record regarding
short-dev-cycles. I honestly don't believe we can get 8.1 released before
July.

- Heikki

Heikki,

What is still missing to complete the 2PC patch?.

Regards,

Hans

Heikki Linnakangas wrote:
> On Wed, 19 Jan 2005, Tom Lane wrote:
>
>> "Marc G. Fournier" <scrappy(at)postgresql(dot)org> writes:
>>
>>> If the patch is ready to be committed early in the cycle, I'd say most
>>> definitely ... just depends on how late in the cycle its ready ...
>>
>>
>> My recollection is that it's quite far from being complete. I had hoped
>> to spend some time during the 8.1 cycle helping Heikki finish it up,
>> but if we stick to the 2-month-dev-cycle idea I'm afraid there's no way
>> it'll be done in time. I thought that "some time" would probably amount
>> to a solid man-month or so, and there's no way I can spend half my time
>> on just one feature for this cycle.
>>
>> If Heikki wants this in for 8.1, the right thing to do is vote against
>> the short-dev-cycle idea. But we need a plausible answer about what to
>> do about ARC to make that credible...
>
>
> I'm not sure what I want.
>
> If the 8.1 cycle really is a short one, say 3 months, then I have no
> problem waiting for 8.2. But we have a very bad track record regarding
> short-dev-cycles. I honestly don't believe we can get 8.1 released
> before July.
>
> - Heikki
>
> ---------------------------(end of broadcast)---------------------------
> TIP 8: explain analyze is your friend

--
Cybertec Geschwinde u Schoenig
Schoengrabern 134, A-2020 Hollabrunn, Austria
Tel: +43/660/816 40 77
www.cybertec.at, www.postgresql.at

On Sun, 23 Jan 2005, Hans-Jürgen Schönig wrote:

> Heikki,
>
> What is still missing to complete the 2PC patch?.

Here's my TODO on things that need to be done:

* large objects
* guc variables
* notify/listen

Large objects and notify/listen should be quite straightforward. GUC
variables need some thinking, but shouldn't be much work.

As the patch gets more attention, I'm sure more issues will come up.

- Heikki

On Sun, Jan 23, 2005 at 01:37:30PM +0200, Heikki Linnakangas wrote:

> As the patch gets more attention, I'm sure more issues will come up.

I see the changes to the lock manager are huge. Can you explain what's
the idea behind those? Do you release the locks and then reacquire
them, or do you reassign them to a pseudo process? Are there
possibilities of deadlock somewhere?

--
Alvaro Herrera (<alvherre[(at)]dcc(dot)uchile(dot)cl>)
Thou shalt study thy libraries and strive not to reinvent them without
cause, that thy code may be short and readable and thy days pleasant
and productive. (7th Commandment for C Programmers)

On Sun, 23 Jan 2005, Alvaro Herrera wrote:

> On Sun, Jan 23, 2005 at 01:37:30PM +0200, Heikki Linnakangas wrote:
>
>> As the patch gets more attention, I'm sure more issues will come up.

> I see the changes to the lock manager are huge. Can you explain what's
> the idea behind those? Do you release the locks and then reacquire
> them, or do you reassign them to a pseudo process?

I reassign them to a pseudo process (persistedLocksProc).

Much of the changes in lock.c are just about moving code around.
Some copy-paste code has been put in functions. LockAcquire has been
changed to take PGPROC as an argument, so that locks can be acquired on
behalf of the pseudo process.

Then there's completely new code for persisting locks on PREPARE
TRANSACTION and reacquiring them on recovery.

If it helps, I could try to split it into two patches, one with code
rearrangements that don't change current behaviour, and then the actual
2PC stuff on top of that.

> Are there possibilities of deadlock somewhere?

Not that I know of.

- Heikki

On Sun, Jan 23, 2005 at 07:32:55PM +0200, Heikki Linnakangas wrote:

> If it helps, I could try to split it into two patches, one with code
> rearrangements that don't change current behaviour, and then the actual
> 2PC stuff on top of that.

I think that'd be a good idea, because such a patch could be merged
right now, and the actual 2PC stuff would be smaller and easier to
review.

You'd only need a committer to actually commit the initial patch ...

--
Alvaro Herrera (<alvherre[(at)]dcc(dot)uchile(dot)cl>)
"No hay cielo posible sin hundir nuestras raíces
en la profundidad de la tierra" (Malucha Pinto)