Re: [Fwd: Re: [PATCHES] 64-bit CommandIds]

Hi,

what's your opinion on this?
I saw response only from Alvaro on the -patches list.

Thanks in advance,
Zoltán Böszörményi

-------- Eredeti üzenet --------
Tárgy: Re: [PATCHES] 64-bit CommandIds
Dátum: Tue, 04 Mar 2008 21:52:25 +0100
Feladó: Zoltan Boszormenyi <zb(at)cybertec(dot)at>
Címzett: pgsql-patches <pgsql-patches(at)postgresql(dot)org>
CC: Alvaro Herrera <alvherre(at)commandprompt(dot)com>, Hans-Juergen Schoenig
<hs(at)cybertec(dot)at>
Hivatkozások: <47CD8665(dot)7070903(at)cybertec(dot)at>
<20080304174110(dot)GK4755(at)alvh(dot)no-ip(dot)org>

Alvaro Herrera írta:
> Zoltan Boszormenyi wrote:
>
>
>> attached is our patch against HEAD which enables extending CommandIds
>> to 64-bit. This is for enabling long transactions that really do that much
>> non-read-only work in one transaction.
>>
>
> I think you should add a pg_control field and corresponding check, to
> avoid a 64bit-Cid postmaster to start on a 32bit-Cid data area and vice
> versa.
>

I added the check but I needed to add it BEFORE checking for
toast_max_chunk_size otherwise it complained about this more
cryptic problem. I think it's cleaner to report this failure to know
why toast_max_chunk_size != TOAST_MAX_CHUNK_SIZE.

Best regards,
Zoltán Böszörményi

--
----------------------------------
Zoltán Böszörményi
Cybertec Schönig & Schönig GmbH
http://www.postgresql.at/

--
----------------------------------
Zoltán Böszörményi
Cybertec Schönig & Schönig GmbH
http://www.postgresql.at/

"Zoltan Boszormenyi" <zb(at)cybertec(dot)at> writes:

> Hi,
>
> what's your opinion on this?
> I saw response only from Alvaro on the -patches list.

I don't understand. The patch only affects configuration and SQL data type
code. It doesn't actually store the 64-bit commandid anywhere which would be
the actual hard part.

Do "phantom" command ids mean this all just works magically? Ie, the limit of
2^32 <cmin,cmax> pairs is still there but as long as you don't have to store
more than that many you get to have 2^64 raw ephemeral commandids?

--
Gregory Stark
EnterpriseDB http://www.enterprisedb.com
Ask me about EnterpriseDB's Slony Replication support!

Gregory Stark wrote:
> I don't understand. The patch only affects configuration and SQL data type
> code. It doesn't actually store the 64-bit commandid anywhere which would be
> the actual hard part.

Sure it does, this is the significant part of the patch:

*** pgsql.orig/src/include/c.h 2008-03-02 13:44:45.000000000 +0100
--- pgsql-cid64/src/include/c.h 2008-03-04 21:05:23.000000000 +0100
*************** typedef TransactionId MultiXactId;
*** 382,388 ****
--- 382,392 ----

typedef uint32 MultiXactOffset;

+ #ifdef USE_64BIT_COMMANDID
+ typedef uint64 CommandId;
+ #else
typedef uint32 CommandId;
+ #endif

#define FirstCommandId ((CommandId) 0)

CommandId type is used in htup.h and elsewhere, which changes the
on-disk format.

--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com

On Mar 10, 2008, at 12:06 PM, Heikki Linnakangas wrote:
> Gregory Stark wrote:
>> I don't understand. The patch only affects configuration and SQL
>> data type
>> code. It doesn't actually store the 64-bit commandid anywhere
>> which would be
>> the actual hard part.
>
> Sure it does, this is the significant part of the patch:
>
> *** pgsql.orig/src/include/c.h 2008-03-02 13:44:45.000000000 +0100
> --- pgsql-cid64/src/include/c.h 2008-03-04 21:05:23.000000000 +0100
> *************** typedef TransactionId MultiXactId;
> *** 382,388 ****
> --- 382,392 ----
>
> typedef uint32 MultiXactOffset;
>
> + #ifdef USE_64BIT_COMMANDID
> + typedef uint64 CommandId;
> + #else
> typedef uint32 CommandId;
> + #endif
>
> #define FirstCommandId ((CommandId) 0)
>
> CommandId type is used in htup.h and elsewhere, which changes the
> on-disk format.

If we're going to make this a ./configure option, ISTM we should do
the same with XID size as well. I know there are high-velocity
databases that could use that.
--
Decibel!, aka Jim C. Nasby, Database Architect decibel(at)decibel(dot)org
Give your computer some brain candy! www.distributed.net Team #1828

"Decibel!" <decibel(at)decibel(dot)org> writes:

> If we're going to make this a ./configure option, ISTM we should do the same
> with XID size as well. I know there are high-velocity databases that could use
> that.

Keep in mind we just changed things so that read-only transactions don't
consume xids. That means you would have to be actually modifying 2-billion
records before wrap-around becomes an issue.

If you're modifying 2-billion records that quickly presumably you're going to
have other pressing reasons to run vacuum aside from xid freezing...

Also, consider that you're suggesting increasing the per-tuple overhead from
24 bytes to, if my arithmetic is right, 40 bytes.

So really you would need, say, a system with enough i/o bandwidth to handle
2-billion updates or inserts per day and with enough spare i/o bandwidth that
another 16-bytes on every one of those updates is ok, but without the ability
to run vacuum.

Also, we still have hope that the visibility map info will make running vacuum
even less of an imposition.

All that said I don't really see much reason not to make it an option. I just
don't think anyone really needs it. In 5-10 years though...

--
Gregory Stark
EnterpriseDB http://www.enterprisedb.com
Ask me about EnterpriseDB's 24x7 Postgres support!

Gregory Stark <stark(at)enterprisedb(dot)com> writes:
> All that said I don't really see much reason not to make it an option. I just
> don't think anyone really needs it. In 5-10 years though...

The manpower we'd have to invest in making it work and keeping it
working would be enough reason ...

regards, tom lane

>
> "Decibel!" <decibel(at)decibel(dot)org> writes:
>
>> If we're going to make this a ./configure option, ISTM we should
>> do the same
>> with XID size as well. I know there are high-velocity databases
>> that could use
>> that.
>
> Keep in mind we just changed things so that read-only transactions
> don't
> consume xids. That means you would have to be actually modifying 2-
> billion
> records before wrap-around becomes an issue.
>
> If you're modifying 2-billion records that quickly presumably
> you're going to
> have other pressing reasons to run vacuum aside from xid freezing...
>
> Also, consider that you're suggesting increasing the per-tuple
> overhead from
> 24 bytes to, if my arithmetic is right, 40 bytes.
>
> So really you would need, say, a system with enough i/o bandwidth
> to handle
> 2-billion updates or inserts per day and with enough spare i/o
> bandwidth that
> another 16-bytes on every one of those updates is ok, but without
> the ability
> to run vacuum.
>
> Also, we still have hope that the visibility map info will make
> running vacuum
> even less of an imposition.
>
> All that said I don't really see much reason not to make it an
> option. I just
> don't think anyone really needs it. In 5-10 years though...
>

Doing this for XIDs is pretty useless this days.
It is only targeted for command ids which are consumed heavily by
stored procedure languages.
It happens once on a while that a complex business logic procedure
runs out of command ids inside a transaction.
the idea is to give users a chance to avoid that.
touching XIDs does not make sense to me at all.

many thanks,

hans

--
Cybertec Schönig & Schönig GmbH
PostgreSQL Solutions and Support
Gröhrmühlgasse 26, 2700 Wiener Neustadt
Tel: +43/1/205 10 35 / 340
www.postgresql.at, www.cybertec.at

Hans-Juergen Schoenig <hs(at)cybertec(dot)at> writes:
> Doing this for XIDs is pretty useless this days.
> It is only targeted for command ids which are consumed heavily by
> stored procedure languages.
> It happens once on a while that a complex business logic procedure
> runs out of command ids inside a transaction.
> the idea is to give users a chance to avoid that.
> touching XIDs does not make sense to me at all.

In view of the fact that 8.3 greatly reduced the CommandID consumption
of typical plpgsql code
http://archives.postgresql.org/pgsql-committers/2007-11/msg00585.php
I wonder whether the case for wider CIDs hasn't likewise taken a
major hit.

regards, tom lane