Re: pgsql-server/src/backend catalog/index.c comma ...

CVSROOT: /cvsroot
Module name: pgsql-server
Changes by: tgl(at)svr1(dot)postgresql(dot)org 03/09/19 16:57:42

Modified files:
src/backend/catalog: index.c
src/backend/commands: indexcmds.c

Log message:
Seems like a bad idea that REINDEX TABLE supports (or thinks it does)
reindexing system tables without ignoring system indexes, when the
other two varieties of REINDEX disallow it. Make all three act the same,
and simplify downstream code accordingly.

> -----Original Message-----
> From: Tom Lane
>
> CVSROOT: /cvsroot
> Module name: pgsql-server
> Changes by: tgl(at)svr1(dot)postgresql(dot)org 03/09/19 16:57:42
>
> Modified files:
> src/backend/catalog: index.c
> src/backend/commands: indexcmds.c
>
> Log message:
> Seems like a bad idea that REINDEX TABLE supports (or
> thinks it does)
> reindexing system tables without ignoring system
> indexes,

Why ?

regards.
Hiroshi Inoue

"Hiroshi Inoue" <inoue(at)tpf(dot)co(dot)jp> writes:
>> Log message:
>> Seems like a bad idea that REINDEX TABLE supports (or
>> thinks it does)
>> reindexing system tables without ignoring system
>> indexes,

> Why ?

I'd ask the question the other way: why would it be a good idea to allow
this in REINDEX TABLE and not in the other two cases? And did it really
work?

The REINDEX code is messy and fragile enough that I think we should do
whatever we can to simplify it.

regards, tom lane

>
> "Hiroshi Inoue" <inoue(at)tpf(dot)co(dot)jp> writes:
> >> Log message:
> >> Seems like a bad idea that REINDEX TABLE supports (or
> >> thinks it does)
> >> reindexing system tables without ignoring system
> >> indexes,
>
> > Why ?
>
> I'd ask the question the other way: why would it be a good
> idea to allow
> this in REINDEX TABLE and not in the other two cases? And
> did it really
> work?

Yes. I would revert your change.

regards,
Hiroshi Inoue

"Hiroshi Inoue" <inoue(at)tpf(dot)co(dot)jp> writes:
>> I'd ask the question the other way: why would it be a good
>> idea to allow
>> this in REINDEX TABLE and not in the other two cases? And
>> did it really
>> work?

> Yes. I would revert your change.

You didn't answer the first question: why's this a good idea?
It seems risky and of little value to try to support system
table reindexing without disabling system indexes.

Also, your assertion that it works doesn't convince me. That business
in reindex_table about doing two setRelhasindex() calls gave me the
willies. Why was that needed? "to keep consistency with WAL" isn't
enough commentary for code as strange as that. And having a
CommandCounterIncrement() that's executed in some cases and not others
is a recipe for bugs; we've been burnt by that before.

regards, tom lane

> "Hiroshi Inoue" <inoue(at)tpf(dot)co(dot)jp> writes:
> >> I'd ask the question the other way: why would it be a good
> >> idea to allow
> >> this in REINDEX TABLE and not in the other two cases? And
> >> did it really
> >> work?
>
> > Yes. I would revert your change.
>
> You didn't answer the first question: why's this a good idea?
> It seems risky and

> of little value to try to support system
> table reindexing without disabling system indexes.

Why could you determine it ? Is PostgreSQL your system ?
Because It is never of little value of cource, I supported it.
I intended to support on-line REINDEX from the first, I first
implemented REINDEX in standalone mode not in bootstrap(Jan's
idea) mode. I also intended to support on-line reindexing nailed
relations but I didn't have time to achive it.

> Also, your assertion that it works doesn't convince me. That business
> in reindex_table about doing two setRelhasindex() calls gave me the
> willies. Why was that needed?

The setRelhasIndex() has no meaning to other backends, i.e the false state
is never visible to other backends.

> "to keep consistency with WAL" isn't
> enough commentary for code as strange as that. And having a
> CommandCounterIncrement() that's executed in some cases and not others
> is a recipe for bugs; we've been burnt by that before.

The code you are referring is to reindex pg_class relation. The code has
never
active unless an #ifdef is defined.

regards,
Hiroshi Inoue

"Hiroshi Inoue" <inoue(at)tpf(dot)co(dot)jp> writes:
> Why could you determine it ? Is PostgreSQL your system ?

Well, if you prefer, we can have a discussion and vote about it on pghackers.
But you have not answered my questions. Why was the code set up to
allow live reindexing of system tables via REINDEX TABLE, but not via
REINDEX INDEX or REINDEX DATABASE? It seems to me that those three
cases ought to work equally well or badly. Let's remove the restriction
from all three, if it's not needed.

>> Also, your assertion that it works doesn't convince me. That business
>> in reindex_table about doing two setRelhasindex() calls gave me the
>> willies. Why was that needed?

> The setRelhasIndex() has no meaning to other backends, i.e the false state
> is never visible to other backends.

Then why bother? I agree that it's probably not needed for other
backends --- since you have an exclusive lock on the table, nobody else
should be trying to examine it. But then why do it at all? The only
thing you need is to force IsIgnoringSystemIndexes true locally, which
would seem to me a much more secure way of proceeding.

The goal that I am trying to accomplish here is to make the reindexing
code more visibly safe and correct. I'm not convinced that we still
need live-reindexing capability in 7.4 (I think the btree space
recycling code solves the problem better). But I'm happy to leave it in
if these other questions can be addressed.

regards, tom lane

> "Hiroshi Inoue" <inoue(at)tpf(dot)co(dot)jp> writes:
> > Why could you determine it ? Is PostgreSQL your system ?
>
> Well, if you prefer, we can have a discussion and vote about
> it on pghackers.

Oh discussion *first* is good but You committed *first*.
So isn't it reasonable to revert your change *first* ?

This is the second time you disable the on-line reindex
functionality for system tables. Why must I explain the
same thing many times ?

May I ask again ? Is PostgreSQL your system ?

regards,
Hiroshi Inoue

On Sun, 21 Sep 2003, Hiroshi Inoue wrote:

> > "Hiroshi Inoue" <inoue(at)tpf(dot)co(dot)jp> writes:
> > > Why could you determine it ? Is PostgreSQL your system ?
> >
> > Well, if you prefer, we can have a discussion and vote about
> > it on pghackers.
>
> Oh discussion *first* is good but You committed *first*.
> So isn't it reasonable to revert your change *first* ?
>
> This is the second time you disable the on-line reindex
> functionality for system tables. Why must I explain the
> same thing many times ?

Actually, as a comment here, since I *think* I understand where Tom is
coming from ... and since I've either missed it, or it hasn't been
answered yet ... why was the original patch incomplete in only addressing
1 of 3 REINDEX conditions? Is there a reason why that one condition
is/was safe to do it with, but not the other 2?

Again ... if I understand Tom's objections to, and reason for reversing,
this patch correctly ...

"Marc G. Fournier" <scrappy(at)postgresql(dot)org> writes:
> Actually, as a comment here, since I *think* I understand where Tom is
> coming from ... and since I've either missed it, or it hasn't been
> answered yet ... why was the original patch incomplete in only addressing
> 1 of 3 REINDEX conditions? Is there a reason why that one condition
> is/was safe to do it with, but not the other 2?

That's exactly what's bothering me. Where I'd like to end up is that
either all three variants of REINDEX allow this, or all three do not.
I don't understand why only REINDEX TABLE should support it.

regards, tom lane

> -----Original Message-----
> From: Marc G. Fournier [mailto:scrappy(at)postgresql(dot)org]
> Sent: Sunday, September 21, 2003 6:45 AM
> To: Hiroshi Inoue
> Cc: 'Tom Lane'; pgsql-committers(at)postgresql(dot)org
> Subject: Re: [COMMITTERS] pgsql-server/src/backend
> catalog/index.c comma ...
>
> On Sun, 21 Sep 2003, Hiroshi Inoue wrote:
>
> > > "Hiroshi Inoue" <inoue(at)tpf(dot)co(dot)jp> writes:
> > > > Why could you determine it ? Is PostgreSQL your system ?
> > >
> > > Well, if you prefer, we can have a discussion and vote about
> > > it on pghackers.
> >
> > Oh discussion *first* is good but You committed *first*.
> > So isn't it reasonable to revert your change *first* ?
> >
> > This is the second time you disable the on-line reindex
> > functionality for system tables. Why must I explain the
> > same thing many times ?
>
> Actually, as a comment here, since I *think* I understand where Tom is
> coming from ... and since I've either missed it, or it hasn't been
> answered yet ... why was the original patch incomplete in
> only addressing
> 1 of 3 REINDEX conditions? Is there a reason why that one condition
> is/was safe to do it with, but not the other 2?

Sorry to trouble you.

In the first place, REINDEX is neither an SQL standard command nor
a preferable one.

When I introduced REINDEX command before 7.0, it was not
transaction-safe and only allowed to call under standalone mode
essentially.

Before 7.1, the introduction of pg_class.relfilenode gave us a possibilty
to make REINDEX command transaction-safe and I tried to make
REINDEX available under postmaster and the result was
1.All user indexes/tables could be REINDEXed under postmaster
2.System tables except shared or nailed ones could be REINDEXed
under postmaster.

Note that we couldn't reindex all system tables under postmaster.
It's the main reason why I didn't implement REINDEX DATABASE
under postmaster. As for REINDEX, I have

Tue Nov 20 02:46:13 2001 UTC (22 months ago) Tom committed
the following change which disables the functionality to reindex
system tables under postmaster.
Some minor tweaks of REINDEX processing: grab exclusive
lock a little earlier, make error checks more uniform.

The above change was the first time he disables the functionality.
I noticed the change, complained to him and
Thu Feb 7 00:27:30 2002 UTC (19 months, 1 week ago) I
Removed a check for REINDEX TABLE.

And this is the second time he disables the functionality without
any notification to me. Honestly I don't understand why I have to
explain this kind of thing only so as to revert the change.

regards,
Hiroshi Inoue

I've been looking at the issues involved in reindexing system tables,
and I now have what I think is a fairly defensible set of proposals.

We should whenever possible use the same reindexing technique used by
CLUSTER: assign a new relfilenode number, build the new index in that
file, and apply an ordinary heap_update operation to update the index's
pg_class row with the new relfilenode value. This technique is fully
crash-safe and transaction-safe (meaning it can be rolled back even after
completion, in case of failure later in the same transaction). However,
there are several pitfalls for applying it to system indexes:

1. There is a problem for a shared index, because we have no way to
update pg_class.relfilenode in all the other databases besides ours.
I see no good way around this problem.

2. There is a problem for a nailed-in-cache index, because the relcache
isn't prepared to cope with relfilenode updates for such indexes.
However, that is fixable.

3. There is a problem for an index on pg_class itself: doing heap_update
on a pg_class row must make new index entries. We have to be careful
that the new row does appear in the updated index, while not making
entries in old-and-possibly-broken indexes. This is doable.

4. There is a problem with updating indexes that might be used for catalog
fetches executed during the index_build process: if we try to use the
partially-rebuilt index for such a fetch we will fail. In the case of
disaster recovery the problem doesn't exist because we'll have
IsIgnoringSystemIndexes true, but for an ordinary on-line indexing
operation there's definitely a risk. Presently the code relies on
"deactivating indexes" to prevent this, but I think it can be done more
simply, because we only need to suppress access to the target index
locally in the reindexing backend. (Other backends will be locked out by
means of an AccessExclusiveLock on the system catalog in question.)

In short then, we can fix things so that the new-relfilenode path can be
taken in all cases except shared indexes. For shared indexes, we have
little choice but to truncate and rebuild the index in-place. (There is
then no need to update its pg_class row at all.) This is of course
horribly crash-unsafe.

The code presently uses a "deactivated indexes" flag (namely, setting
pg_class.relhasindex false) to provide some protection against a crash
midway through an in-place reindex. However, this concept is really quite
useless for a shared index, because only one database could contain the
deactivation flag. Accesses from any other database would still try to
use the broken index.

Based on this analysis, I feel that the "deactivated indexes" mechanism
is of no further value and should be retired. We should instead simply
acknowledge that reindexing shared indexes is dangerous. I propose that
it be allowed only from a standalone backend. Specifically I think that
REINDEX INDEX and REINDEX TABLE should error out if pointed to a shared
index or table, unless running in a standalone backend; REINDEX DATABASE
should skip over the shared indexes unless running in a standalone
backend. (I'm not convinced that either -O or -P needs to be insisted on
by REINDEX, btw, but we can discuss that separately.) We'll have to warn
users not to try to restart the database when reindexing of a shared table
wasn't successfully completed.

Details to back up the above claims:

Part of the code needed to fix the relcache restriction on nailed indexes
is already there, but ifdef'd out; that's the part that re-reads the
index's pg_class row after receiving SI inval for it. There are some
cases not yet handled though. In the first place, if the nailed index
being modified is pg_class_oid_index, ScanPgRelation will try to use that
same index to load the updated row, and will fail because it'll be trying
to use the old relfilenode. We can easily force a seqscan in that case,
however. A more subtle problem is that if an SI buffer overflow occurs,
RelationCacheInvalidate walks through the relation cache in a random
order. I believe that the correct update order has to be first
pg_class_oid_index, then the other nailed indexes, and finally all other
relation cache entries. pg_class_oid_index has to be updated first (with
the aforementioned seqscan), since it will be used by ScanPgRelation to
reread the pg_class rows for the other nailed indexes. Only when we are
sure the nailed indexes are up-to-date can we safely rebuild other
relcache entries.

Assigning a new relfilenode to indexes of pg_class is not a big deal when
we don't have an index-corruption problem. We simply have to make the new
heap_updated row before we start index_build (which indeed the code does
already); both old and new rows will be indexed in the new index, and all
is well. However, if we suspect index corruption then it is important not
to try to make entries into the old indexes, because we might crash trying
to update a corrupt index. I propose the following behavior:

REINDEX INDEX: no special action; this is not the thing to use
when index corruption is suspected anyway.

REINDEX TABLE: while operating on pg_class, arrange for the
relcache's list of known indexes of pg_class to contain only the
indexes already reindexed. In this way, only those indexes will
be updated by CatalogUpdateIndexes(), and so no untrustworthy
indexes will be touched while making new pg_class rows.

REINDEX DATABASE: take care to process pg_class first.

To avoid catalog accesses via an index that's being rebuilt, we can simply
generalize the SetReindexProcessing() state to include the OID of the
index currently being rebuilt. Places that presently make checks for
IsIgnoringSystemIndexes can check this as well and fall back to seqscans
when the targeted index would have been used.

One other point: IsIgnoringSystemIndexes seems presently to be taken to
mean not only that we don't *read* the system indexes, but that we don't
*write* them either. I think this is horribly dangerous; it effectively
means that the only thing you can safely do in a backend started with -P
is REINDEX. If you make any other changes to the system catalogs then
you've re-corrupted the indexes. Also, you don't have any protection
against duplicate entries getting made, since the unique indexes are
disabled. It would be a lot safer to define IsIgnoringSystemIndexes as
preventing the system indexes from being used for lookups, while still
updating the indexes on any catalog modification. This will not affect
the safety of REINDEX and will make other operations much less prone to
pilot error.

Comments?

regards, tom lane

First it should have been discussed before your commitment or at least
it should be discussed after reversing your change.

I require you to explain me why you committed the change
with no discussion and little investigation.

I also noticed that your change for catalog/index.c

Revision 1.200 / (download) - annotate - [select for diffs] , Mon Sep 23
00:42:48 2002 UTC (11 months, 4 weeks ago) by tgl
Changes since 1.199: +124 -161 lines
Diff to previous 1.199
Get rid of bogus use of heap_mark4update in reindex operations (cf.
recent bug report). Fix processing of nailed-in-cache indexes;
it appears that REINDEX DATABASE has been broken for months :-(.

removed an #ifndef ENABLE_REINDEX_NAILED_RELATIONS
trial stuff. I'm very surprised to see it now.

regards,
Hiroshi Inoue

> -----Original Message-----
> From: Tom Lane [mailto:tgl(at)sss(dot)pgh(dot)pa(dot)us]
>
> I've been looking at the issues involved in reindexing system tables,
> and I now have what I think is a fairly defensible set of proposals.
>
> We should whenever possible use the same reindexing technique used by
> CLUSTER: assign a new relfilenode number, build the new index in that
> file, and apply an ordinary heap_update operation to update
> the index's
> pg_class row with the new relfilenode value. This technique is fully
> crash-safe and transaction-safe (meaning it can be rolled
> back even after
> completion, in case of failure later in the same
> transaction). However,
> there are several pitfalls for applying it to system indexes:
>
> 1. There is a problem for a shared index, because we have no way to
> update pg_class.relfilenode in all the other databases besides ours.
> I see no good way around this problem.
>
> 2. There is a problem for a nailed-in-cache index, because
> the relcache
> isn't prepared to cope with relfilenode updates for such indexes.
> However, that is fixable.
>
> 3. There is a problem for an index on pg_class itself: doing
> heap_update
> on a pg_class row must make new index entries. We have to be careful
> that the new row does appear in the updated index, while not making
> entries in old-and-possibly-broken indexes. This is doable.
>
> 4. There is a problem with updating indexes that might be
> used for catalog
> fetches executed during the index_build process: if we try to use the
> partially-rebuilt index for such a fetch we will fail. In the case of
> disaster recovery the problem doesn't exist because we'll have
> IsIgnoringSystemIndexes true, but for an ordinary on-line indexing
> operation there's definitely a risk. Presently the code relies on
> "deactivating indexes" to prevent this, but I think it can be
> done more
> simply, because we only need to suppress access to the target index
> locally in the reindexing backend. (Other backends will be
> locked out by
> means of an AccessExclusiveLock on the system catalog in question.)
>
> In short then, we can fix things so that the new-relfilenode
> path can be
> taken in all cases except shared indexes. For shared indexes, we have
> little choice but to truncate and rebuild the index in-place.
> (There is
> then no need to update its pg_class row at all.) This is of course
> horribly crash-unsafe.
>
> The code presently uses a "deactivated indexes" flag (namely, setting
> pg_class.relhasindex false) to provide some protection against a crash
> midway through an in-place reindex. However, this concept is
> really quite
> useless for a shared index, because only one database could
> contain the
> deactivation flag. Accesses from any other database would
> still try to
> use the broken index.
>
> Based on this analysis, I feel that the "deactivated indexes"
> mechanism
> is of no further value and should be retired. We should
> instead simply
> acknowledge that reindexing shared indexes is dangerous. I
> propose that
> it be allowed only from a standalone backend. Specifically I
> think that
> REINDEX INDEX and REINDEX TABLE should error out if pointed
> to a shared
> index or table, unless running in a standalone backend;
> REINDEX DATABASE
> should skip over the shared indexes unless running in a standalone
> backend. (I'm not convinced that either -O or -P needs to be
> insisted on
> by REINDEX, btw, but we can discuss that separately.) We'll
> have to warn
> users not to try to restart the database when reindexing of a
> shared table
> wasn't successfully completed.
>
> Details to back up the above claims:
>
> Part of the code needed to fix the relcache restriction on
> nailed indexes
> is already there, but ifdef'd out; that's the part that re-reads the
> index's pg_class row after receiving SI inval for it. There are some
> cases not yet handled though. In the first place, if the nailed index
> being modified is pg_class_oid_index, ScanPgRelation will try
> to use that
> same index to load the updated row, and will fail because
> it'll be trying
> to use the old relfilenode. We can easily force a seqscan in
> that case,
> however. A more subtle problem is that if an SI buffer
> overflow occurs,
> RelationCacheInvalidate walks through the relation cache in a random
> order. I believe that the correct update order has to be first
> pg_class_oid_index, then the other nailed indexes, and
> finally all other
> relation cache entries. pg_class_oid_index has to be updated
> first (with
> the aforementioned seqscan), since it will be used by
> ScanPgRelation to
> reread the pg_class rows for the other nailed indexes. Only
> when we are
> sure the nailed indexes are up-to-date can we safely rebuild other
> relcache entries.
>
> Assigning a new relfilenode to indexes of pg_class is not a
> big deal when
> we don't have an index-corruption problem. We simply have to
> make the new
> heap_updated row before we start index_build (which indeed
> the code does
> already); both old and new rows will be indexed in the new
> index, and all
> is well. However, if we suspect index corruption then it is
> important not
> to try to make entries into the old indexes, because we might
> crash trying
> to update a corrupt index. I propose the following behavior:
>
> REINDEX INDEX: no special action; this is not the thing to use
> when index corruption is suspected anyway.
>
> REINDEX TABLE: while operating on pg_class, arrange for the
> relcache's list of known indexes of pg_class to contain only the
> indexes already reindexed. In this way, only those indexes will
> be updated by CatalogUpdateIndexes(), and so no untrustworthy
> indexes will be touched while making new pg_class rows.
>
> REINDEX DATABASE: take care to process pg_class first.
>
> To avoid catalog accesses via an index that's being rebuilt,
> we can simply
> generalize the SetReindexProcessing() state to include the OID of the
> index currently being rebuilt. Places that presently make checks for
> IsIgnoringSystemIndexes can check this as well and fall back
> to seqscans
> when the targeted index would have been used.
>
> One other point: IsIgnoringSystemIndexes seems presently to
> be taken to
> mean not only that we don't *read* the system indexes, but
> that we don't
> *write* them either. I think this is horribly dangerous; it
> effectively
> means that the only thing you can safely do in a backend
> started with -P
> is REINDEX. If you make any other changes to the system catalogs then
> you've re-corrupted the indexes. Also, you don't have any protection
> against duplicate entries getting made, since the unique indexes are
> disabled. It would be a lot safer to define
> IsIgnoringSystemIndexes as
> preventing the system indexes from being used for lookups, while still
> updating the indexes on any catalog modification. This will
> not affect
> the safety of REINDEX and will make other operations much
> less prone to
> pilot error.
>
> Comments?
>
> regards, tom lane
>

"Hiroshi Inoue" <inoue(at)tpf(dot)co(dot)jp> writes:
> I require you to explain me why you committed the change
> with no discussion and little investigation.

If you want an apology for not having discussed it in advance, I'll
gladly offer one. It was poorly done.

I do, however, think that the reindexing code needs work. Can we
get on with discussing the technical issues?

regards, tom lane

On Mon, Sep 22, 2003 at 04:56:35AM +0900, Hiroshi Inoue wrote:
> First it should have been discussed before your commitment or at least
> it should be discussed after reversing your change.
>
> I require you to explain me why you committed the change
> with no discussion and little investigation.
>
> I also noticed that your change for catalog/index.c
>
> Revision 1.200 / (download) - annotate - [select for diffs] , Mon Sep 23
> 00:42:48 2002 UTC (11 months, 4 weeks ago) by tgl

You know you're talking about a commit of 1 year ago?

Kurt

Kurt Roeckx wrote:
>
> On Mon, Sep 22, 2003 at 04:56:35AM +0900, Hiroshi Inoue wrote:
> > First it should have been discussed before your commitment or at least
> > it should be discussed after reversing your change.
> >
> > I require you to explain me why you committed the change
> > with no discussion and little investigation.
> >
> > I also noticed that your change for catalog/index.c
> >
> > Revision 1.200 / (download) - annotate - [select for diffs] , Mon Sep 23
> > 00:42:48 2002 UTC (11 months, 4 weeks ago) by tgl
>
> You know you're talking about a commit of 1 year ago?

Yes of cource. Unfortunately I haven't had enough time to
check all the changes since long. This time I was lucky to
find a change to spoil my effort. Is it strange for me to
doubt other places ? I looked again the current code about
REINDEX command and found the above change which is closely
related to this topic.

regards,
Hiroshi Inoue
http://www.geocities.jp/inocchichichi/psqlodbc/

Tom Lane wrote:
>
> "Hiroshi Inoue" <inoue(at)tpf(dot)co(dot)jp> writes:
> > I require you to explain me why you committed the change
> > with no discussion and little investigation.
>
> If you want an apology for not having discussed it in advance, I'll
> gladly offer one. It was poorly done.

Thanks.
> I do, however, think that the reindexing code needs work. Can we
> get on with discussing the technical issues?

OK but I would need some time to remember the code.

regards,
Hiroshi Inoue
http://www.geocities.jp/inocchichichi/psqlodbc/

Here is the proposed patch. This fixes the problems that prevented the
relcache from coping with relfilenode changes for nailed-in-cache
indexes. Accordingly, REINDEX is now crash-safe and transaction-safe
for all cases except reindexing shared system catalogs. That case is
permitted only in a standalone backend (!IsUnderPostmaster). We no
longer require -O to reindex system tables, and since -P now suppresses
only reads not writes of system indexes, it is safe to allow from the
client side. Upshot: reindexing system catalogs can be done without a
standalone backend for all cases except shared catalogs.

regards, tom lane

I've just put back your previous change, sorry.
As I already mentioned many times it must be the first thing.

Though I don't remenber my code completely yet, I would
reply to some points.
Unfortunately REINDEX wasn't a eagerly wanted command when
I implemented it. Though I wanted to introduce a per index
flag, I unwillingly used an existent per table flag(relhasindex)
instead. Because it was impossible to make REINDEX transaction-safe
then, such flag was needed to suppress inconsistency as less
as possible.
I also unwillingly introduced the ReindexProcessing mode(flag)
because I didn't think of other quick solutions.

IIRC the main reason why I gave up the REINDEX functionality
on nailed relations was the difficulty of reindexing pg_class
and the handling of relcache overflow. I didn't have much time
to test it. In addtion REINDEX wasn't a recognized command
then and I found no one to discuss the situation.

Tom Lane wrote:
>
> I've been looking at the issues involved in reindexing system tables,
> and I now have what I think is a fairly defensible set of proposals.
>
> We should whenever possible use the same reindexing technique used by
> CLUSTER:

REINDEX was the first command which used the pg_class.relfilenode
functionality. The pg_class.relfilenode was essentially my proposal.

> 1. There is a problem for a shared index, because we have no way to
> update pg_class.relfilenode in all the other databases besides ours.
> I see no good way around this problem.

So the current REINDEX disallows on-line reindex on shared relations.

> 2. There is a problem for a nailed-in-cache index, because the relcache
> isn't prepared to cope with relfilenode updates for such indexes.
> However, that is fixable.

My code works pretty good with nailed relations except pg_class
#if defined (ENABLE_REINDEX_NAILED_RELATIONS).

> 3. There is a problem for an index on pg_class itself: doing heap_update
> on a pg_class row must make new index entries. We have to be careful
> that the new row does appear in the updated index, while not making
> entries in old-and-possibly-broken indexes. This is doable.

Yes.

Sorry I have no time to continue the discussion now.

regards,
Hiroshi Inoue
http://www.geocities.jp/inocchichichi/psqlodbc/

Hiroshi Inoue wrote:
> instead. Because it was impossible to make REINDEX transaction-safe
> then, such flag was needed to suppress inconsistency as less
> as possible.

This mean that the actual REINDEX is not transaction-safe ?

Regards
Gaetano Mendola

> -----Original Message-----
> From: Gaetano Mendola [mailto:mendola(at)bigfoot(dot)com]
>
> Hiroshi Inoue wrote:
> > instead. Because it was impossible to make REINDEX transaction-safe
> > then, such flag was needed to suppress inconsistency as less
> > as possible.
>
> This mean that the actual REINDEX is not transaction-safe ?

No.
It was not transaction-safe long time ago.

regards,
Hiroshi Inoue

Hiroshi Inoue wrote:
Gaetano Mendola [mailto:mendola(at)bigfoot(dot)com] wrote:
>>
>>Hiroshi Inoue wrote:
>>
>>>instead. Because it was impossible to make REINDEX transaction-safe
>>>then, such flag was needed to suppress inconsistency as less
>>>as possible.
>>
>>This mean that the actual REINDEX is not transaction-safe ?
>
>
> No.
> It was not transaction-safe long time ago.

Anyway I think there is a nasty bug somewhere, see my last posts
about "duplication primary key".

Regards
Gaetano Mendola

Where are we on this? Does the code path now make sense, at least?

---------------------------------------------------------------------------

Hiroshi Inoue wrote:
> > -----Original Message-----
> > From: Marc G. Fournier [mailto:scrappy(at)postgresql(dot)org]
> > Sent: Sunday, September 21, 2003 6:45 AM
> > To: Hiroshi Inoue
> > Cc: 'Tom Lane'; pgsql-committers(at)postgresql(dot)org
> > Subject: Re: [COMMITTERS] pgsql-server/src/backend
> > catalog/index.c comma ...
> >
> > On Sun, 21 Sep 2003, Hiroshi Inoue wrote:
> >
> > > > "Hiroshi Inoue" <inoue(at)tpf(dot)co(dot)jp> writes:
> > > > > Why could you determine it ? Is PostgreSQL your system ?
> > > >
> > > > Well, if you prefer, we can have a discussion and vote about
> > > > it on pghackers.
> > >
> > > Oh discussion *first* is good but You committed *first*.
> > > So isn't it reasonable to revert your change *first* ?
> > >
> > > This is the second time you disable the on-line reindex
> > > functionality for system tables. Why must I explain the
> > > same thing many times ?
> >
> > Actually, as a comment here, since I *think* I understand where Tom is
> > coming from ... and since I've either missed it, or it hasn't been
> > answered yet ... why was the original patch incomplete in
> > only addressing
> > 1 of 3 REINDEX conditions? Is there a reason why that one condition
> > is/was safe to do it with, but not the other 2?
>
> Sorry to trouble you.
>
> In the first place, REINDEX is neither an SQL standard command nor
> a preferable one.
>
> When I introduced REINDEX command before 7.0, it was not
> transaction-safe and only allowed to call under standalone mode
> essentially.
>
> Before 7.1, the introduction of pg_class.relfilenode gave us a possibilty
> to make REINDEX command transaction-safe and I tried to make
> REINDEX available under postmaster and the result was
> 1.All user indexes/tables could be REINDEXed under postmaster
> 2.System tables except shared or nailed ones could be REINDEXed
> under postmaster.
>
> Note that we couldn't reindex all system tables under postmaster.
> It's the main reason why I didn't implement REINDEX DATABASE
> under postmaster. As for REINDEX, I have
>
> Tue Nov 20 02:46:13 2001 UTC (22 months ago) Tom committed
> the following change which disables the functionality to reindex
> system tables under postmaster.
> Some minor tweaks of REINDEX processing: grab exclusive
> lock a little earlier, make error checks more uniform.
>
> The above change was the first time he disables the functionality.
> I noticed the change, complained to him and
> Thu Feb 7 00:27:30 2002 UTC (19 months, 1 week ago) I
> Removed a check for REINDEX TABLE.
>
> And this is the second time he disables the functionality without
> any notification to me. Honestly I don't understand why I have to
> explain this kind of thing only so as to revert the change.
>
> regards,
> Hiroshi Inoue
>
>
>
> ---------------------------(end of broadcast)---------------------------
> TIP 2: you can get off all lists at once with the unregister command
> (send "unregister YourEmailAddressHere" to majordomo(at)postgresql(dot)org)
>

--
Bruce Momjian | http://candle.pha.pa.us
pgman(at)candle(dot)pha(dot)pa(dot)us | (610) 359-1001
+ If your life is a hard drive, | 13 Roberts Road
+ Christ can be your backup. | Newtown Square, Pennsylvania 19073

Bruce Momjian <pgman(at)candle(dot)pha(dot)pa(dot)us> writes:
> Where are we on this? Does the code path now make sense, at least?

I committed the fixes a few minutes ago. I now actually believe that
reindexing system tables works ;-) ... hopefully Hiroshi does too.

regards, tom lane

> -----Original Message-----
> From: Tom Lane [mailto:tgl(at)sss(dot)pgh(dot)pa(dot)us]
>
> Bruce Momjian <pgman(at)candle(dot)pha(dot)pa(dot)us> writes:
> > Where are we on this? Does the code path now make sense, at least?
>
> I committed the fixes a few minutes ago. I now actually believe that
> reindexing system tables works ;-) ... hopefully Hiroshi does too.

I don't see your commit message yet.

Anyway I I'm not completely agree with your proposal or patch.
Different from you, I'm far from full time developper and I haven't
have much time to check your proposal or patch.
Is it wrong to believe I have a right to conitune the discussion ?

regards,
Hiroshi Inoue

Hiroshi Inoue wrote:
> > -----Original Message-----
> > From: Tom Lane [mailto:tgl(at)sss(dot)pgh(dot)pa(dot)us]
> >
> > Bruce Momjian <pgman(at)candle(dot)pha(dot)pa(dot)us> writes:
> > > Where are we on this? Does the code path now make sense, at least?
> >
> > I committed the fixes a few minutes ago. I now actually believe that
> > reindexing system tables works ;-) ... hopefully Hiroshi does too.
>
> I don't see your commit message yet.
>
> Anyway I I'm not completely agree with your proposal or patch.
> Different from you, I'm far from full time developper and I haven't
> have much time to check your proposal or patch.
> Is it wrong to believe I have a right to conitune the discussion ?

Please continue the discussion, but Hiroshi, we need you to look at the
code and give us your ideas.

--
Bruce Momjian | http://candle.pha.pa.us
pgman(at)candle(dot)pha(dot)pa(dot)us | (610) 359-1001
+ If your life is a hard drive, | 13 Roberts Road
+ Christ can be your backup. | Newtown Square, Pennsylvania 19073

"Hiroshi Inoue" <inoue(at)tpf(dot)co(dot)jp> writes:
>> I committed the fixes a few minutes ago. I now actually believe that
>> reindexing system tables works ;-) ... hopefully Hiroshi does too.

> I don't see your commit message yet.

CVS reports to the committers list seem to be wedged this afternoon :-(.
I already complained to Marc about it. But in the meantime here is the
cvs2cl entry:

2003-09-24 14:54 tgl

* doc/src/sgml/ref/postgres-ref.sgml,
doc/src/sgml/ref/reindex.sgml, src/backend/access/index/genam.c,
src/backend/access/transam/xact.c, src/backend/catalog/index.c,
src/backend/catalog/pg_largeobject.c,
src/backend/commands/functioncmds.c,
src/backend/commands/indexcmds.c, src/backend/commands/vacuum.c,
src/backend/executor/execUtils.c,
src/backend/executor/nodeIndexscan.c,
src/backend/storage/ipc/sinval.c, src/backend/tcop/postgres.c,
src/backend/tcop/utility.c, src/backend/utils/cache/relcache.c,
src/backend/utils/cache/syscache.c,
src/backend/utils/init/miscinit.c, src/include/miscadmin.h,
src/include/catalog/index.h, src/include/utils/errcodes.h,
src/include/utils/rel.h, src/include/utils/relcache.h: Repair some
REINDEX problems per recent discussions. The relcache is now able
to cope with assigning new relfilenode values to nailed-in-cache
indexes, so they can be reindexed using the fully crash-safe
method. This leaves only shared system indexes as special cases.
Remove the 'index deactivation' code, since it provides no useful
protection in the shared- index case. Require reindexing of shared
indexes to be done in standalone mode, but remove other
restrictions on REINDEX. -P (IgnoreSystemIndexes) now prevents
using indexes for lookups, but does not disable index updates. It
is therefore safe to allow from PGOPTIONS. Upshot: reindexing
system catalogs can be done without a standalone backend for all
cases except shared catalogs.

It's essentially the same as what I posted two days ago, except I
thought it would be a good idea to make REINDEX TABLE take care of the
associated TOAST table too, rather than expecting users to deal with
the toast table separately.

> Is it wrong to believe I have a right to conitune the discussion ?

Not at all, but I already waited two days for your response, and I can't
put development on hold indefinitely. For now, those changes are in,
and you are welcome to suggest further improvements at your leisure.

regards, tom lane

> -----Original Message-----
> From: Tom Lane [mailto:tgl(at)sss(dot)pgh(dot)pa(dot)us]
>
> "Hiroshi Inoue" <inoue(at)tpf(dot)co(dot)jp> writes:
> >> I committed the fixes a few minutes ago. I now actually
> believe that
> >> reindexing system tables works ;-) ... hopefully Hiroshi does too.
>
> > I don't see your commit message yet.
>
> CVS reports to the committers list seem to be wedged this
> afternoon :-(.
> I already complained to Marc about it. But in the meantime
> here is the
> cvs2cl entry:
>
> Remove the 'index deactivation' code, since it provides
> no useful
> protection in the shared- index case.

Seems a funny reason to me.
Well, do you also remove reltuples or relpages from pg_class ?

As I already mentioned, I don't love the ReindexProcessing global
flag from the first. For example, it's not thread-safe.

> > Is it wrong to believe I have a right to conitune the discussion ?
>
> Not at all, but I already waited two days for your response,
> and I can't
> put development on hold indefinitely.

I've spent a pretty much time only to put back your change.
What I've got until now was just a temporary neutral state.
It's not a neutral state now again. Should I be satisfied with it ?

regards,
Hiroshi Inoue

"Hiroshi Inoue" <inoue(at)tpf(dot)co(dot)jp> writes:
>> Remove the 'index deactivation' code, since it provides
>> no useful protection in the shared- index case.

> Seems a funny reason to me.

Well, as you know I never liked that code; modifying permanent on-disk
data didn't seem like a sensible way to protect against intra-transaction
interlock problems. So when I saw a chance to get rid of it, I thought
we should take it.

> I've spent a pretty much time only to put back your change.

Why? As far as I know, the modified code does everything you want.

regards, tom lane

> -----Original Message-----
> From: Tom Lane [mailto:tgl(at)sss(dot)pgh(dot)pa(dot)us]
>
> "Hiroshi Inoue" <inoue(at)tpf(dot)co(dot)jp> writes:
> >> Remove the 'index deactivation' code, since it provides
> >> no useful protection in the shared- index case.
>
> > Seems a funny reason to me.
>
> Well, as you know I never liked that code; modifying permanent on-disk
> data didn't seem like a sensible way to protect against intra-transaction
> interlock problems. So when I saw a chance to get rid of it, I thought
> we should take it.

You dislike it and so what ?
Are you interested in the discussion ?

> > I've spent a pretty much time only to put back your change.
>
> Why? As far as I know, the modified code does everything you want.

How do you what I want ?
In the first place, there has been no discussion like discussion
in this topic.

regards,
Hiroshi Inoue

"Hiroshi Inoue" <Inoue(at)tpf(dot)co(dot)jp> writes:
>> From: Tom Lane [mailto:tgl(at)sss(dot)pgh(dot)pa(dot)us]
>> Why? As far as I know, the modified code does everything you want.

> How do you what I want ?

If you aren't going to tell me, I guess I won't know :-(. It seems
like you want to turn this into some kind of personal issue. If you
want to tell me what technical problems you see with this code, I'm
very willing to listen.

regards, tom lane

> -----Original Message-----
> From: Tom Lane [mailto:tgl(at)sss(dot)pgh(dot)pa(dot)us]
>
> "Hiroshi Inoue" <Inoue(at)tpf(dot)co(dot)jp> writes:
> >> From: Tom Lane [mailto:tgl(at)sss(dot)pgh(dot)pa(dot)us]
> >> Why? As far as I know, the modified code does everything you want.
>
> > How do you what I want ?
>
> If you aren't going to tell me, I guess I won't know :-(. It seems
> like you want to turn this into some kind of personal issue.

Personal issue ?
Are you going to dodge the issue ?

regards,
Hiroshi Inoue

Tom, would you summarize what REINDEX currently _doesn't_ do?

---------------------------------------------------------------------------

Tom Lane wrote:
> I've been looking at the issues involved in reindexing system tables,
> and I now have what I think is a fairly defensible set of proposals.
>
> We should whenever possible use the same reindexing technique used by
> CLUSTER: assign a new relfilenode number, build the new index in that
> file, and apply an ordinary heap_update operation to update the index's
> pg_class row with the new relfilenode value. This technique is fully
> crash-safe and transaction-safe (meaning it can be rolled back even after
> completion, in case of failure later in the same transaction). However,
> there are several pitfalls for applying it to system indexes:
>
> 1. There is a problem for a shared index, because we have no way to
> update pg_class.relfilenode in all the other databases besides ours.
> I see no good way around this problem.
>
> 2. There is a problem for a nailed-in-cache index, because the relcache
> isn't prepared to cope with relfilenode updates for such indexes.
> However, that is fixable.
>
> 3. There is a problem for an index on pg_class itself: doing heap_update
> on a pg_class row must make new index entries. We have to be careful
> that the new row does appear in the updated index, while not making
> entries in old-and-possibly-broken indexes. This is doable.
>
> 4. There is a problem with updating indexes that might be used for catalog
> fetches executed during the index_build process: if we try to use the
> partially-rebuilt index for such a fetch we will fail. In the case of
> disaster recovery the problem doesn't exist because we'll have
> IsIgnoringSystemIndexes true, but for an ordinary on-line indexing
> operation there's definitely a risk. Presently the code relies on
> "deactivating indexes" to prevent this, but I think it can be done more
> simply, because we only need to suppress access to the target index
> locally in the reindexing backend. (Other backends will be locked out by
> means of an AccessExclusiveLock on the system catalog in question.)
>
> In short then, we can fix things so that the new-relfilenode path can be
> taken in all cases except shared indexes. For shared indexes, we have
> little choice but to truncate and rebuild the index in-place. (There is
> then no need to update its pg_class row at all.) This is of course
> horribly crash-unsafe.
>
> The code presently uses a "deactivated indexes" flag (namely, setting
> pg_class.relhasindex false) to provide some protection against a crash
> midway through an in-place reindex. However, this concept is really quite
> useless for a shared index, because only one database could contain the
> deactivation flag. Accesses from any other database would still try to
> use the broken index.
>
> Based on this analysis, I feel that the "deactivated indexes" mechanism
> is of no further value and should be retired. We should instead simply
> acknowledge that reindexing shared indexes is dangerous. I propose that
> it be allowed only from a standalone backend. Specifically I think that
> REINDEX INDEX and REINDEX TABLE should error out if pointed to a shared
> index or table, unless running in a standalone backend; REINDEX DATABASE
> should skip over the shared indexes unless running in a standalone
> backend. (I'm not convinced that either -O or -P needs to be insisted on
> by REINDEX, btw, but we can discuss that separately.) We'll have to warn
> users not to try to restart the database when reindexing of a shared table
> wasn't successfully completed.
>
> Details to back up the above claims:
>
> Part of the code needed to fix the relcache restriction on nailed indexes
> is already there, but ifdef'd out; that's the part that re-reads the
> index's pg_class row after receiving SI inval for it. There are some
> cases not yet handled though. In the first place, if the nailed index
> being modified is pg_class_oid_index, ScanPgRelation will try to use that
> same index to load the updated row, and will fail because it'll be trying
> to use the old relfilenode. We can easily force a seqscan in that case,
> however. A more subtle problem is that if an SI buffer overflow occurs,
> RelationCacheInvalidate walks through the relation cache in a random
> order. I believe that the correct update order has to be first
> pg_class_oid_index, then the other nailed indexes, and finally all other
> relation cache entries. pg_class_oid_index has to be updated first (with
> the aforementioned seqscan), since it will be used by ScanPgRelation to
> reread the pg_class rows for the other nailed indexes. Only when we are
> sure the nailed indexes are up-to-date can we safely rebuild other
> relcache entries.
>
> Assigning a new relfilenode to indexes of pg_class is not a big deal when
> we don't have an index-corruption problem. We simply have to make the new
> heap_updated row before we start index_build (which indeed the code does
> already); both old and new rows will be indexed in the new index, and all
> is well. However, if we suspect index corruption then it is important not
> to try to make entries into the old indexes, because we might crash trying
> to update a corrupt index. I propose the following behavior:
>
> REINDEX INDEX: no special action; this is not the thing to use
> when index corruption is suspected anyway.
>
> REINDEX TABLE: while operating on pg_class, arrange for the
> relcache's list of known indexes of pg_class to contain only the
> indexes already reindexed. In this way, only those indexes will
> be updated by CatalogUpdateIndexes(), and so no untrustworthy
> indexes will be touched while making new pg_class rows.
>
> REINDEX DATABASE: take care to process pg_class first.
>
> To avoid catalog accesses via an index that's being rebuilt, we can simply
> generalize the SetReindexProcessing() state to include the OID of the
> index currently being rebuilt. Places that presently make checks for
> IsIgnoringSystemIndexes can check this as well and fall back to seqscans
> when the targeted index would have been used.
>
> One other point: IsIgnoringSystemIndexes seems presently to be taken to
> mean not only that we don't *read* the system indexes, but that we don't
> *write* them either. I think this is horribly dangerous; it effectively
> means that the only thing you can safely do in a backend started with -P
> is REINDEX. If you make any other changes to the system catalogs then
> you've re-corrupted the indexes. Also, you don't have any protection
> against duplicate entries getting made, since the unique indexes are
> disabled. It would be a lot safer to define IsIgnoringSystemIndexes as
> preventing the system indexes from being used for lookups, while still
> updating the indexes on any catalog modification. This will not affect
> the safety of REINDEX and will make other operations much less prone to
> pilot error.
>
> Comments?
>
> regards, tom lane
>
> ---------------------------(end of broadcast)---------------------------
> TIP 9: the planner will ignore your desire to choose an index scan if your
> joining column's datatypes do not match
>

--
Bruce Momjian | http://candle.pha.pa.us
pgman(at)candle(dot)pha(dot)pa(dot)us | (610) 359-1001
+ If your life is a hard drive, | 13 Roberts Road
+ Christ can be your backup. | Newtown Square, Pennsylvania 19073

Hiroshi Inoue wrote:
> > -----Original Message-----
> > From: Tom Lane [mailto:tgl(at)sss(dot)pgh(dot)pa(dot)us]
> >
> > "Hiroshi Inoue" <Inoue(at)tpf(dot)co(dot)jp> writes:
> > >> From: Tom Lane [mailto:tgl(at)sss(dot)pgh(dot)pa(dot)us]
> > >> Why? As far as I know, the modified code does everything you want.
> >
> > > How do you what I want ?
> >
> > If you aren't going to tell me, I guess I won't know :-(. It seems
> > like you want to turn this into some kind of personal issue.
>
> Personal issue ?
> Are you going to dodge the issue ?

Hiroshi, Tom already apologized for doing this without prior discussion.
What open issues are there?

--
Bruce Momjian | http://candle.pha.pa.us
pgman(at)candle(dot)pha(dot)pa(dot)us | (610) 359-1001
+ If your life is a hard drive, | 13 Roberts Road
+ Christ can be your backup. | Newtown Square, Pennsylvania 19073

Bruce Momjian <pgman(at)candle(dot)pha(dot)pa(dot)us> writes:
> Tom, would you summarize what REINDEX currently _doesn't_ do?

As of CVS tip I think the only deficiency is that indexes on the shared
catalogs (pg_database, pg_shadow, pg_group) have to be reindexed in
place, rather than being rebuilt with a new relfilenode as is done for
CLUSTER or TRUNCATE. In-place reindexing isn't crash-safe, since if
you fail you're left with a half-built (effectively corrupt) index.

I don't see any way to avoid that, though, since we cannot change the
relfilenode value for a shared index.

I was toying with the notion of changing btree index build to not write
the metapage until the index is fully built; in this way, at least the
corrupted state of the index would be obvious. (You'd get "not a btree"
failures.)

regards, tom lane

Tom Lane wrote:
> Bruce Momjian <pgman(at)candle(dot)pha(dot)pa(dot)us> writes:
> > Tom, would you summarize what REINDEX currently _doesn't_ do?
>
> As of CVS tip I think the only deficiency is that indexes on the shared
> catalogs (pg_database, pg_shadow, pg_group) have to be reindexed in
> place, rather than being rebuilt with a new relfilenode as is done for
> CLUSTER or TRUNCATE. In-place reindexing isn't crash-safe, since if
> you fail you're left with a half-built (effectively corrupt) index.
>
> I don't see any way to avoid that, though, since we cannot change the
> relfilenode value for a shared index.
>
> I was toying with the notion of changing btree index build to not write
> the metapage until the index is fully built; in this way, at least the
> corrupted state of the index would be obvious. (You'd get "not a btree"
> failures.)

Oh, that's great. I can't imagine a lot of traffic in those shared
tables anyway. So you implemented all the ideas in the email --- great.

--
Bruce Momjian | http://candle.pha.pa.us
pgman(at)candle(dot)pha(dot)pa(dot)us | (610) 359-1001
+ If your life is a hard drive, | 13 Roberts Road
+ Christ can be your backup. | Newtown Square, Pennsylvania 19073

On Sat, Sep 27, 2003 at 06:37:22PM -0400, Tom Lane wrote:
> Bruce Momjian <pgman(at)candle(dot)pha(dot)pa(dot)us> writes:
> > Tom, would you summarize what REINDEX currently _doesn't_ do?
>
> As of CVS tip I think the only deficiency is that indexes on the shared
> catalogs (pg_database, pg_shadow, pg_group) have to be reindexed in
> place, rather than being rebuilt with a new relfilenode as is done for
> CLUSTER or TRUNCATE. In-place reindexing isn't crash-safe, since if
> you fail you're left with a half-built (effectively corrupt) index.
>
> I don't see any way to avoid that, though, since we cannot change the
> relfilenode value for a shared index.

What about creating a separate filenode anyway and renaming the files
afterwards? It would not be an atomic operation anyway, but it would be
better than the current setup IMHO.

(It seems unlikely that one would have > 1GB indexes for shared
relations...)

--
Alvaro Herrera (<alvherre[a]dcc.uchile.cl>)
"La espina, desde que nace, ya pincha" (Proverbio africano)

Alvaro Herrera <alvherre(at)dcc(dot)uchile(dot)cl> writes:
> On Sat, Sep 27, 2003 at 06:37:22PM -0400, Tom Lane wrote:
>> I don't see any way to avoid that, though, since we cannot change the
>> relfilenode value for a shared index.

> What about creating a separate filenode anyway and renaming the files
> afterwards? It would not be an atomic operation anyway, but it would be
> better than the current setup IMHO.

I think it would be difficult to persuade the buffer manager and storage
manager to work with this; from their point of view you'd be moving a
relation underneath them. I doubt it's really worth the trouble; how
often do you need to reindex a shared catalog?

regards, tom lane

Tom Lane wrote:
> > What about creating a separate filenode anyway and renaming the files
> > afterwards? It would not be an atomic operation anyway, but it would be
> > better than the current setup IMHO.
>
> I think it would be difficult to persuade the buffer manager and storage
> manager to work with this; from their point of view you'd be moving a
> relation underneath them. I doubt it's really worth the trouble; how
> often do you need to reindex a shared catalog?

The point I missed originally is that he is talking about shared
catalogs, not system catalogs, which work fine in CVS.

--
Bruce Momjian | http://candle.pha.pa.us
pgman(at)candle(dot)pha(dot)pa(dot)us | (610) 359-1001
+ If your life is a hard drive, | 13 Roberts Road
+ Christ can be your backup. | Newtown Square, Pennsylvania 19073

> -----Original Message-----
> From: Bruce Momjian [mailto:pgman(at)candle(dot)pha(dot)pa(dot)us]
>
> Hiroshi Inoue wrote:
> > > -----Original Message-----
> > > From: Tom Lane [mailto:tgl(at)sss(dot)pgh(dot)pa(dot)us]
> > >
> > > "Hiroshi Inoue" <Inoue(at)tpf(dot)co(dot)jp> writes:
> > > >> From: Tom Lane [mailto:tgl(at)sss(dot)pgh(dot)pa(dot)us]
> > > >> Why? As far as I know, the modified code does
> everything you want.
> > >
> > > > How do you what I want ?
> > >
> > > If you aren't going to tell me, I guess I won't know :-(.
> It seems
> > > like you want to turn this into some kind of personal issue.
> >
> > Personal issue ?
> > Are you going to dodge the issue ?
>
> Hiroshi, Tom already apologized for doing this without prior
> discussion.

Though there has been no discussion like discussion on this
item between Tom and me, his code is already there.
Is it reasonbale ? As I already mentioned many many times
putting back his change should have been the first thing but
it wasn't.

regards,
Hiroshi Inoue

Hiroshi Inoue wrote:
> > > Personal issue ?
> > > Are you going to dodge the issue ?
> >
> > Hiroshi, Tom already apologized for doing this without prior
> > discussion.
>
> Though there has been no discussion like discussion on this
> item between Tom and me, his code is already there.
> Is it reasonbale ? As I already mentioned many many times
> putting back his change should have been the first thing but
> it wasn't.

That is true --- he should have discussed it first (he apologized), and
put it back when there was objection, but I think he was waiting to find
out what the objection was. Now that it is done, is there something
that still needs attention. He said that reindex covers everything but
reindexing of shared relation indexes, which is only pg_database,
pg_user, and pg_group, all of which are low modification tables.

--
Bruce Momjian | http://candle.pha.pa.us
pgman(at)candle(dot)pha(dot)pa(dot)us | (610) 359-1001
+ If your life is a hard drive, | 13 Roberts Road
+ Christ can be your backup. | Newtown Square, Pennsylvania 19073

"Hiroshi Inoue" <inoue(at)tpf(dot)co(dot)jp> writes:
> Though there has been no discussion like discussion on this
> item between Tom and me, his code is already there.
> Is it reasonbale ? As I already mentioned many many times
> putting back his change should have been the first thing but
> it wasn't.

You already reverted the first version. I followed full protocol
in the second version: I posted a design spec and then a proposed
patch, and I waited what seemed a reasonable length of time for
comments before committing. You do not have the right to expect
that commits will be held off indefinitely if you don't have the
time to respond to discussion.

I should also point out that according to the CVS logs, you have not
touched the reindex code in nearly two years. If you were actively
working on it, I'd surely not have joggled your elbow. But you can't
expect that there is a permanent KEEP OUT sign on code you once worked
on.

regards, tom lane

On Sat, Sep 27, 2003 at 08:18:07PM -0400, Bruce Momjian wrote:
> Tom Lane wrote:
> > > What about creating a separate filenode anyway and renaming the files
> > > afterwards? It would not be an atomic operation anyway, but it would be
> > > better than the current setup IMHO.
> >
> > I think it would be difficult to persuade the buffer manager and storage
> > manager to work with this; from their point of view you'd be moving a
> > relation underneath them. I doubt it's really worth the trouble; how
> > often do you need to reindex a shared catalog?
>
> The point I missed originally is that he is talking about shared
> catalogs, not system catalogs, which work fine in CVS.

Well, my idea was to reduce the window of time during which the index
would be corrupt, i.e. not completely rebuilt. This is only an issue
with shared indexes, because other system indexes do use the changing
relfilenode thingie already.

However, the main reason for reindexing is a corrupt index, so if for
some reason the new index is also corrupt (e.g. the machine crashes
midway) there's no point in having a separate index filenode anyway.

--
Alvaro Herrera (<alvherre[a]dcc.uchile.cl>)
"The eagle never lost so much time, as
when he submitted to learn of the crow." (William Blake)

Bruce Momjian wrote:
>
> Hiroshi Inoue wrote:
> > > > Personal issue ?
> > > > Are you going to dodge the issue ?
> > >
> > > Hiroshi, Tom already apologized for doing this without prior
> > > discussion.
> >
> > Though there has been no discussion like discussion on this
> > item between Tom and me, his code is already there.
> > Is it reasonbale ? As I already mentioned many many times
> > putting back his change should have been the first thing but
> > it wasn't.
>
> That is true --- he should have discussed it first (he apologized),

As I already mentioned many times, thers hasn't been any
discussion like discussion yet though it should have been
the first.

After all it was me not Tom who put back his first change.
When I did it, he already posted another patch. It seems
clear to me he saw no necessity to put it back.

Also note his patch was not applicable after my putting
back the change, He has never posted other applicable patches.

I told him that I need some time to remember my code.
I also tried to reply some points of his proposal though
the trial was far from being sufficient. He suddenly applied
his modified patch without any replies to me.

In my feeling, what he has done is to dodge my questions.
Some people whom I asked have the same feeling.

He also ignored my question about "2 phase commit" in
pgsql-hackers, for example.

Should I be patient with the treatment ?

regards,
Hiroshi Inoue
http://www.geocities.jp/inocchichichi/psqlodbc/

On Mon, 29 Sep 2003, Hiroshi Inoue wrote:

> He also ignored my question about "2 phase commit" in pgsql-hackers, for
> example.

Actually, I've been following that thread pretty closely, and I believe I
missed your question :(

"Marc G. Fournier" <scrappy(at)postgresql(dot)org> writes:
> On Mon, 29 Sep 2003, Hiroshi Inoue wrote:
>> He also ignored my question about "2 phase commit" in pgsql-hackers, for
>> example.

> Actually, I've been following that thread pretty closely, and I believe I
> missed your question :(

[ checks back in thread ] I didn't respond to that message because
I didn't have anything much to add to the discussion at that point.
It wasn't anything personal; in fact I don't think I particularly
noticed that the message was from Hiroshi as opposed to anyone else
who was participating.

regards, tom lane

"Marc G. Fournier" wrote:
>
> On Mon, 29 Sep 2003, Hiroshi Inoue wrote:
>
> > He also ignored my question about "2 phase commit" in pgsql-hackers, for
> > example.
>
> Actually, I've been following that thread pretty closely, and I believe I
> missed your question :(

OK I would explain it again.

Bruced asked.

> Bruce Momjian <pgman(at)candle(dot)pha(dot)pa(dot)us> writes:
>> Tom Lane wrote:
>>> You're not considering the possibility of a transient communication
>>> failure.

>> Can't the master re-send the request after a timeout?

Tom's answer was.

> Not "it can", but "it has to".

IMHO the partcipants(masters) don't have to bear such
heavy responsibility. My answer is

Yes "it can", but "it doesn't have to".

Of cource, the cooridnater(slave) has the responsibility
to retry the commit operation for the in-doubt transaction.

regards,
Hiroshi Inoue

On Mon, 29 Sep 2003, Hiroshi Inoue wrote:

> "Marc G. Fournier" wrote:
> >
> > On Mon, 29 Sep 2003, Hiroshi Inoue wrote:
> >
> > > He also ignored my question about "2 phase commit" in pgsql-hackers, for
> > > example.
> >
> > Actually, I've been following that thread pretty closely, and I believe I
> > missed your question :(
>
> OK I would explain it again.
>
> Bruced asked.
>
> > Bruce Momjian <pgman(at)candle(dot)pha(dot)pa(dot)us> writes:
> >> Tom Lane wrote:
> >>> You're not considering the possibility of a transient communication
> >>> failure.
>
> >> Can't the master re-send the request after a timeout?
>
> Tom's answer was.
>
> > Not "it can", but "it has to".
>
> IMHO the partcipants(masters) don't have to bear such
> heavy responsibility. My answer is
>
> Yes "it can", but "it doesn't have to".
>
> Of cource, the cooridnater(slave) has the responsibility
> to retry the commit operation for the in-doubt transaction.

'k, a statement, not a question, which is why it didn't stick with me as
being unanswered :(

Tom Lane wrote:
>
> "Hiroshi Inoue" <inoue(at)tpf(dot)co(dot)jp> writes:
> > Though there has been no discussion like discussion on this
> > item between Tom and me, his code is already there.
> > Is it reasonbale ? As I already mentioned many many times
> > putting back his change should have been the first thing but
> > it wasn't.
>
> You already reverted the first version. I followed full protocol
> in the second version: I posted a design spec and then a proposed
> patch, and I waited what seemed a reasonable length of time for
> comments before committing.

> You do not have the right to expect
> that commits will be held off indefinitely if you don't have the
> time to respond to discussion.

But you were able to tell me when you would commit the
change, weren't you ? In the first place this issue was
started from your mistake and you had to be careful not
to repeat such mistake.

> I should also point out that according to the CVS logs, you have not
> touched the reindex code in nearly two years. If you were actively
> working on it, I'd surely not have joggled your elbow.

As I already mentioned many times, what you did first was
to disable a functionality. AFAIR Bruce or you always asked
the lists about the meaning when removing a code even if the
code is ancient. As for REINDEX, what you only had to do was
to ask me if the change has no problem.

regards,
Hiroshi Inoue
http://www.geocities.jp/inocchichichi/psqlodbc/

Hiroshi Inoue <Inoue(at)tpf(dot)co(dot)jp> writes:
> But you were able to tell me when you would commit the
> change, weren't you ? In the first place this issue was
> started from your mistake and you had to be careful not
> to repeat such mistake.

I already apologized for that error. I'm not sure what else
you want me to do.

> As for REINDEX, what you only had to do was
> to ask me if the change has no problem.

I posted a proposal and a patch for you to comment on.
So far you've given me no useful feedback on either,
only complaints that are looking more and more like
personal attacks.

regards, tom lane

> -----Original Message-----
> From: Tom Lane [mailto:tgl(at)sss(dot)pgh(dot)pa(dot)us]
>
> Hiroshi Inoue <Inoue(at)tpf(dot)co(dot)jp> writes:
> > But you were able to tell me when you would commit the
> > change, weren't you ? In the first place this issue was
> > started from your mistake and you had to be careful not
> > to repeat such mistake.
>
> I already apologized for that error. I'm not sure what else
> you want me to do.

Did you tell me when you would commit your patch ?
Sorry if I missed it.

> > As for REINDEX, what you only had to do was
> > to ask me if the change has no problem.
>
> I posted a proposal and a patch for you to comment on.

??? My above comment is a part of my reply to your comment
> I should also point out that according to the CVS logs, you have not
> touched the reindex code in nearly two years.

... And so I don't understand what your above reply is for.

regards,
Hiroshi Inoue