WIP patch for system-catalog vacuuming via a relation map

Attached is the current state of my work on letting system catalogs
be processed by new-style VACUUM FULL (a/k/a CLUSTER). I haven't done
the WAL support nor worried about interlocking concurrent updates of
relation map files, but it passes the regression tests and can do
VACUUM FULL of every system catalog. So I think this is proof that
the approach is workable and there are no further gotchas to stumble
over in dark recesses of the system. The patch adds about 1250 lines
of code, mostly in a new file relmapper.c, and will be bigger once
WAL support is added. However, I anticipate that we will be able to
rip out probably three times that much code by eliminating VACUUM FULL
INPLACE (commands/vacuum.c is alone almost 4000 lines, though of course
it won't all go away). So it will be a substantial net reduction in
code size, plus we will get the previously discussed benefits such as
removing reindex-in-place for shared indexes.

Barring objections I'm going to press ahead with completing and
committing this; then in a separate patch remove VACUUM FULL INPLACE.

regards, tom lane

Tom Lane wrote:

> Barring objections I'm going to press ahead with completing and
> committing this; then in a separate patch remove VACUUM FULL INPLACE.

With the second patch, we will continue to support reading
XVAC_MOVED_OUT and IN hint bits, but never set them, correct?

--
Alvaro Herrera http://www.CommandPrompt.com/
PostgreSQL Replication, Consulting, Custom Development, 24x7 support

Alvaro Herrera <alvherre(at)commandprompt(dot)com> writes:
> Tom Lane wrote:
>> Barring objections I'm going to press ahead with completing and
>> committing this; then in a separate patch remove VACUUM FULL INPLACE.

> With the second patch, we will continue to support reading
> XVAC_MOVED_OUT and IN hint bits, but never set them, correct?

Right, the tqual support has to stay for as long as we support binary
upgrade from previous releases. (Unless we want to work out some way
of ensuring no MOVED bits remain set in a database.) There is one
heck of a lot of other infrastructure and complexity that can go away,
though.

regards, tom lane

> Barring objections I'm going to press ahead with completing and
> committing this; then in a separate patch remove VACUUM FULL INPLACE.

Was it our determination that we could remove VFI if we eliminated the
system catalogs? I'm fine with it, I just thought some people had a
marginal use case for VFI.

--Josh Berkus

Josh Berkus <josh(at)agliodbs(dot)com> writes:
>> Barring objections I'm going to press ahead with completing and
>> committing this; then in a separate patch remove VACUUM FULL INPLACE.

> Was it our determination that we could remove VFI if we eliminated the
> system catalogs? I'm fine with it, I just thought some people had a
> marginal use case for VFI.

I thought the consensus was to remove it if possible. There may still
be some "marginal" use cases, but they don't justify the work that'd
be needed to make it play safely with HS; let alone fixing the other
longstanding gotchas with it, like the double-commit risk.

regards, tom lane

Tom Lane wrote:
> Josh Berkus <josh(at)agliodbs(dot)com> writes:
> >> Barring objections I'm going to press ahead with completing and
> >> committing this; then in a separate patch remove VACUUM FULL INPLACE.
>
> > Was it our determination that we could remove VFI if we eliminated the
> > system catalogs? I'm fine with it, I just thought some people had a
> > marginal use case for VFI.
>
> I thought the consensus was to remove it if possible. There may still
> be some "marginal" use cases, but they don't justify the work that'd
> be needed to make it play safely with HS; let alone fixing the other
> longstanding gotchas with it, like the double-commit risk.

Oh, yea, I remember, HS killed it because of the work required to make
FULL work for 9.0. Thanks.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

Tom Lane wrote:
> Josh Berkus <josh(at)agliodbs(dot)com> writes:
>>> Barring objections I'm going to press ahead with completing and
>>> committing this; then in a separate patch remove VACUUM FULL INPLACE.
>
>> Was it our determination that we could remove VFI if we eliminated the
>> system catalogs? I'm fine with it, I just thought some people had a
>> marginal use case for VFI.
>
> I thought the consensus was to remove it if possible. There may still
> be some "marginal" use cases, but they don't justify the work that'd
> be needed to make it play safely with HS; let alone fixing the other
> longstanding gotchas with it, like the double-commit risk.

I think part of the plan was to also provide an online reorg tool that
works by doing dummy UPDATEs, which means that you can get serialization
errors in serializable mode, but doesn't need to lock the table.

--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com

Heikki Linnakangas <heikki(dot)linnakangas(at)enterprisedb(dot)com> writes:
> Tom Lane wrote:
>> I thought the consensus was to remove it if possible. There may still
>> be some "marginal" use cases, but they don't justify the work that'd
>> be needed to make it play safely with HS; let alone fixing the other
>> longstanding gotchas with it, like the double-commit risk.

> I think part of the plan was to also provide an online reorg tool that
> works by doing dummy UPDATEs, which means that you can get serialization
> errors in serializable mode, but doesn't need to lock the table.

Yeah. There's a good deal of interest in incremental/partial vacuuming.
But that wouldn't make use of the existing VFI infrastructure either.

regards, tom lane