Adjusting index special storage for pg_filedump's convenience

Historically, pg_filedump
http://sources.redhat.com/rhdb/utilities.html
has relied on the size of a page's special space to determine which kind
of index it is looking at (btree, hash, etc) so that it can dump the
contents of the special space nicely. This is pretty ugly of course,
but there isn't a whole lot of other context available. (Before you
say "why not look at the metapage?", consider that we may be looking at
a segment file that doesn't contain the metapage, and gist and gin don't
use one anyway.) As of 8.2 it's entirely broken for gist because gist
and btree now have the same-size special space, ie 16 bytes; and it
looks like bitmap indexes will too.

We put in a workaround a long time ago to make it possible to tell the
difference between btree and hash special space, which are also the same
size: there's an unused 16 bits in hash special space that we fill with
a specific value. As of 8.2 this doesn't work as well as it used to,
because the corresponding space in a btree page is now used for a vacuum
cycle ID and so there's 1 chance in 65536 of a false match. Still, it's
a lot better than nothing.

I'd like to tweak things for 8.3 so that pg_filedump can work reasonably
well again. It looks like the hash solution would work for gist, gin,
and bitmap: rearranging fields would allow us to put in a 16-bit ID
field in all three cases. (For bitmap, I'm assuming that
bm_hrl_words_used could be reduced to 16 bits without problem --- it is
a per-page count not something larger, right?)

One problem with that is that with four special values, there'd be 1
chance in 16384 of misidentifying a btree page because of chance values
of the vacuum cycle ID. This can be improved a bit if we put the flags
fields (for those index types that have 'em) in a consistent place too:
we can disbelieve that an index is of type X if it doesn't have a flags
value that fits. I don't see any way to make it completely bulletproof
without enlarging the special space, which seems an unreasonable price
to pay. But even one chance in 16K is way better than the current
situation.

Thoughts, objections, better ideas?

regards, tom lane

On Mon, 9 Apr 2007, Tom Lane wrote:

> We put in a workaround a long time ago to make it possible to tell the
> difference between btree and hash special space, which are also the same
> size: there's an unused 16 bits in hash special space that we fill with
> a specific value. As of 8.2 this doesn't work as well as it used to,
> because the corresponding space in a btree page is now used for a vacuum
> cycle ID and so there's 1 chance in 65536 of a false match. Still, it's
> a lot better than nothing.

Sounds... reasonable. Especially if you add the flags test below.

>
> I'd like to tweak things for 8.3 so that pg_filedump can work reasonably
> well again. It looks like the hash solution would work for gist, gin,
> and bitmap: rearranging fields would allow us to put in a 16-bit ID
> field in all three cases. (For bitmap, I'm assuming that
> bm_hrl_words_used could be reduced to 16 bits without problem --- it is
> a per-page count not something larger, right?)

Yes, I've reduced this already but hadn't in previous patches, from
memory. I'd add a filler of uint16 now. Got a number I should use?

> One problem with that is that with four special values, there'd be 1
> chance in 16384 of misidentifying a btree page because of chance values
> of the vacuum cycle ID. This can be improved a bit if we put the flags
> fields (for those index types that have 'em) in a consistent place too:
> we can disbelieve that an index is of type X if it doesn't have a flags
> value that fits. I don't see any way to make it completely bulletproof
> without enlarging the special space, which seems an unreasonable price
> to pay. But even one chance in 16K is way better than the current
> situation.

Sounds like the only workable approach.

Thanks,

Gavin

Gavin Sherry <swm(at)alcove(dot)com(dot)au> writes:
> On Mon, 9 Apr 2007, Tom Lane wrote:
>> ... I don't see any way to make it completely bulletproof
>> without enlarging the special space, which seems an unreasonable price
>> to pay. But even one chance in 16K is way better than the current
>> situation.

> Sounds like the only workable approach.

Actually, I realized after writing that that it *is* possible to make it
bulletproof: all we have to do is make the BTCycleId wrap around at a
little less than 64K, which adds about one line of code and doesn't
materially change its reliability. That leaves a few bitpatterns free
for IDs of other index types with no chance of collision. I made hash
use 0xFF80 and gist 0xFF81; please use 0xFF82 for bitmaps. (GIN turns
out not to need a code because its special space is a different size,
so we can tell it apart anyway.)

See patch already committed here:
http://archives.postgresql.org/pgsql-committers/2007-04/msg00125.php

regards, tom lane

Tom Lane wrote:
> Gavin Sherry <swm(at)alcove(dot)com(dot)au> writes:
>> On Mon, 9 Apr 2007, Tom Lane wrote:
>>> ... I don't see any way to make it completely bulletproof
>>> without enlarging the special space, which seems an unreasonable price
>>> to pay. But even one chance in 16K is way better than the current
>>> situation.
>
>> Sounds like the only workable approach.
>
> Actually, I realized after writing that that it *is* possible to make it
> bulletproof: all we have to do is make the BTCycleId wrap around at a
> little less than 64K, which adds about one line of code and doesn't
> materially change its reliability. That leaves a few bitpatterns free
> for IDs of other index types with no chance of collision. I made hash
> use 0xFF80 and gist 0xFF81; please use 0xFF82 for bitmaps. (GIN turns
> out not to need a code because its special space is a different size,
> so we can tell it apart anyway.)
>
> See patch already committed here:
> http://archives.postgresql.org/pgsql-committers/2007-04/msg00125.php

That's a clever trick, but I can't help thinking we really should have
an explicit field in the page header to indicate what kind of a page it
is. It would make life simpler for any external tools that want to peek
into pages, including migration utilities after a release or two. We've
also been talking about setting hint bits and doing some kind of retail
vacuuming in bgwriter with HOT. To do that, we need to identify heap
pages in the bgwriter. While heap pages can currently be identified by
the fact that they don't have a special area, it feels hackish, and we
might want to do something like that for index pages too in the future.

We now have a 16-bit pd_flags field in the page header. We could use a
few bits from that.

--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com

Heikki Linnakangas <heikki(at)enterprisedb(dot)com> writes:
> That's a clever trick, but I can't help thinking we really should have
> an explicit field in the page header to indicate what kind of a page it
> is.

I think we should save the pd_flags field for cases where we really need
it ...

regards, tom lane

Heikki Linnakangas wrote:

>
> That's a clever trick, but I can't help thinking we really should have
> an explicit field in the page header to indicate what kind of a page it
> is. It would make life simpler for any external tools that want to peek
> into pages, including migration utilities after a release or two. We've
> also been talking about setting hint bits and doing some kind of retail
> vacuuming in bgwriter with HOT. To do that, we need to identify heap
> pages in the bgwriter. While heap pages can currently be identified by
> the fact that they don't have a special area, it feels hackish, and we
> might want to do something like that for index pages too in the future.
>
> We now have a 16-bit pd_flags field in the page header. We could use a
> few bits from that.
>

+1

or add one extra field

Zdenek