Proposal: PageLayout footprint

Current content of control file is insufficient to check if database is
compatible with postgres server. I think we should create footprint file which
will contains all page data structures and their member offset and length (and
maybe more).

As a first step which is useful at least for in-place-upgrade is footprint
switch which prints page layout structures. I attach WIP patch.

I'm thinking also to create pg_footprint view with (version, structure name,
attribute, offset, size) columns. But I'm not sure about usefulness of this
catalog (?maybe for regression tests?).

Comments, ideas?

Thanks Zdenek

--
Zdenek Kotala Sun Microsystems
Prague, Czech Republic http://sun.com/postgresql

Zdenek Kotala wrote:
> Current content of control file is insufficient to check if database is
> compatible with postgres server.

It is? Do you have an example of where it's insufficient?

--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com

Heikki Linnakangas napsal(a):
> Zdenek Kotala wrote:
>> Current content of control file is insufficient to check if database
>> is compatible with postgres server.
>
> It is? Do you have an example of where it's insufficient?
>

Current control file contain following information (related to page layout):

maxAlign
blcksz
toast_max_chunk_size

But you don't have control how aligned is each member of data structure.

By my opinion -fipa-struct-reorg GCC option could break structure. And maybe
there are more compiler magic switches and optimization on different platforms
which can modify structure alignment or member order. It probably does not
happen often but footprint should protect people to shot himself.

Zdenek

PS: And of course toast_max_chunk_size is not insufficient as well. There are
more constants like MaxHeapTupleSize and so on, but it is different story.

Zdenek Kotala wrote:
> By my opinion -fipa-struct-reorg GCC option could break structure.

That option would probably break a lot of things. Like our
"variable-sized array as last field of a struct" hacks.

> And
> maybe there are more compiler magic switches and optimization on
> different platforms which can modify structure alignment or member
> order. It probably does not happen often but footprint should protect
> people to shot himself.

We depend on a certain member order and alignment rules. If we're
worried about that, we should add checks in configure instead, to barf
if you try to use such options.

--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com

"Zdenek Kotala" <Zdenek(dot)Kotala(at)Sun(dot)COM> writes:

> By my opinion -fipa-struct-reorg GCC option could break structure. And maybe
> there are more compiler magic switches and optimization on different platforms
> which can modify structure alignment or member order. It probably does not
> happen often but footprint should protect people to shot himself.

My version of GCC doesn't have that option, what does it do?

If structure members aren't in the order they're defined and padded to the
alignment they're declared to have in pg_type then Postgres catalogs won't
work anyways.

--
Gregory Stark
EnterpriseDB http://www.enterprisedb.com
Ask me about EnterpriseDB's RemoteDBA services!

Gregory Stark wrote:
> "Zdenek Kotala" <Zdenek(dot)Kotala(at)Sun(dot)COM> writes:
>> By my opinion -fipa-struct-reorg GCC option could break structure. And maybe
>> there are more compiler magic switches and optimization on different platforms
>> which can modify structure alignment or member order. It probably does not
>> happen often but footprint should protect people to shot himself.
>
> My version of GCC doesn't have that option, what does it do?

From gcc man page:

> -fipa-struct-reorg
> Perform structure reorganization optimization, that change C-like
> structures layout in order to better utilize spatial locality.
> This transformation is affective for programs containing arrays of
> structures. Available in two compilation modes: profile-based
> (enabled with -fprofile-generate) or static (which uses built-in
> heuristics). Require -fipa-type-escape to provide the safety of
> this transformation. It works only in whole program mode, so it
> requires -fwhole-program and -combine to be enabled. Structures
> considered cold by this transformation are not affected (see
> --param struct-reorg-cold-struct-ratio=value).

--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com

Heikki Linnakangas napsal(a):
> Zdenek Kotala wrote:
>> By my opinion -fipa-struct-reorg GCC option could break structure.
>
> That option would probably break a lot of things. Like our
> "variable-sized array as last field of a struct" hacks.

Yes, it is extreme case.

>> And maybe there are more compiler magic switches and optimization on
>> different platforms which can modify structure alignment or member
>> order. It probably does not happen often but footprint should protect
>> people to shot himself.
>
> We depend on a certain member order and alignment rules. If we're
> worried about that, we should add checks in configure instead, to barf
> if you try to use such options.
>

You are able to check order, but you cannot complain about structure member
alignment during configure time. But if you have two binaries which you get from
two sources then you need to verify that both binaries has same structure footprint.

Similar is 32/64 bit compilation. It is handled on x86 by MAXALIGN but MAXALIGN
is same on SPARC for both binaries, but I'm not sure if it works correctly.

Any other usage is to protect developers to make a mistake and break silently
compatibility, but it should be caught by --footprint switch.

Zdenek

Zdenek Kotala wrote:
> Similar is 32/64 bit compilation. It is handled on x86 by MAXALIGN but
> MAXALIGN is same on SPARC for both binaries, but I'm not sure if it
> works correctly.

Are 32/64-bit binaries on Sparc incompatible, then? Does our control
file check catch it? If not, what's causing it, and would the
--footprint switch catch it?

--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com

Heikki Linnakangas napsal(a):
> Zdenek Kotala wrote:
>> Similar is 32/64 bit compilation. It is handled on x86 by MAXALIGN but
>> MAXALIGN is same on SPARC for both binaries, but I'm not sure if it
>> works correctly.
>
> Are 32/64-bit binaries on Sparc incompatible, then? Does our control
> file check catch it? If not, what's causing it, and would the
> --footprint switch catch it?
>

It is what I don't know. controlfile only says "incorrect checksum in control
file". It seems to me that CRC computing does not work correctly. I check
pg_control footprint and it is same for 32/64. It seems to me a false positive
complain, but ...

Footprint switch should help show you why it is incompatible and also protect
you during development to break some structure.

Zdenek

Zdenek Kotala wrote:
> It is what I don't know. controlfile only says "incorrect checksum in
> control file". It seems to me that CRC computing does not work
> correctly. I check pg_control footprint and it is same for 32/64. It
> seems to me a false positive complain, but ...

Endianness perhaps? Per comment in ControlFileData:

> * This data is used to check for hardware-architecture compatibility of
> * the database and the backend executable. We need not check endianness
> * explicitly, since the pg_control version will surely look wrong to a
> * machine of different endianness, but we do need to worry about MAXALIGN
> * and floating-point format. (Note: storage layout nominally also
> * depends on SHORTALIGN and INTALIGN, but in practice these are the same
> * on all architectures of interest.)

> Footprint switch should help show you why it is incompatible and also
> protect you during development to break some structure.

Apparently the footprint switch didn't provide any insight into the
Sparc 32/64-bit issue, so I'm not too impressed..

--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com

Heikki Linnakangas napsal(a):
> Zdenek Kotala wrote:
>> It is what I don't know. controlfile only says "incorrect checksum in
>> control file". It seems to me that CRC computing does not work
>> correctly. I check pg_control footprint and it is same for 32/64. It
>> seems to me a false positive complain, but ...
>
> Endianness perhaps? Per comment in ControlFileData:

No. It is same Endianness. SPARC allows to map memory with different endian, but
it needs extra magic.

Zdenek